Panoramic video encoding method, apparatus, device and readable storage medium

Abstract

The present disclosure relates to a panoramic video encoding method, device, equipment and readable storage medium. Compared with the prior art, the embodiment of the present disclosure considers the subjective quality of the video by predicting the panoramic video through window prediction, predicting a plurality of candidate windows of the panoramic video and an importance coefficient of each candidate window, further performs saliency prediction based on the plurality of candidate windows to obtain a window saliency map, solves the problem of waste of computing power or code rate in saliency processing of the whole frame, calculates the allocated code rate of each coding tree unit in the panoramic video based on the plurality of candidate windows of the panoramic video, the importance coefficient of each candidate window, the window saliency map and a preset code rate allocation rule of the coding tree unit, encodes the panoramic video based on the allocated code rate of each coding tree unit, greatly saves the code rate under the condition of guaranteeing the subjective viewing quality, is not limited to the online encoding scene, can be applied to the offline viewing scene, and has strong applicability.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a panoramic video encoding method, apparatus, device, and readable storage medium. Background Technology

[0002] In planar video coding, the R-λ method is the most advanced rate control method.

[0003] Existing panoramic video coding schemes are mainly optimized through the following three methods: 1) Spatial distortion-based optimization schemes: Many panoramic video coding methods take into account the spatial distortion caused by the Equirectangular Projection (ERP) format and propose to modify the application of the R-λ method in panoramic video based on some projection-friendly Peak Signal-to-Noise Ratio (PSNR) weights; 2) Subjective quality enhancement schemes: Some panoramic video coding methods enhance subjective quality by considering the salient regions of video frames; 3) Viewpoint adaptation schemes: Considering that viewers can only see the image within the viewpoint while watching panoramic video, some online panoramic video coding methods propose to allocate more bitrate to the viewpoint area.

[0004] However, the above methods 1) do not take into account the subjective quality of the video, resulting in poor subjective quality; methods 2) that perform saliency processing on the entire frame result in wasted computing power or bitrate; and methods 3) are limited to online encoding scenarios and cannot be directly applied to offline viewing scenarios. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a panoramic video encoding method, apparatus, device, and readable storage medium.

[0006] In a first aspect, embodiments of this disclosure provide a panoramic video encoding method, the method comprising: Window prediction is performed on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window; Based on the multiple candidate windows, a saliency prediction is performed to obtain a window saliency map; Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, the allocated bitrate of each coding tree unit in the panoramic video is calculated. The panoramic video is encoded based on the allocated bitrate of each coding tree unit.

[0007] In some embodiments, the step of performing window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and an importance coefficient for each candidate window includes: For any two consecutive frames of a panoramic video, convolutional feature extraction is performed on the two frames respectively to obtain the spatial features of the two frames; The spatial features of the two frames are input into the bidirectional encoding and decoding module to obtain the panoramic features of the two frames. The panoramic features of the two frames include the intra-frame features and the inter-frame features of the two frames. The panoramic features of the two frames are input to a fully connected layer for mapping to obtain the viewports and viewport information of the two frames. The viewport information of the two frames includes the viewport coordinates of the two frames and the movement vector of the viewport between the two frames. The viewports of the two frames are filtered to determine multiple candidate viewports of the two frames in the panoramic video and the matching relationship between the multiple candidate viewports of the two frames.

[0008] In some embodiments, inputting the spatial features of the two frames of images into a bidirectional encoding / decoding module to obtain the panoramic features of the two frames of images includes: The spatial features of the two frames are input into the encoder in the bidirectional encoding and decoding module for intra-frame feature extraction to obtain the intra-frame features of the two frames. The intra-frame features of the two frames are input into the decoder in the bidirectional encoding and decoding module for inter-frame feature extraction to obtain the inter-frame features of the two frames.

[0009] In some embodiments, the step of performing saliency prediction based on the plurality of candidate windows to obtain a window saliency map includes: Multiple candidate windows are input into an initial convolutional group for feature extraction to obtain the input image features; Residual feature extraction is performed on the input image features to obtain a multi-level spatial feature map; Based on the receptive field of the multi-level spatial feature map at different sizes, the focus is predicted to obtain a multi-level feature map. The feature size is transformed and the feature information is fused to obtain the saliency map of the window.

[0010] In some embodiments, the step of extracting residual features from the input image features to obtain a multi-level spatial feature map includes: The input image features are sequentially passed through multiple residual modules in series to obtain an initial multi-level spatial feature map; The initial multi-level spatial feature map is corrected by the convolutional block attention module to obtain a multi-level spatial feature map.

[0011] In some embodiments, the step of predicting the points of interest based on the receptive fields of the multi-level spatial feature maps at different sizes to obtain multi-level feature maps includes: The multi-level spatial feature map is input into the feature pyramid. Each level of the feature pyramid contains an increasing number of channels and a decreasing feature map size. The feature pyramid is used to predict the points of interest, resulting in a multi-level feature map.

[0012] In some embodiments, calculating the allocated bitrate for each coding tree unit in the panoramic video based on multiple candidate viewports of the panoramic video, the importance coefficient of each candidate viewport, the saliency map of the viewport, and a preset bitrate allocation rule for coding tree units includes: Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, the weight corresponding to each coding tree unit in the panoramic video is calculated. Determine the distortion and fixed code rate for each coding tree unit; Based on the weight corresponding to each coding tree unit, the distortion of each coding tree unit, and the fixed bitrate, the allocated bitrate of each coding tree unit in the panoramic video is calculated. Secondly, embodiments of this disclosure provide a panoramic video encoding device, comprising: The first obtaining module is used to perform window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window. The second obtaining module is used to perform saliency prediction based on the multiple candidate windows to obtain a window saliency map; The calculation module is used to calculate the allocated bitrate of each coding tree unit in the panoramic video based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit. The encoding module is used to encode the panoramic video based on the allocated bitrate of each encoding tree unit.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: Memory; Processor; and Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0014] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method as described in the first aspect.

[0015] Fifthly, embodiments of this disclosure also provide a computer program product comprising a computer program or instructions that, when executed by a processor, implement the method described in the first aspect.

[0016] The panoramic video encoding method, apparatus, device, and readable storage medium provided in this disclosure perform window prediction on the panoramic video to obtain multiple candidate windows and importance coefficients for each candidate window. Based on the multiple candidate windows, saliency prediction is performed to obtain a window saliency map. Based on the multiple candidate windows, the importance coefficients of each candidate window, the window saliency map, and a preset bitrate allocation rule for coding tree units, the allocated bitrate of each coding tree unit in the panoramic video is calculated. The panoramic video is then encoded based on the allocated bitrate of each coding tree unit. Compared with the prior art, the embodiments disclosed herein have the following advantages: By performing window prediction on the panoramic video, multiple candidate windows and the importance coefficient of each candidate window are predicted, taking into account the subjective quality of the video, resulting in better subjective quality. Furthermore, based on the multiple candidate windows, saliency prediction is performed to obtain a window saliency map, solving the problem of wasted computing power or bitrate in saliency processing of the entire frame. Based on the multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the window saliency map, and the preset bitrate allocation rules of the coding tree unit, the allocated bitrate of each coding tree unit in the panoramic video is calculated, and the panoramic video is encoded based on the allocated bitrate of each coding tree unit, saving bitrate. It is not limited to online encoding scenarios and can be applied to offline viewing scenarios, making it highly applicable. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a panoramic video encoding method provided in this embodiment of the disclosure; Figure 2A flowchart of a panoramic video encoding method provided in another embodiment of this disclosure; Figure 3 A flowchart of a panoramic video encoding method provided in another embodiment of this disclosure; Figure 4 is a schematic diagram of the panoramic video encoding framework provided in an embodiment of this disclosure; Figure 5 is a schematic diagram of the window prediction process provided in the embodiments of this disclosure; Figure 6 is a schematic diagram of the saliency prediction process provided in the embodiments of this disclosure; Figure 7 is a schematic diagram of the panoramic video encoding device provided in an embodiment of this disclosure; Figure 8 is a schematic diagram of the structure of the electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0020] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0022] In planar video coding, the R-λ method is the most advanced rate control method.

[0023] Existing panoramic video coding schemes are mainly optimized through three methods: 1) Spatial distortion-based optimization schemes: Many panoramic video coding methods take into account the spatial distortion caused by the Equirectangular Projection (ERP) format and propose to modify the application of the R-λ method in panoramic video based on some projection-friendly Peak Signal-to-Noise Ratio (PSNR) weights; 2) Subjective quality enhancement schemes: Some panoramic video coding methods enhance subjective quality by considering the salient regions of video frames; 3) Viewpoint adaptation schemes: Considering that viewers can only see the image within the viewpoint while watching panoramic video, some online panoramic video coding methods propose to allocate more bitrate to the viewpoint area.

[0024] However, the above methods 1) do not take into account the subjective quality of the video, resulting in poor subjective quality; methods 2) that perform saliency processing on the entire frame result in wasted computing power or bitrate; and methods 3) are limited to online encoding scenarios and cannot be directly applied to offline viewing scenarios.

[0025] To address this issue, this disclosure provides a panoramic video encoding method, which will be described below with reference to specific embodiments.

[0026] Figure 1 A flowchart illustrating the panoramic video encoding method provided in this embodiment of the disclosure. Figure 1 As shown, the method includes the following steps: S101. Perform window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window.

[0027] In this step, the electronic device performs window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window. For example... Figure 4 As shown, the original panoramic video frame is used as input, and a convolutional network and a deep learning model (transformer) based on self-attention mechanism are combined to perform window prediction on the panoramic video, predicting multiple candidate windows of the panoramic video and the importance coefficient of each candidate window.

[0028] S102. Based on the multiple candidate windows, perform saliency prediction to obtain a window saliency map.

[0029] After obtaining multiple candidate viewports from the panoramic video, the electronic device performs saliency prediction based on these candidate viewports to obtain a viewport saliency map. For example... Figure 4 As shown, the electronic device uses a residual convolutional network to perform saliency detection inside candidate windows.

[0030] S103. Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, calculate the allocated bitrate of each coding tree unit in the panoramic video.

[0031] In this step, the electronic device calculates the allocated bitrate for each coding tree unit in the panoramic video based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules for coding tree units. For example... Figure 4 As shown, the electronic device obtains the coding weight of each coding tree unit (CTU) based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit. It then calculates the allocated bitrate of each coding tree unit in the panoramic video using the R-λ method, thereby realizing window-adaptive coding tree unit traversal and bitrate allocation.

[0032] S104. Encode the panoramic video based on the allocated bitrate of each coding tree unit.

[0033] After calculating the allocated bitrate for each coding tree unit in the panoramic video, the electronic device encodes the panoramic video according to the allocated bitrate for each coding tree unit. Different bitrates are allocated to different coding tree units, saving bitrate and ensuring the subjective quality of the video within the viewport. Encoding the panoramic video according to the allocated bitrate for each coding tree unit is suitable for offline scenarios.

[0034] This embodiment of the disclosure performs window prediction on the panoramic video to obtain multiple candidate windows and an importance coefficient for each candidate window. Based on the multiple candidate windows, saliency prediction is performed to obtain a window saliency map. Based on the multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the window saliency map, and a preset bitrate allocation rule for coding tree units, the allocated bitrate of each coding tree unit in the panoramic video is calculated. The panoramic video is then encoded based on the allocated bitrate of each coding tree unit. Compared with the prior art, the embodiments disclosed herein have the following advantages: By performing window prediction on the panoramic video, multiple candidate windows and the importance coefficient of each candidate window are predicted, taking into account the subjective quality of the video, resulting in better subjective quality. Furthermore, based on the multiple candidate windows, saliency prediction is performed to obtain a window saliency map, solving the problem of wasted computing power or bitrate in saliency processing of the entire frame. Based on the multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the window saliency map, and the preset bitrate allocation rules of the coding tree unit, the allocated bitrate of each coding tree unit in the panoramic video is calculated, and the panoramic video is encoded based on the allocated bitrate of each coding tree unit, saving bitrate. It is not limited to online encoding scenarios and can be applied to offline viewing scenarios, making it highly applicable.

[0035] Figure 2 A flowchart of a panoramic video encoding method provided in another embodiment of this disclosure is shown below. Figure 2 As shown, the method includes the following steps: S201. For any two consecutive frames of panoramic video, perform convolution feature extraction on the two frames respectively to obtain the spatial features of the two frames.

[0036] like Figure 5 As shown, the network structure for candidate window prediction comprises three modules: a spherical residual module, a bidirectional encoding / decoding module, and a spherical matching module. For any two consecutive frames (tk) and t of a panoramic video, inputting frames (tk) and t simultaneously into the spherical residual module for convolutional feature extraction yields the spatial features of these two frames. and Specifically, the spherical residual module uses a multi-level residual network connection to fuse multi-level features, thereby obtaining the structural information of the panoramic video frame.

[0037] S202. Input the spatial features of the two frames of images into the bidirectional encoding and decoding module to obtain the panoramic features of the two frames of images. The panoramic features of the two frames of images include the intra-frame features of the two frames of images and the inter-frame features of the two frames of images.

[0038] like Figure 5 As shown, and The images are fed into a bidirectional encoding / decoding module for further extraction of panoramic features from a spatiotemporal perspective. In some embodiments, the panoramic features of the two frames include intra-frame features and inter-frame features.

[0039] In some embodiments, S202 may include, but is not limited to, S2021 and S2022: S2021. Input the spatial features of the two frames of images into the encoder in the bidirectional encoding and decoding module to extract intra-frame features, thereby obtaining the intra-frame features of the two frames of images.

[0040] In the bidirectional encoding and decoding module, the encoder is first used to extract intra-frame features to obtain the intra-frame features of the two frames of images.

[0041] S2022. Input the intra-frame features of the two frames into the decoder in the bidirectional encoding and decoding module to extract inter-frame features, thereby obtaining the inter-frame features of the two frames.

[0042] Furthermore, the intra-frame features of the two images are fed into the decoder for inter-frame feature extraction. This allows the information from the two frames to interact with each other in the decoder using a mutual attention mechanism, ultimately yielding the inter-frame features. and .

[0043] S203. Map the panoramic features of the two frames of images into the fully connected layer to obtain the viewports and viewport information of the two frames of images. The viewport information of the two frames of images includes the viewport coordinates of the two frames of images and the movement vector of the viewport between the two frames of images.

[0044] After obtaining the panoramic features of the two frames, the electronic device inputs these features into a fully connected layer for mapping, obtaining the viewports and viewport information for the two frames. The viewport information includes the viewport coordinates and the movement vector of the viewport between the two frames. Specifically, in the spherical matching module of the electronic device, the inter-frame features of the two frames are fed into a fully connected layer, mapping to obtain the viewports and viewport information for the two frames. The viewport information includes the viewport coordinates of the two frames. , and the movement vector of the viewport between two frames. , .

[0045] S204. Filter the windows of the two frames of images respectively to determine multiple candidate windows of the two frames of images in the panoramic video and the matching relationship between the multiple candidate windows of the two frames of images.

[0046] For the viewports of the two frames, the electronic device uses the Hungarian algorithm to filter the viewports of the two frames, and determines multiple candidate viewports of the two frames in the panoramic video and the matching relationship between the multiple candidate viewports of the two frames.

[0047] S205. Input multiple candidate windows into the initial convolution group for feature extraction to obtain the input image features.

[0048] like Figure 6 As shown, the network structure for saliency prediction includes an encoder, a feature pyramid, and a decoder. The encoder includes an initialization module (i.e., an initialization convolutional group), multiple cascaded residual modules, and a convolutional block attention module (CBAM). The electronic device extracts features from multiple candidate windows input into the initialization convolutional group based on the encoder, obtaining the input image features. The initialization convolutional group includes a 7×7 convolutional layer and a max-pooling layer.

[0049] S206. Perform residual feature extraction on the input image features to obtain a multi-level spatial feature map.

[0050] The electronic device performs residual feature extraction on the input image features to obtain a multi-level spatial feature map.

[0051] In some embodiments, S206 may include, but is not limited to, S2061 and S2062: S2061. The input image features are sequentially passed through multiple residual modules connected in series to obtain an initial multi-level spatial feature map.

[0052] The electronic device sequentially passes the input image features through multiple residual modules connected in series, and extracts hierarchical spatial features based on the residual modules to obtain an initial multi-level spatial feature map.

[0053] S2062. The initial multi-level spatial feature map is corrected by the convolutional block attention module to obtain a multi-level spatial feature map.

[0054] A convolutional block attention module is also inserted between every two adjacent residual modules. The initial multi-level spatial feature map is adaptively corrected by the convolutional block attention module to obtain the multi-level spatial feature map.

[0055] S207. Based on the receptive field of the multi-level spatial feature map at different sizes, predict the points of interest to obtain multi-level feature maps.

[0056] Electronic devices utilize low-level, mid-level, and high-level feature information from multi-level spatial feature maps to predict the point of human attention through receptive fields of different sizes. The receptive field is defined as the area of ​​the input image that a pixel on the feature map output by each layer of a convolutional neural network maps back to. In other words, a point on the feature map, relative to the size of the input image, represents the area of ​​the input image that the convolutional neural network's features can see.

[0057] In some embodiments, the step of predicting the points of interest based on the receptive fields of the multi-level spatial feature maps at different sizes to obtain multi-level feature maps includes: inputting the multi-level spatial feature maps into a feature pyramid, wherein each level of the feature pyramid contains an increasing number of channels and a decreasing feature map size, and predicting the points of interest through the feature pyramid to obtain multi-level feature maps.

[0058] like Figure 6 As shown, the feature maps in each layer of the pyramid contain an increasing number of channels (e.g., 64, 128, 256, 512) and a decreasing feature map size (e.g., 1 / 4, 1 / 8, 1 / 16, 1 / 32). The electronic device inputs the multi-level spatial feature maps into the feature pyramid and uses the feature pyramid to predict the points of interest, thus obtaining multi-level feature maps.

[0059] S208. Perform feature size transformation and feature information fusion on the multi-level feature map to obtain a window saliency map.

[0060] After obtaining the multi-level feature maps, the electronic device performs feature size transformation and feature information fusion on the multi-level feature maps to obtain a window saliency map. For example... Figure 6As shown, the electronic device upsamples the multi-level feature maps based on the decoder to restore a consistent feature map size and fuses the multi-level saliency estimates. Features from each layer of the feature pyramid are input into the corresponding decoding module in the decoder. Each decoding module consists of several "upsampling-convolution" units. Each "upsampling-convolution" unit consists of an upsampling layer and a convolutional layer, used to restore the size of the feature map. The feature map of each layer of the feature pyramid contains an increasing number of channels, and each corresponding decoding module in the decoder also contains an increasing number of "upsampling-convolution" units (e.g., 2, 3, 4, 5) to meet the requirements of the multi-level structure. After passing through the multi-level decoding modules, the restored saliency estimates are concatenated and passed through a convolutional layer with a kernel size of 1 and a non-linear soft-thresholding function (softmax) to generate the final output saliency map.

[0061] In some embodiments, the electronic device obtains the output saliency map through a saliency prediction model. The loss function of the model is determined by... The loss of the saliency map is composed of the outputs of the multi-stage decoder:

[0062]

[0063] in and Represent The saliency map of the level prediction and the saliency map generated from the relevant real data. Include and Two parts, respectively representing and The KL (Kullback-Leibler) divergence and CC (correlation coefficient) between them. and Represent and The calculation. Furthermore. and The two hyperparameters are continuously adjusted based on the test results during training. and It is the weighting coefficient.

[0064] S209. Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, calculate the allocated bitrate of each coding tree unit in the panoramic video.

[0065] Specifically, the implementation process and principle of S209 and S103 are the same, and will not be repeated here.

[0066] S210. Encode the panoramic video based on the allocated bitrate of each coding tree unit.

[0067] Specifically, the implementation process and principle of S210 and S104 are the same, and will not be repeated here.

[0068] Compared with the prior art, the embodiments disclosed herein have the following advantages: By performing window prediction on the panoramic video, multiple candidate windows and the importance coefficient of each candidate window are predicted, taking into account the subjective quality of the video, resulting in better subjective quality. Furthermore, based on the multiple candidate windows, saliency prediction is performed to obtain a window saliency map, solving the problem of wasted computing power or bitrate in saliency processing of the entire frame. Based on the multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the window saliency map, and the preset bitrate allocation rules of the coding tree unit, the allocated bitrate of each coding tree unit in the panoramic video is calculated, and the panoramic video is encoded based on the allocated bitrate of each coding tree unit, saving bitrate. It is not limited to online encoding scenarios and can be applied to offline viewing scenarios, making it highly applicable.

[0069] Figure 3 A flowchart of a panoramic video encoding method provided in another embodiment of this disclosure is shown below. Figure 3 As shown, the method includes the following steps: S301. Perform window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window.

[0070] Specifically, the implementation process and principle of S301 and S101 are the same, and will not be repeated here.

[0071] S302. Based on the multiple candidate windows, perform saliency prediction to obtain a window saliency map.

[0072] Specifically, the implementation process and principle of S302 and S102 are the same, and will not be repeated here.

[0073] S303. Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, calculate the weight corresponding to each coding tree unit in the panoramic video.

[0074] In this step, the electronic device calculates the weight corresponding to each coding tree unit in the panoramic video based on multiple candidate viewports, the importance coefficient of each candidate viewport, the saliency map of the viewport, and the preset bitrate allocation rules for coding tree units. Specifically, the electronic device traverses the coding tree units. During the traversal, it adjusts the weight of the coding tree unit based on whether its pixels are in a candidate viewport. The first coding tree unit (CTU) in the Pixel weights are assigned to each pixel. If pixels In the candidate window, pixels will be calculated according to the formula. Assign pixel weights, otherwise leave the pixel unweighted. The pixel weights are set to positive numbers close to 0.

[0075] Obtain pixel weights After that, the The weight of each coding tree unit can be calculated using the following formula:

[0076] S304. Determine the distortion and fixed code rate for each coding tree unit.

[0077] Furthermore, the electronic equipment determines the distortion and fixed bit rate for each coding tree unit. (This is for...) Frame image, at a fixed bitrate The optimization objective for the bitrate allocation is to minimize the distortion of the i-th coding tree unit in the candidate window. .

[0078] S305. Based on the weight corresponding to each coding tree unit, the distortion of each coding tree unit, and the fixed bit rate, calculate the allocated bit rate of each coding tree unit in the panoramic video.

[0079] The electronic device calculates the allocated bitrate for each coding tree unit in the panoramic video based on the weight corresponding to each coding tree unit, the distortion of each coding tree unit, and the fixed bitrate.

[0080] Specifically, the optimization objective can be expressed as follows:

[0081] in and These represent the distortion and allocated bitrate for each CTU, respectively. A hyperbolic model is used here to express the relationship between distortion and bitrate:

[0082] in and These are parameters related to the panoramic video content, updated after each frame is encoded and used in the next frame at the same frame level. Based on CTU weights. The window adaptive distortion is defined as follows:

[0083] according to The above equation can be transformed into an unrestricted optimization problem:

[0084] Differentiating the above equation and setting its derivative to zero, we get:

[0085] Assumption , , So for the first The allocated code rate for each CTU is as follows:

[0086] Based on the allocated bitrate for each frame available:

[0087] Finally, using recursive Taylor expansion (RTE), the allocated code rate for each CTU can be calculated according to the above formula. .

[0088] S306. Encode the panoramic video based on the allocated bitrate of each coding tree unit.

[0089] Specifically, the implementation process and principle of S306 and S104 are the same, and will not be repeated here.

[0090] This embodiment of the disclosure performs window prediction on the panoramic video to obtain multiple candidate windows and an importance coefficient for each candidate window. Based on these candidate windows, saliency prediction is performed to obtain a window saliency map. Further, based on the multiple candidate windows, the importance coefficient of each candidate window, the window saliency map, and a preset bitrate allocation rule for coding tree units, the weight corresponding to each coding tree unit in the panoramic video is calculated, and the distortion and fixed bitrate of each coding tree unit are determined. Then, based on the weight of each coding tree unit, the distortion of each coding tree unit, and the fixed bitrate, the allocated bitrate of each coding tree unit in the panoramic video is calculated, and the panoramic video is encoded based on the allocated bitrate of each coding tree unit. Compared with the prior art, the embodiments disclosed herein have the following advantages: By performing window prediction on the panoramic video, multiple candidate windows and the importance coefficient of each candidate window are predicted, taking into account the subjective quality of the video, resulting in better subjective quality. Furthermore, based on the multiple candidate windows, saliency prediction is performed to obtain a window saliency map, solving the problem of wasted computing power or bitrate in saliency processing of the entire frame. Based on the multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the window saliency map, and the preset bitrate allocation rules of the coding tree unit, the allocated bitrate of each coding tree unit in the panoramic video is calculated, and the panoramic video is encoded based on the allocated bitrate of each coding tree unit, saving bitrate. It is not limited to online encoding scenarios and can be applied to offline viewing scenarios, making it highly applicable.

[0091] Figure 7 This is a schematic diagram of the structure of a panoramic video encoding device provided in an embodiment of this disclosure. The panoramic video encoding device can be an electronic device as described in the above embodiment, or it can be a component or assembly within that electronic device. The panoramic video encoding device provided in this embodiment can execute the processing flow provided in the panoramic video encoding method embodiment, such as... Figure 4 As shown, the panoramic video encoding device 70 includes: a first obtaining module 71, a second obtaining module 72, a calculation module 73, and an encoding module 74; wherein, the first obtaining module 71 is used to perform window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window; the second obtaining module 72 is used to perform saliency prediction based on the multiple candidate windows to obtain a window saliency map; the calculation module 73 is used to calculate the allocated bitrate of each coding tree unit in the panoramic video based on the multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the window saliency map, and the preset bitrate allocation rules of coding tree units; the encoding module 74 is used to encode the panoramic video based on the allocated bitrate of each coding tree unit.

[0092] Optionally, when the first obtaining module 71 performs window prediction on the panoramic video to obtain multiple candidate windows and the importance coefficient of each candidate window, it is specifically used for: for any two consecutive frames of the panoramic video, performing convolutional feature extraction on the two frames respectively to obtain the spatial features of the two frames; inputting the spatial features of the two frames into a bidirectional encoding and decoding module to obtain the panoramic features of the two frames, the panoramic features of the two frames including the intra-frame features and the inter-frame features of the two frames; inputting the panoramic features of the two frames into a fully connected layer for mapping to obtain the windows and window information of the two frames, the window information of the two frames including the window coordinates of the two frames and the movement vector of the window between the two frames; filtering the windows of the two frames respectively to determine the multiple candidate windows of the two frames in the panoramic video and the matching relationship between the multiple candidate windows of the two frames.

[0093] Optionally, when the first obtaining module 71 inputs the spatial features of the two frames of images into the bidirectional encoding and decoding module to obtain the panoramic features of the two frames of images, it is specifically used to: input the spatial features of the two frames of images into the encoder in the bidirectional encoding and decoding module for intra-frame feature extraction to obtain the intra-frame features of the two frames of images; and input the intra-frame features of the two frames of images into the decoder in the bidirectional encoding and decoding module for inter-frame feature extraction to obtain the inter-frame features of the two frames of images.

[0094] Optionally, when the second obtaining module 72 performs saliency prediction based on the multiple candidate windows to obtain a window saliency map, it is specifically used for: inputting the multiple candidate windows into an initialization convolutional group for feature extraction to obtain input image features; performing residual feature extraction on the input image features to obtain a multi-level spatial feature map; predicting points of interest based on the receptive fields of the multi-level spatial feature map at different sizes to obtain a multi-level feature map; and performing feature size transformation and feature information fusion on the multi-level feature map to obtain a window saliency map.

[0095] Optionally, when the second obtaining module 72 performs residual feature extraction on the input image features to obtain a multi-level spatial feature map, it is specifically used to: pass the input image features sequentially through multiple residual modules connected in series to obtain an initial multi-level spatial feature map; and correct the initial multi-level spatial feature map through a convolutional block attention module to obtain a multi-level spatial feature map.

[0096] Optionally, when the second obtaining module 72 predicts the points of interest based on the receptive fields of the multi-level spatial feature maps at different sizes to obtain multi-level feature maps, it is specifically used to: input the multi-level spatial feature maps into a feature pyramid, wherein each level of the feature pyramid contains an increasing number of channels and a decreasing feature map size, and predict the points of interest through the feature pyramid to obtain multi-level feature maps.

[0097] Optionally, when the calculation module 73 calculates the allocated bitrate of each coding tree unit in the panoramic video based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, it is specifically used to: calculate the weight corresponding to each coding tree unit in the panoramic video based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit; determine the distortion and fixed bitrate of each coding tree unit; and calculate the allocated bitrate of each coding tree unit in the panoramic video based on the weight corresponding to each coding tree unit, the distortion and fixed bitrate of each coding tree unit.

[0098] Figure 7 The panoramic video encoding device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0099] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device can be the electronic device described in the above embodiment. The electronic device provided in this embodiment of the disclosure can execute the processing flow provided in the panoramic video encoding method embodiment, such as... Figure 8 As shown, the electronic device 80 includes: a memory 81, a processor 82, a computer program, and a communication interface 83; wherein the computer program is stored in the memory 81 and configured to be executed by the processor 82 using the panoramic video encoding method described above.

[0100] In addition, this disclosure also provides a storage medium storing a computer program thereon, which is executed by a processor to implement the panoramic video encoding method described in the above embodiments.

[0101] Furthermore, this disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the panoramic video encoding method described above.

[0102] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0103] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0104] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0105] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: Window prediction is performed on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window; Based on the multiple candidate windows, a saliency prediction is performed to obtain a window saliency map; Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, the allocated bitrate of each coding tree unit in the panoramic video is calculated. The panoramic video is encoded based on the allocated bitrate of each coding tree unit.

[0106] In addition, the electronic device can also perform other steps in the panoramic video encoding method described above.

[0107] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0110] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0111] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A panoramic video encoding method, characterized in that, include: Window prediction is performed on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window; Based on the multiple candidate windows, a saliency prediction is performed to obtain a window saliency map; Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, the allocated bitrate of each coding tree unit in the panoramic video is calculated. The panoramic video is encoded based on the allocated bitrate of each coding tree unit; The preset code tree unit's code rate allocation rule is to allocate more code rate to regions with high importance and high salience in the candidate window, and less code rate to regions with low importance and low salience in the candidate window.

2. The method according to claim 1, characterized in that, The step of performing window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window includes: For any two consecutive frames of a panoramic video, convolutional feature extraction is performed on the two frames respectively to obtain the spatial features of the two frames; The spatial features of the two frames are input into the bidirectional encoding and decoding module to obtain the panoramic features of the two frames. The panoramic features of the two frames include the intra-frame features and the inter-frame features of the two frames. The panoramic features of the two frames are input to a fully connected layer for mapping to obtain the viewports and viewport information of the two frames. The viewport information of the two frames includes the viewport coordinates of the two frames and the movement vector of the viewport between the two frames. The windows of the two frames are filtered to determine multiple candidate windows of the two frames in the panoramic video and the matching relationship between the multiple candidate windows of the two frames. The bidirectional encoding / decoding module includes two encoders and two decoders. The two encoders share parameters and are used to extract intra-frame features from the spatial features of two frames of images to obtain the intra-frame features of the two frames of images. The two decoders employ a mutual attention mechanism and are used to extract inter-frame features from the intra-frame features of the two frames of images. The intra-frame features of the two frames of images interact with each other in the decoders using the mutual attention mechanism to obtain the inter-frame features of the two frames of images.

3. The method according to claim 2, characterized in that, The step of inputting the spatial features of the two frames of images into the bidirectional encoding / decoding module to obtain the panoramic features of the two frames of images includes: The spatial features of the two frames are input into the encoder in the bidirectional encoding and decoding module for intra-frame feature extraction to obtain the intra-frame features of the two frames. The intra-frame features of the two frames are input into the decoder in the bidirectional encoding and decoding module for inter-frame feature extraction to obtain the inter-frame features of the two frames.

4. The method according to claim 1, characterized in that, The step of performing saliency prediction based on the multiple candidate windows to obtain a window saliency map includes: Multiple candidate windows are input into an initial convolutional group for feature extraction to obtain the input image features; Residual feature extraction is performed on the input image features to obtain a multi-level spatial feature map; Based on the receptive field of the multi-level spatial feature map at different sizes, the focus is predicted to obtain a multi-level feature map. The feature size is transformed and the feature information is fused to obtain the saliency map of the window.

5. The method according to claim 4, characterized in that, The residual feature extraction of the input image features yields a multi-level spatial feature map, including: The input image features are sequentially passed through multiple residual modules in series to obtain an initial multi-level spatial feature map; The initial multi-level spatial feature map is corrected by the convolutional block attention module to obtain a multi-level spatial feature map.

6. The method according to claim 4, characterized in that, The prediction of the point of interest based on the receptive field at different sizes of the multi-level spatial feature map yields a multi-level feature map, including: The multi-level spatial feature map is input into the feature pyramid. Each level of the feature pyramid contains an increasing number of channels and a decreasing feature map size. The feature pyramid is used to predict the points of interest, resulting in a multi-level feature map.

7. The method according to claim 1, characterized in that, The calculation of the allocated bitrate for each coding tree unit in the panoramic video, based on multiple candidate viewports of the panoramic video, the importance coefficient of each candidate viewport, the saliency map of the viewport, and the preset bitrate allocation rules for coding tree units, includes: Based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit, the weight corresponding to each coding tree unit in the panoramic video is calculated. Determine the distortion of each coding tree unit and the fixed bitrate of the panoramic video; Based on the weight corresponding to each coding tree unit, the distortion of each coding tree unit, and the fixed bitrate, the allocated bitrate of each coding tree unit in the panoramic video is calculated.

8. A panoramic video encoding device, characterized in that, include: The first obtaining module is used to perform window prediction on the panoramic video to obtain multiple candidate windows of the panoramic video and the importance coefficient of each candidate window. The second obtaining module is used to perform saliency prediction based on the multiple candidate windows to obtain a window saliency map; The calculation module is used to calculate the allocated bitrate of each coding tree unit in the panoramic video based on multiple candidate windows of the panoramic video, the importance coefficient of each candidate window, the saliency map of the window, and the preset bitrate allocation rules of the coding tree unit. An encoding module is used to encode the panoramic video based on the allocated bitrate of each encoding tree unit; The preset code tree unit's code rate allocation rule is to allocate more code rate to regions with high importance and high salience in the candidate window, and less code rate to regions with low importance and low salience in the candidate window.

9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images