A panoramic video saliency prediction method based on spherical geometry perception and a video compression method

Abstract

The application discloses a panoramic video saliency prediction method and a video compression method based on spherical geometry perception, and comprises the following steps: converting each video frame into a spherical block sequence through an adaptive spherical block projection mechanism, obtaining the projection features of each spherical block, and determining the embedding vector of the spherical block in combination with the position coding of the spherical block; arranging the embedding vectors of all spherical blocks in spatial order to obtain a block embedding feature tensor, extracting spatial features through multiple serially connected spatial Transformer blocks, reshaping the extracted spatial features according to time steps, extracting spatio-temporal features through a lightweight time sequence Transformer, fusing the panoramic frame reconstruction and the panoramic equator prior features to obtain a single-channel saliency map of multiple video frames, integrating the single-channel saliency map of the multiple video frames into a saliency video frame by frame, and generating a continuous weight matrix based on the saliency video to compress the panoramic video.

Description

Technical Field

[0001] This invention relates to the field of multimedia video processing technology, specifically to a panoramic video saliency prediction method and a video compression method based on spherical geometry perception. Background Technology

[0002] With the rapid development of technologies such as Virtual Reality (VR) and Metaverse, panoramic video, with its 360° immersive visual experience, has been widely used in fields such as security monitoring, live streaming, film and television entertainment, and autonomous driving. Visual saliency prediction, as a core foundational technology for panoramic video processing, can simulate the human visual attention mechanism and accurately locate salient areas in the video that attract human attention, making it crucial for solving visual tasks related to panoramic video.

[0003] In terms of saliency prediction, existing methods mostly process panoramic videos directly based on the Equirectangular Projection (ERP) format. However, ERP projection introduces significant geometric distortions when projecting the entire sphere onto a plane, especially in polar regions where it manifests as pixel stretching and semantic distortion, thus affecting model performance. Furthermore, traditional convolutional neural networks have limitations in their ability to model features in distorted regions; and existing models based on Vision Transformer (ViT) typically employ positional encoding based on two-dimensional planar grids, making it difficult to accurately characterize spatial adjacency relationships and global dependencies in spherical scenes.

[0004] In video compression, panoramic videos typically have higher resolution and larger frame sizes, with the data volume of a single panoramic image far exceeding that of traditional 2D videos. However, directly employing a uniform compression strategy can easily lead to significant visual quality degradation in salient areas, while retaining a large amount of redundant data in non-salient areas. Some methods have attempted to introduce saliency information for regional differentiation, but their saliency prediction accuracy is limited, and they lack a unified system-level design, making it difficult to meet the stability and interactivity requirements of practical applications. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a panoramic video saliency prediction method and video compression method based on spherical geometry perception. It can combine spherical geometric characteristics to predict the saliency of panoramic videos and directly use the predicted saliency map to guide the differentiated compression processing of panoramic videos, so as to maintain good subjective visual quality while controlling the data scale.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0007] A panoramic video saliency prediction method based on spherical geometry perception includes the following steps:

[0008] Step S1: Parse the panoramic video in ERP format into continuous video frames, and convert each video frame into a spherical block sequence through an adaptive spherical block projection mechanism;

[0009] Step S2: Obtain the projection features of each spherical block in the spherical block sequence, and determine the embedding vector of the spherical block by combining the position encoding of the spherical block;

[0010] Step S3: Arrange the embedding vectors of all spherical blocks in the multi-video frames according to the spatial order of the spherical blocks to obtain the block embedding feature tensor;

[0011] Step S4: Extract spatial features from the block-embedded feature tensor through multiple concatenated spatial Transformer blocks, reshape the extracted spatial features according to time steps, and extract spatiotemporal features through a lightweight temporal Transformer;

[0012] Step S5: The extracted spatiotemporal features are fused with panoramic equatorial prior features through panoramic frame reconstruction to obtain single-channel saliency maps of multiple video frames. The single-channel saliency maps of multiple video frames are then integrated frame by frame into a saliency video.

[0013] Further, step S1 includes the following sub-steps:

[0014] Step S1.1: Parse the panoramic video in ERP format into continuous video frames, and convert the ERP pixel coordinates of the video frames to spherical latitude and longitude coordinates;

[0015] Step S1.2: Divide the polar angles in the spherical latitude and longitude coordinates into latitude zones with fixed angular intervals;

[0016] Step S1.3: Determine the circumference of the sphere based on the central polar angle of each latitude zone, and adaptively adjust the number of longitude blocks in each latitude zone based on the circumference of the sphere.

[0017]

[0018] in, Indicates the first Number of longitude blocks in each latitude zone Indicates the first The central polar angle of each latitude zone , and They represent the first The upper and lower bounds of the angles for each latitude zone This represents the radian span of each latitudinal zone;

[0019] Step S1.4: Divide the latitude zones according to the number of longitude blocks in each latitude zone to obtain the spherical block sequence.

[0020] Further, step S2 includes the following sub-steps:

[0021] Step S2.1: Map the latitude and longitude coordinates of each spherical block in the spherical block sequence to ERP pixel coordinates, flatten the spherical block into a one-dimensional vector, and obtain the projection features of the spherical block through linear layer projection;

[0022] Step S2.2: Generate a spherical position encoding vector using the center latitude and longitude coordinates of each spherical block:

[0023]

[0024] in, Indicates the first The center latitude and longitude coordinates of each spherical block, where k represents the dimension index of the feature vector. Indicates frequency index, , Represents the dimension of a vector. Indicates the first The angular frequency parameters corresponding to each frequency. , Indicates the first Wavelength parameters corresponding to each frequency , Represents the reference wavelength constant. Indicates the first The spherical position encoding vector of the k-th dimension in a spherical block;

[0025] Step S2.3: Sum the projection features of the spherical blocks and the spherical position encoding vector to obtain the embedding vector of the spherical blocks.

[0026] Furthermore, the process of extracting spatial features for each spatial Transformer block in step S4 is as follows:

[0027]

[0028] in, Indicates the first Spatial features extracted by a spatial Transformer block hour, Represents the block embedding feature tensor; express The query can learn parameter vectors. express The key can be learned parameter vector. express The value of the parameter vector can be learned. Indicates the first The dimension of the attention head in a spatial Transformer block. Represents the normalized exponential function, This indicates the transpose operation.

[0029] Furthermore, the process of extracting spatiotemporal features using a lightweight temporal Transformer in step S4 is as follows:

[0030]

[0031] in, This represents the spatial features extracted by the last spatial Transformer block, reshaped over time steps. express The query can learn parameter vectors. express The key can be learned parameter vector. express The value of the parameter vector can be learned. This represents the dimension of the attention head in a lightweight temporal Transformer. This represents the spatiotemporal features extracted by the lightweight temporal Transformer. Represents the normalized exponential function, This indicates the transpose operation.

[0032] Furthermore, step S5 includes the following sub-steps:

[0033] Step S5.1: Project the extracted spatiotemporal features back to the pixel space through a linear layer. Based on the ERP pixel coordinates of each spherical block, stitch back the complete three-channel video frame through bilinear interpolation. Then, fuse the complete three-channel video frame into a single-channel saliency feature map through a 3×3 convolutional layer.

[0034] Step S5.2: Set the mean and standard deviation of the Gaussian distribution according to the height and width of the single-channel saliency feature map, generate N single-channel Gaussian prior sub-maps using N different standard deviations, and stitch the N single-channel Gaussian prior sub-maps along the channel dimension to obtain the N-channel panoramic equatorial prior feature map.

[0035] Step S5.3: Expand the single-channel saliency feature map to N channels, concatenate it with the N-channel panoramic equatorial prior feature map, and map it to a single-channel feature map through a 1×1 convolutional layer with 1 channel.

[0036] Step S5.4: Normalize the single-channel feature map using the Sigmoid function, and then magnify the normalized single-channel feature map by 255 times to obtain the single-channel saliency map. Integrate the single-channel saliency maps of all video frames into a saliency video.

[0037] Furthermore, the present invention also provides a video compression method based on spherical geometry perception, comprising the following steps:

[0038] Step A1: Perform grayscale normalization on each frame of the saliency prediction video obtained by the panoramic video saliency prediction method based on spherical geometry perception to generate a single-channel binarized ROI mask.

[0039] Step A2: Generate a continuous weight matrix from the inverse matrix of the single-channel binarized ROI mask through Euclidean distance transformation, numerical clipping normalization, and Gaussian smoothing filtering;

[0040] Step A3: Compress the video frames of the panoramic video in ERP format to obtain a compressed panoramic image;

[0041] Step A4: The video frames of the panoramic video in ERP format and the compressed panoramic image are weighted and fused based on a continuous weight matrix to generate a single-frame compressed image.

[0042] Further, step A1 includes the following sub-steps:

[0043] Step A1.1: Perform grayscale normalization on each frame of the saliency video to obtain the saliency feature matrix;

[0044] Step A1.2: Set the saliency threshold and generate an initial single-channel binary mask from the saliency feature matrix;

[0045] Step A1.3: Set the ROI percentage for each frame in the saliency video, and calculate the number of target pixels in each frame based on the size of each frame;

[0046] Step A1.4: Construct an elliptical morphological operator and perform iterative morphological dilation on the single-channel binary mask of each frame until the number of pixels in the single-channel binary mask of each frame reaches the target number of pixels, thus obtaining a single-channel binary ROI mask.

[0047] Furthermore, step A3 includes the following sub-steps:

[0048] Step A3.1: Filter and downsample the video frames of the panoramic video in ERP format;

[0049] Step A3.2: Set the compression quality factor, perform lossy encoding compression on the downsampled video frames, and generate compressed frames;

[0050] Step A3.3: Upsample the compressed frame to restore it to the original size of the video frame to obtain the compressed panoramic image.

[0051] Furthermore, the process of generating a single-frame compressed image in step A4 is as follows:

[0052]

[0053] in, For single-frame compressed images, The video frames are in ERP format panoramic video. For compressed panoramic images, is a continuous weight matrix, and ⊙ is the element-wise multiplication operator.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The panoramic video saliency prediction method based on spherical geometry perception of this invention effectively alleviates the geometric distortion of ERP format video frames in polar regions by introducing an adaptive spherical block projection mechanism, thereby improving the accuracy of feature extraction for each spherical block in the spherical block sequence. Combined with the position encoding of spherical blocks, it can accurately perceive the absolute position and spatial adjacency of spherical blocks, overcoming the limitations of traditional planar position encoding. By adopting an architecture combining spatial Transformer blocks and lightweight temporal Transformers, it efficiently captures the spatiotemporal features in panoramic videos and integrates panoramic equatorial prior features, further improving the consistency of human perception of saliency videos. The panoramic video saliency prediction method of this invention can generate high-precision and highly robust panoramic video saliency maps.

[0056] (2) The video compression method based on spherical geometry perception of the present invention constructs a smooth continuous weight matrix by using the generated salient video to guide the original panoramic video frame and the single frame compressed image after global compression to perform pixel-level soft weighted fusion. This not only preserves the original high-fidelity image quality of the salient area, but also effectively reduces the information redundancy and coding rate of the non-salient area, and realizes intelligent allocation of the code rate. At the same time, through the soft fusion strategy based on the whole image, it ensures a seamless and smooth transition between the high-quality salient area and the low-quality background area, and completely eliminates the boundary hard damage and visual artifacts that may be generated by traditional partition compression. Thus, it saves a lot of transmission bandwidth while ensuring the global visual consistency of the panoramic video and an excellent subjective viewing experience. Attached Figure Description

[0057] Figure 1 This is a flowchart of the panoramic video saliency prediction method based on spherical geometry perception according to the present invention;

[0058] Figure 2This is a schematic diagram of the adaptive spherical block projection mechanism in this invention;

[0059] Figure 3 This is a schematic diagram of spatiotemporal feature extraction in this invention;

[0060] Figure 4 This is a schematic diagram illustrating the fusion of extracted spatiotemporal features with panoramic equatorial prior features in this invention through panoramic frame reconstruction.

[0061] Figure 5 The flowchart of the video compression method based on spherical geometry perception of the present invention. Detailed Implementation

[0062] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0063] like Figure 1 The flowchart below shows the panoramic video saliency prediction method based on spherical geometry perception according to the present invention. The panoramic video saliency prediction method includes the following steps:

[0064] Step S1: The panoramic video in ERP format is parsed into continuous video frames. To overcome the severe geometric stretching distortion of ERP projection in high-latitude regions, each video frame is converted into a low-distortion spherical block sequence through an adaptive spherical block projection mechanism. In this invention, the adaptive spherical block projection mechanism adjusts the density of the spherical blocks according to latitude, achieving approximately uniform sampling of the sphere and significantly reducing distortion interference. Figure 2 This includes the following sub-steps:

[0065] Step S1.1: Parse the panoramic video in ERP format into consecutive video frames, with the size of a single video frame being [size missing]. ERP pixel coordinates of video frames Convert to spherical latitude and longitude coordinates :

[0066]

[0067]

[0068] in, It is the azimuth angle. Polar angle, The height of the video frame. The width of the video frame. For the RGB three channels of the video frame.

[0069] Step S1.2: Based on the geometric properties of the sphere, determine the polar angle in the spherical latitude and longitude coordinates. Latitude zones are divided into fixed angular intervals, where the fixed angular intervals are... The arc intervals corresponding to the latitude zones For the t-th latitude zone, its central polar angle is... , and They represent the first The upper and lower bounds of the angles for each latitude zone.

[0070] Step S1.3: Determine the circumference of the sphere based on the central polar angle of each latitude zone, and adaptively adjust the number of longitude blocks in each latitude zone based on the circumference of the sphere.

[0071]

[0072] in, Indicates the first Number of longitude blocks in each latitude zone For the first The circumference of the sphere for each latitude zone is set to a minimum of 4 for the number of longitude blocks in each latitude zone to avoid insufficient blocks near the poles.

[0073] Step S1.4: Divide the latitude zones according to the number of longitude blocks in each latitude zone to obtain a spherical block sequence. To meet the needs of subsequent feature extraction, each spherical block needs to be converted into a planar image of a fixed size. This invention adopts a local reprojection strategy based on ERP coordinates, which reprojects the spherical blocks onto the corresponding regions of the ERP image through a local projection function: Calculate the corresponding pixel coordinate range based on the latitude and longitude boundaries of the spherical blocks. When the longitude range crosses the image boundary, such as At that time, blocks are extracted by splicing partial regions from the left and right sides of the image. To adapt to the input requirements of feature extraction, bilinear interpolation is used to uniformly adjust the spherical blocks to a 16×16 planar image, thereby making the dimension of the spherical block sequence [missing information]. ,in, For batch size, This represents the total number of blocks in the spherical surface.

[0074] Step S2: Obtain the projection features of each spherical block in the spherical block sequence, and determine the embedding vector of the spherical block by combining the position encoding of the spherical block. The position encoding of the spherical block enables 3D spherical spatial perception, allowing the embedding vector of the spherical block to accurately capture cross-boundary objects and global geometric structure features. For example... Figure 3 This includes the following sub-steps:

[0075] Step S2.1: Each of the spherical block sequence The spherical blocks are flattened into 768-dimensional vectors, and then projected onto a feature dimension of D=768 through a linear layer to obtain the projected features of the spherical blocks. ;

[0076] Step S2.2: To address the issue that traditional location coding cannot reflect spherical geometric relationships, a spherical location coding system is designed. A spherical location coding vector is generated based on the center latitude and longitude coordinates of each spherical block.

[0077]

[0078] in, Indicates the first The center latitude and longitude coordinates of each spherical block, k represents the dimension index of the feature vector, and the value of k ranges from 1 to 2. , Indicates frequency index, , Represents the dimension of a vector. Indicates the first The angular frequency parameters corresponding to each frequency. , Indicates the first Wavelength parameters corresponding to each frequency , Represents the reference wavelength constant. Indicates the first The spherical position encoding vector in the k-th dimension of a spherical block. By representing the absolute coordinates of the sphere using sine and cosine functions with increasing frequency, the geometric positions of the spherical blocks on the sphere can be perceived. Let represent a D-dimensional real vector space.

[0079] Step S2.3: Sum the projection features of the spherical blocks and the spherical position encoding vector to obtain the embedding vector of the spherical blocks. .

[0080] Step S3: Arrange the embedding vectors of all spherical blocks in the multi-video frames according to the spatial order of the spherical blocks to obtain the block embedding feature tensor. .

[0081] Step S4: The block-embedded feature tensor captures the global spatial dependencies between spherical blocks through multiple concatenated spatial Transformer blocks, extracting spatial features. The extracted spatial features fuse spherical geometric location information and image semantic features; the extracted spatial features are then reshaped according to time steps. Spatiotemporal features are extracted using a lightweight temporal Transformer, where For time-series batch size, The number of consecutive frames is represented by this two-stage architecture with spatiotemporal separation. This architecture can model the global spatial dependence of panoramic video and the motion changes between consecutive frames separately, thereby efficiently decoupling spatiotemporal features. While making full use of video spatiotemporal information to improve prediction accuracy, it effectively reduces computational complexity.

[0082] The process of extracting spatial features for each spatial Transformer block in this invention is as follows:

[0083]

[0084] in, Indicates the first Spatial features extracted by a spatial Transformer block hour, Represents the block embedding feature tensor ; express The query can learn parameter vectors. express The key can be learned parameter vector. express The value of the parameter vector can be learned. Indicates the first The dimension of the attention head in each spatial Transformer block, k=8. Represents the normalized exponential function, This indicates the transpose operation.

[0085] The process of extracting spatiotemporal features using a lightweight temporal Transformer in this invention is as follows:

[0086]

[0087] in, This represents the spatial features extracted by the last spatial Transformer block, reshaped over time steps. express The query can learn parameter vectors. express The key can be learned parameter vector. express The value of the parameter vector can be learned. This represents the dimension of the attention head in a lightweight temporal Transformer. This represents the spatiotemporal features extracted by the lightweight temporal Transformer. Represents the normalized exponential function, This indicates the transpose operation.

[0088] Step S5: Since users tend to focus their attention on the equatorial region when watching panoramic videos, a panoramic equatorial Gaussian prior constraint is introduced. The extracted spatiotemporal features are fused with the panoramic equatorial prior features after panoramic frame reconstruction to obtain single-channel saliency maps for multiple video frames. These single-channel saliency maps are then integrated frame by frame into a salient video, effectively suppressing noise interference in non-interested areas and making the predicted salient video more consistent with human visual perception habits. Figure 4 This includes the following sub-steps:

[0089] Step S5.1: Project the extracted spatiotemporal features back to the B×M×3×16×16 pixel space through a linear layer. Based on the ERP pixel coordinates of each spherical block, restore the original block size through bilinear interpolation, stitch back the complete three-channel video frame, and fuse the complete three-channel video frame into a single-channel saliency feature map through a 3×3 convolutional layer.

[0090] Step S5.2: Set the mean and standard deviation of the Gaussian distribution according to the height and width of the single-channel saliency feature map, generate N single-channel Gaussian prior sub-maps using N different standard deviations, and stitch the N single-channel Gaussian prior sub-maps along the channel dimension to obtain the N-channel panoramic equatorial prior feature map.

[0091] In this invention, the panoramic equatorial Gaussian prior is a panoramic equatorial prior map that exhibits a Gaussian distribution along the y-axis and is fully filled along the x-axis, wherein the y-axis is Gaussian distributed... The mathematical form is:

[0092]

[0093] and Let represent the mean and standard deviation of the Gaussian distribution, respectively. , n is an integer and satisfies , The vertical coordinates representing the spatiotemporal characteristics.

[0094] Step S5.3: Expand the single-channel saliency feature map to N channels, concatenate it with the N-channel panoramic equatorial prior feature map, and map it to a single-channel feature map through a 1×1 convolutional layer with 1 channel.

[0095] Step S5.4: Normalize the single-channel feature map using the Sigmoid function, then enlarge the normalized single-channel feature map by 255 times and convert it to uint8 integer to obtain the single-channel saliency map. Integrate the single-channel saliency maps of all video frames into a saliency video.

[0096] In one technical solution of the present invention, a video compression method based on spherical geometry perception is also provided, such as... Figure 5It includes the following steps:

[0097] Step A1: Each frame in the salient video obtained by the panoramic video saliency prediction method based on spherical geometry perception is normalized to grayscale, generating a single-channel binary ROI mask. This transforms continuous saliency probabilities into clear region divisions, achieving pixel-level accurate segmentation of salient foreground and non-salient background, laying the foundation for subsequent differential compression. This includes the following sub-steps:

[0098] Step A1.1: Extract each frame from the saliency video. Gray-level normalization is performed to convert the data into a floating-point numerical range of [0.0, 1.0], resulting in the saliency feature matrix. salient feature matrix In this context, a larger pixel value indicates a higher visual salience at the corresponding location.

[0099] Step A1.2: Set the significance threshold , The adjustable range is 0.0~0.9, which generates the initial single-channel binary mask from the saliency feature matrix. :

[0100]

[0101] In this context, a mask value of 1 indicates that the pixel is a highly salient area, which is a visually sensitive area; a mask value of 0 indicates that the pixel is a non-salient area, which is a visually insensitive area.

[0102] Step A1.3: Set the ROI percentage for each frame in the saliency video. , The adjustable range is 0.0~0.8, designed to flexibly control the proportion of significant areas in the image, so as to find the best balance between visual quality and bitrate when bandwidth is limited, and to calculate the number of target pixels in each frame in combination with the size of each frame. :

[0103]

[0104] in, This indicates the floor function.

[0105] Step A1.4: To ensure the connectivity and integrity of the mask, and to avoid excessive mask expansion leading to compression failure, a mathematical morphological dilation operation using elliptical structuring elements is used to optimize the initial mask, with added boundary protection constraints: An elliptical morphological operator of size K=(15,15) is constructed, and iterative morphological dilation is performed on the single-channel binary mask of each frame:

[0106]

[0107] In the formula, M(0) = Let M(k) be the initial single-channel binary mask, and M(k) be the single-channel binary ROI mask after the k-th iteration expansion. For mathematical morphology binary dilation operators;

[0108] After each iteration of dilation, count the total number of pixels with a value of 1 in the single-channel binary ROI mask, and repeat step A1.4 until the number of pixels in the single-channel binary mask of each frame reaches the target number of pixels. This yields a single-channel binary ROI mask.

[0109] Step A2: To eliminate hard boundary transition artifacts between visually sensitive and non-visually sensitive areas and achieve smooth transitions between regions, the inverse matrix of the single-channel binarized ROI mask is transformed using Euclidean distance, numerical clipping normalization, and Gaussian smoothing filtering to generate a continuous weight matrix; specifically, this includes the following sub-steps:

[0110] Step A2.1: Convert the inverse matrix of the single-channel binarized ROI mask Perform a Euclidean distance transformation, calculate the Euclidean distance from each pixel in the non-salient region of mask value 0 to the nearest highly salient region of mask value 1, and generate a distance matrix. :

[0111]

[0112] Where (x, y) are the two-dimensional coordinates of the image pixels. This represents the pixel value of the single-channel binarized ROI mask at coordinates (x, y). It is a Euclidean distance transformation operator. Distance matrix The pixel value at coordinates (x, y), distance matrix In this context, a larger pixel value indicates that the location is farther away from the highly salient region, and the compression intensity can be higher.

[0113] Step A2.2: By performing numerical truncation and normalization operations, the values ​​of the distance matrix are constrained to the [0,1] interval to avoid numerical overflow.

[0114]

[0115] in, The pixel values ​​of the normalized distance matrix. This is the distance transformation transition range coefficient. To find the maximum value operator, To achieve the minimum value operator, the two are combined to implement the truncation constraint of the numerical interval [0,1].

[0116] Step A2.3: Map the normalized distance matrix to a soft fusion weight matrix, and the weight values ​​of the soft fusion weight matrix are... Positively correlated with the saliency of pixels, the soft fusion weight matrix is ​​used to define the fusion ratio of the original panoramic video frame and the low-quality background reference image: the weight value approaches 1 in the core of the saliency region to preserve the original image quality, approaches 0 in the non-saliency region to adopt compressed image quality, and smoothly decays between the two, thereby achieving a seamless gradient transition from high-definition foreground to compressed background.

[0117] Step A2.4: Further optimize the soft fusion weight matrix using Gaussian filtering. A gradual transition effect is achieved, avoiding abrupt, step-like changes. The Gaussian blur radius is set to r, and the build size is... Gaussian filter kernel Generate a continuous weight matrix, where This is a Gaussian smoothing filter operator. Finally, the single-channel continuous weight matrix is ​​expanded into a three-channel weight matrix. , where [:,:,None] is the matrix dimension expansion operation, and the dimension is kept consistent with the original RGB three-channel video frame as H×W×3, which is used for subsequent pixel-level soft weighted fusion operations.

[0118] Step A3: Compress the video frames of the ERP format panoramic video to obtain a compressed panoramic image; this includes the following sub-steps:

[0119] Step A3.1: Filter and downsample the video frames of the panoramic video in ERP format; specifically, perform a Gaussian low-pass filter on the video frames of the panoramic video in ERP format, with the standard deviation of the filter kernel... This is used to filter high-frequency details and avoid aliasing and jagged edges after downsampling. A downsampling ratio *s* is set, and the filtered video is downsampled proportionally. The size of the downsampled image is H / / s × W / / s. Region interpolation is used to ensure a smooth, jagged-free image after downsampling. The purpose of downsampling here is: when the downsampling ratio is... When compression is performed, the size can be reduced to shift the compression to a lower resolution domain, which can significantly reduce the base of the compressed data and thus improve compression efficiency, while achieving greater data compression. Here, / / is the floor division operator.

[0120] Step A3.2: Set the compression quality factor Furthermore, a hard lower limit constraint is added to the compressibility mass coefficient: To avoid blockiness and color distortion caused by low quality, the downsampled video frames are lossily encoded and compressed to generate compressed frames.

[0121] Step A3.3: Upsample the compressed frame using bicubic interpolation to restore it to the original size of the video frame, thus obtaining the compressed panoramic image.

[0122] Step A4: The video frames of the panoramic video in ERP format and the compressed panoramic image are weighted and fused based on a continuous weight matrix to generate a single-frame compressed image. The panoramic video frames in ERP format are responsible for providing lossless image quality details in highly salient areas, while the compressed panoramic image is responsible for providing low bit rate pixel filling in non-salient areas. This fusion method utilizes the continuous gradation characteristics of the continuous weight matrix to effectively eliminate visual abrupt changes at regional boundaries and achieve a seamless and smooth transition in image quality.

[0123] The process of generating a single-frame compressed image in this invention is as follows:

[0124]

[0125] in, For single-frame compressed images, The video frames are in ERP format panoramic video. For compressed panoramic images, is a continuous weight matrix, and ⊙ is the element-wise multiplication operator.

[0126] In one technical solution of the present invention, to facilitate users' intuitive viewing of the location of highly salient regions, the original image is visually marked: pixel regions with a value of 1 in the single-channel binarized ROI mask are subjected to pixel value weighted fusion.

[0127]

[0128] in This is a matrix of pixel regions with a mask value of 1 in a panoramic video frame in ERP format. This represents the RGB red constant. The rounding operation results in a red-highlighted ROI visualization. The red area represents the highly salient core region that needs to be preserved without loss.

[0129] The panoramic video saliency prediction method of this invention can generate high-precision and robust panoramic video saliency maps. Secondly, the video compression method based on spherical geometry perception of this invention utilizes the generated saliency video to construct a smooth, continuous weight matrix, guiding pixel-level soft-weighted fusion of the original panoramic video frames and globally compressed single-frame images. This not only preserves the original high-fidelity image quality of salient areas but also effectively reduces information redundancy and coding rate in non-salient areas, achieving intelligent bitrate allocation. Furthermore, through a soft fusion strategy based on the entire image, it ensures a seamless and smooth transition between high-quality salient areas and low-quality background areas, completely eliminating boundary artifacts and visual artifacts that may occur with traditional partitioned compression. Thus, it significantly saves transmission bandwidth while ensuring global visual consistency and an excellent subjective viewing experience for the panoramic video.

[0130] The panoramic video saliency prediction method and video compression method based on spherical geometry perception of this invention can be transferred to a variety of panoramic video-related scenarios, such as low-bandwidth transmission of panoramic video live streaming, efficient storage of panoramic security videos, and immersive playback optimization of VR videos. It has a wide range of applications and significant market value.

[0131] In one technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program that enables a computer to execute the panoramic video saliency prediction method and video compression method based on spherical geometry perception of the present invention.

[0132] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the panoramic video saliency prediction method and video compression method based on spherical geometry perception of the present invention.

[0133] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, and portable compact disc read-only memory (CD). ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A panoramic video saliency prediction method based on spherical geometry perception, characterized in that, Includes the following steps: Step S1: Parse the panoramic video in ERP format into continuous video frames, and convert each video frame into a spherical block sequence through an adaptive spherical block projection mechanism, including the following sub-steps: Step S1.1: Parse the panoramic video in ERP format into continuous video frames, and convert the ERP pixel coordinates of the video frames to spherical latitude and longitude coordinates; Step S1.2: Divide the polar angles in the spherical latitude and longitude coordinates into latitude zones with fixed angular intervals; Step S1.3: Determine the circumference of the sphere based on the central polar angle of each latitude zone, and adaptively adjust the number of longitude blocks in each latitude zone based on the circumference of the sphere. in, Indicates the first Number of longitude blocks in each latitude zone Indicates the first The central polar angle of each latitude zone , and They represent the first The upper and lower bounds of the angles for each latitude zone This represents the radian span of each latitudinal zone; Step S1.4: Divide the latitude zones according to the number of longitude blocks in each latitude zone to obtain the spherical block sequence; Step S2: Obtain the projection features of each spherical block in the spherical block sequence, and determine the embedding vector of the spherical block by combining the position encoding of the spherical block, including the following sub-steps: Step S2.1: Map the latitude and longitude coordinates of each spherical block in the spherical block sequence to ERP pixel coordinates, flatten the spherical block into a one-dimensional vector, and obtain the projection features of the spherical block through linear layer projection; Step S2.2: Generate a spherical position encoding vector using the center latitude and longitude coordinates of each spherical block: in, Indicates the first The center latitude and longitude coordinates of each spherical block, where k represents the dimension index of the feature vector. Indicates frequency index, , Represents the dimension of a vector. Indicates the first The angular frequency parameters corresponding to each frequency. , Indicates the first Wavelength parameters corresponding to each frequency , Represents the reference wavelength constant. Indicates the first The spherical position encoding vector of the k-th dimension in a spherical block; Step S2.3: Sum the projection features of the spherical blocks and the spherical position encoding vector to obtain the embedding vector of the spherical blocks; Step S3: Arrange the embedding vectors of all spherical blocks in the multi-video frames according to the spatial order of the spherical blocks to obtain the block embedding feature tensor; Step S4: Extract spatial features from the block-embedded feature tensor through multiple concatenated spatial Transformer blocks, reshape the extracted spatial features according to time steps, and extract spatiotemporal features through a lightweight temporal Transformer; Step S5: The extracted spatiotemporal features are fused with panoramic equatorial prior features through panoramic frame reconstruction to obtain single-channel saliency maps of multiple video frames. The single-channel saliency maps of multiple video frames are then integrated frame by frame into a saliency video, including the following sub-steps: Step S5.1: Project the extracted spatiotemporal features back to the pixel space through a linear layer. Based on the ERP pixel coordinates of each spherical block, stitch back the complete three-channel video frame through bilinear interpolation. Then, fuse the complete three-channel video frame into a single-channel saliency feature map through a 3×3 convolutional layer. Step S5.2: Set the mean and standard deviation of the Gaussian distribution according to the height and width of the single-channel saliency feature map, generate N single-channel Gaussian prior sub-maps using N different standard deviations, and stitch the N single-channel Gaussian prior sub-maps along the channel dimension to obtain the N-channel panoramic equatorial prior feature map. Step S5.3: Expand the single-channel saliency feature map to N channels, concatenate it with the N-channel panoramic equatorial prior feature map, and map it to a single-channel feature map through a 1×1 convolutional layer with 1 channel. Step S5.4: Normalize the single-channel feature map using the Sigmoid function, and then magnify the normalized single-channel feature map by 255 times to obtain the single-channel saliency map. Integrate the single-channel saliency maps of all video frames into a saliency video.

2. The panoramic video saliency prediction method based on spherical geometry perception according to claim 1, characterized in that, The process of extracting spatial features for each spatial Transformer block in step S4 is as follows: in, Indicates the first Spatial features extracted by a spatial Transformer block hour, Represents the block embedding feature tensor; express The query can learn parameter vectors. express The key can be learned parameter vector. express The value of the parameter vector can be learned. Indicates the first The dimension of the attention head in a spatial Transformer block. Represents the normalized exponential function, This indicates the transpose operation.

3. The panoramic video saliency prediction method based on spherical geometry perception according to claim 1, characterized in that, The process of extracting spatiotemporal features using a lightweight temporal Transformer in step S4 is as follows: in, This represents the spatial features extracted by the last spatial Transformer block, reshaped over time steps. express The query can learn parameter vectors. express The key can be learned parameter vector. express The value of the parameter vector can be learned. This represents the dimension of the attention head in a lightweight temporal Transformer. This represents the spatiotemporal features extracted by the lightweight temporal Transformer. Represents the normalized exponential function, This indicates the transpose operation.

4. A video compression method based on spherical geometry perception, characterized in that, Includes the following steps: Step A1: Perform grayscale normalization on each frame of the saliency prediction video obtained by the panoramic video saliency prediction method based on spherical geometry perception as described in any one of claims 1-3 to generate a single-channel binarized ROI mask. Step A2: Generate a continuous weight matrix from the inverse matrix of the single-channel binarized ROI mask through Euclidean distance transformation, numerical clipping normalization, and Gaussian smoothing filtering; Step A3: Compress the video frames of the panoramic video in ERP format to obtain a compressed panoramic image; Step A4: The video frames of the panoramic video in ERP format and the compressed panoramic image are weighted and fused based on a continuous weight matrix to generate a single-frame compressed image.

5. The video compression method based on spherical geometry perception according to claim 4, characterized in that, Step A1 includes the following sub-steps: Step A1.1: Perform grayscale normalization on each frame of the saliency video to obtain the saliency feature matrix; Step A1.2: Set the saliency threshold and generate an initial single-channel binary mask from the saliency feature matrix; Step A1.3: Set the ROI percentage for each frame in the saliency video, and calculate the number of target pixels in each frame based on the size of each frame; Step A1.4: Construct an elliptical morphological operator and perform iterative morphological dilation on the single-channel binary mask of each frame until the number of pixels in the single-channel binary mask of each frame reaches the target number of pixels, thus obtaining a single-channel binary ROI mask.

6. The video compression method based on spherical geometry perception according to claim 4, characterized in that, Step A3 includes the following sub-steps: Step A3.1: Filter and downsample the video frames of the panoramic video in ERP format; Step A3.2: Set the compression quality factor, perform lossy encoding compression on the downsampled video frames, and generate compressed frames; Step A3.3: Upsample the compressed frame to restore it to the original size of the video frame to obtain the compressed panoramic image.

7. The video compression method based on spherical geometry perception according to claim 4, characterized in that, The process of generating a single frame compressed image in step A4 is as follows: in, For single-frame compressed images, The video frames are in ERP format panoramic video. For compressed panoramic images, is a continuous weight matrix, and ⊙ is the element-wise multiplication operator.

Images