Binocular panoramic video compression encoding method and electronic device

Abstract

The application relates to the technical field of panoramic video coding, and discloses a binocular panoramic video compression coding method and electronic equipment. In the application, binocular panoramic video to be coded and gaze direction information are acquired, a spherical surface sampling weight is determined according to the spherical surface direction of a space unit, and a gaze weight and a joint weight are determined in combination with the gaze direction information; a basic quantization parameter of the space unit of left eye video and right eye video is determined according to the joint weight and the left eye video and the right eye video are respectively coded, left eye compression distortion and cross-eye distortion difference are determined based on a coding reconstruction result; a current quantization parameter is determined according to a comparison result of the cross-eye distortion difference and a just-noticeable difference threshold value, and recoding is performed; a spherical surface global distortion constraint, a cross-eye consistency constraint and a gaze area distortion constraint are checked on a coding result, and when at least one constraint does not meet a condition, a quantization parameter of a corresponding space unit is adjusted and recoding is performed again, so that the spherical surface overall quality, binocular consistency and gaze area quality are considered.

Description

Technical Field

[0001] This application relates to the field of panoramic video coding technology, and in particular to a binocular panoramic video compression coding method and electronic device. Background Technology

[0002] Extended Reality (XR) binocular panoramic videos typically consist of stereoscopic visual content composed of panoramic images from both the left and right eyes. They often employ equi-rectangular projection (ERP) to unfold a 360° spherical scene into a two-dimensional image, facilitating compression, storage, and transmission using video encoding standards such as H.265 / H.266. With the increasing application of 5K / 8K resolution video in head-mounted displays and immersive presentations, balancing compression efficiency with viewing quality within limited bitrates has become a crucial challenge.

[0003] Most existing encoding methods follow the processing approach of planar video, applying approximately uniform distortion evaluation and bitrate allocation to different regions of the ERP (Optical Performance Frame) display, without fully considering the differences between the two-dimensional projection position and the actual sampling characteristics of the sphere. Furthermore, binocular video is typically encoded separately, potentially leading to a lack of coordination in compression distortion across the left and right eye regions. Additionally, the user's actual viewing area changes with head movement, making fixed or static region-priority methods ill-suited to adapt to this change. Therefore, at higher compression ratios, issues such as uneven bitrate utilization, localized image quality degradation, and binocular viewing discomfort can easily arise. Summary of the Invention

[0004] The purpose of this application is to provide a binocular panoramic video compression encoding method and electronic device, which aims to improve the rationality of encoding resource allocation and compression efficiency, reduce the user-perceptible binocular distortion difference and the resulting viewing discomfort, and maintain the encoding quality of the user's focus area while controlling the overall bit rate.

[0005] To address the aforementioned technical problems, this application provides a binocular panoramic video compression coding method, comprising: acquiring a binocular panoramic video to be encoded and corresponding gaze direction information, wherein the binocular panoramic video includes a left-eye video and a right-eye video; for spatial units in the binocular panoramic video, determining spherical sampling weights based on the spherical orientation corresponding to the spatial unit, determining gaze weights based on the spherical orientation corresponding to the spatial unit and the gaze direction information, and determining a joint weight of the spatial unit based on the spherical sampling weights and the gaze weights; determining basic quantization parameters of the spatial units of the left-eye video and the right-eye video respectively based on the joint weights; encoding the left-eye video and the right-eye video respectively based on the basic quantization parameters; and determining the compression loss of the spatial units of the left-eye video based on the encoding reconstruction results of the spatial units of the left-eye video and the right-eye video. The system determines the compression distortion of the spatial unit in the right-eye video and the interocular distortion difference between them; it determines the just-perceptible difference threshold corresponding to the spatial unit in the left-eye video and the spatial unit in the right-eye video, and determines the current quantization parameters of the spatial unit in the left-eye video and the spatial unit in the right-eye video based on the comparison result of the interocular distortion difference and the just-perceptible difference threshold; it performs constraint verification on the encoding result obtained by encoding according to the current quantization parameters, the constraint verification including: spherical global distortion constraint determined based on the spherical sampling weight, interocular consistency constraint for the spatial unit in the corresponding left-eye video and the spatial unit in the right-eye video, and gaze region distortion constraint determined based on the gaze direction information; when at least one of the constraints does not meet the corresponding constraint condition, it adjusts the quantization parameters of the corresponding spatial unit according to the unmet constraint, and encodes again based on the adjusted quantization parameters.

[0006] This application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the binocular panoramic video compression encoding method as described above.

[0007] To address the issues in existing binocular panoramic video encoding processes, such as insufficient consideration of the differences between the 2D projection area and the actual spherical sampling characteristics, leading to mismatches between bitrate allocation in different areas and actual user perception needs, lack of coordination in compression distortion of corresponding areas when encoding left and right eye videos separately, and difficulty in balancing fixed or static area-priority encoding methods with varying viewing areas due to changes in gaze direction, this application addresses these problems. It involves acquiring the binocular panoramic video to be encoded and its corresponding gaze direction information, determining spherical sampling weights and gaze weights for each spatial unit, and forming a joint weight. By determining the basic quantization parameters of the spatial units in the left and right eye videos based on the joint weight, the left and right eye videos are encoded separately. Based on the encoding reconstruction results, the compression distortion and cross-eye distortion of the corresponding spatial units for the left and right eyes are determined. The method involves determining a perceptible difference threshold and comparing the cross-eye distortion difference with this threshold to determine the current quantization parameters of the spatial units in the left and right eye videos, thereby coordinating the encoding quality of the corresponding regions of the left and right eyes. Furthermore, this application verifies the encoding results by applying spherical global distortion constraints, cross-eye consistency constraints, and gaze region distortion constraints. When at least one constraint fails to meet its corresponding condition, the quantization parameters of the corresponding spatial units are adjusted and re-encoded. This solves, or at least partially solves, the problems of unreasonable bitrate allocation in spherical space, excessive distortion differences between the left and right eye regions, and difficulty in simultaneously ensuring the image quality of the gaze region. This at least partially improves the rationality of encoding resource allocation and compression efficiency, reduces user-perceptible binocular distortion differences and the resulting viewing discomfort, and maintains the encoding quality of the user-focused region while controlling the overall bitrate. Attached Figure Description

[0008] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0009] Figure 1 This is the overall architecture of binocular panoramic video compression coding involved in this application; Figure 2 This is a flowchart of a binocular panoramic video compression coding method provided in this application; Figure 3 This is another flowchart of the binocular panoramic video compression coding method involved in this application; Figure 4 This is a flowchart of the joint weighting of head movement prediction and fixation point classification involved in this application; Figure 5 This is a flowchart of the binocular just-perceptible difference consistency constraint and cross-eye bitrate adjustment involved in this application; Figure 6 This is a flowchart of the multi-objective constrained closed-loop quantitative control involved in this application; Figure 7 This is a schematic diagram of the structure of the electronic device shown in this application. Detailed Implementation

[0010] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, the technical solutions or means disclosed in this specification can be applied to other scenarios based on these technical contents without creative effort. It should be understood that the terms "system," "apparatus," "unit," and / or "module" used in this specification are used to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other words can achieve the same purpose, they can be replaced by other expressions. Unless otherwise specified, the technical terms describing components, elements, etc., in this specification are not specifically singular and may include plural. Generally speaking, terms such as "comprising" and "including" only indicate the inclusion of clearly identified steps, elements, or components, and these steps, elements, and components do not constitute an exclusive list; the described method or apparatus may also include other steps or components. Flowcharts are used in this specification to illustrate the operational steps performed by the apparatus or system of related embodiments, but unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the execution order of these steps based on the knowledge and information conveyed by the embodiments in this specification. The adjustments include, but are not limited to, swapping the order of steps, merging multiple steps, and splitting a step.

[0011] The method provided in this application will be described in detail below with reference to specific embodiments. It should be understood that the following embodiments are used to illustrate the implementation of the technical solution of this application, and are not intended to limit the scope of protection of this application. Where there is no conflict, the technical features described in the various embodiments can be combined with each other.

[0012] To facilitate understanding of the technical solutions of this application, some terms used in this application are defined and explained below. Unless otherwise expressly stated, all terms used throughout this application shall be understood according to the following definitions.

[0013] XR (Extended Reality): Extended Reality encompasses Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0014] ERP (Equirectangular Projection): An equidistant cylindrical projection method that uniformly unfolds 360° spherical panoramic content into a two-dimensional rectangular image according to latitude and longitude. It is a commonly used storage and encoding format for XR panoramic video. FOV (Field of View): The field of view angle, the range of vision presented to the user by an XR headset at a single viewing time. Mainstream headsets typically display this as approximately 90°–110°. SSIM (Structural Similarity): An objective indicator that measures the degree of similarity in structural information between two images. Its value is usually in the range of [0,1]. The closer the value is to 1, the more structurally similar the reconstructed image is to the original image. In this specification, if "1-SSIM" is used as the distortion value, it represents the structural distortion obtained by subtracting the SSIM value from 1. The larger the value, the more obvious the structural distortion. PSNR (Peak Signal-to-Noise Ratio): An objective indicator for evaluating image compression distortion. S-PSNR (Spherical Peak Signal-to-Noise Ratio) and WS-PSNR (Spherical Solid Angle Weighted Peak Signal-to-Noise Ratio): The former represents the peak signal-to-noise ratio calculated after uniform sampling in the spherical domain, while the latter represents the peak signal-to-noise ratio calculated by weighting the spherical solid angle corresponding to each pixel.

[0015] Just Noticeable Difference (JND) represents the smallest difference in stimuli that the human eye can perceive.

[0016] This application relates to video encoding, extended reality immersive media processing, multimedia data compression, and storage and transmission technologies, and is particularly suitable for high-magnification and high-quality compression encoding of 5K and 8K resolution extended reality binocular panoramic videos. It can be applied to immersive interactive scenarios such as extended reality immersive exhibition hall large-screen displays, head-mounted extended reality device streaming media playback, panoramic video archiving and storage, and edge streaming media distribution.

[0017] Figure 1 This is the overall architecture of the binocular panoramic video compression coding involved in this application. Please refer to [link / reference]. Figure 1The architecture shown in this application, in one implementation, may include a spherical solid angle weighted distortion measurement module, a head movement prediction and fixation point classification joint weighting module, a binocular justifiable difference consistency constraint module, a multi-objective constraint closed-loop quantization control module, and an encoding execution module. The spherical solid angle weighted distortion measurement module outputs the spherical weighted distortion model and corresponding dimensional weight information. The head movement prediction and fixation point classification joint weighting module outputs the visual axis prediction result and joint weight information. The binocular justifiable difference consistency constraint module outputs the mapping relationship between corresponding spatial units of the left and right eyes and justifiable difference threshold constraint information. The multi-objective constraint closed-loop quantization control module uses the aforementioned information to generate basic quantization parameters (QPs) and related control parameters for each spatial unit. Under the constraints of the basic quantization parameters or adjusted quantization parameters, the encoding execution module encodes and evaluates the candidate prediction modes, reference relationships, motion information, transformation partitions, and local quantization parameters of the current spatial unit, and completes standard-compatible coding. After encoding is completed, the encoding reconstruction results and corresponding distortion statistics can be fed back to the multi-objective constraint closed-loop quantization control module to perform subsequent constraint verification and quantization parameter adjustment.

[0018] To address the issue of discrepancies in the actual spherical sampling area corresponding to different latitude regions after ERP panoramic video is unfolded onto a two-dimensional plane, which can easily lead to a mismatch between encoding resource allocation and the actual spherical perception requirements, and to resolve the significant compression distortion differences that may arise when encoding the corresponding regions for the left and right eyes separately, as well as the issue of the user's attention area changing with the gaze direction in head-mounted displays, this application provides a binocular panoramic video compression encoding method. This method uses the spherical sampling characteristics and gaze-related characteristics of spatial units together for quantization parameter determination. After left and right eye encoding reconstruction, it further coordinates the encoding quality of the left and right eyes based on the cross-eye distortion difference and the just-perceptible difference threshold. Simultaneously, it verifies the encoding results through spherical global distortion constraints, cross-eye consistency constraints, and gaze area distortion constraints. This allows limited encoding resources to be more rationally allocated to spatial regions that are more important to the actual spherical quality and the user's current viewing experience, and suppresses the quality differences that the user can perceive between the corresponding regions of the left and right eyes. This at least partially improves compression efficiency, maintains image quality in the gaze area, and reduces binocular competition, eye fatigue, and dizziness.

[0019] The spatial unit referred to in this application is the basic processing object involved in distortion statistics and rate allocation during the encoding process. Its implementation granularity can be a pixel or a coding block. The basic quantization parameters are the first round of quantization parameter allocation results pre-generated for the current spatial unit based on spherical sampling weights and gaze weights. The current candidate coding parameters are the coding candidate parameters further evaluated for the current spatial unit during the encoding execution process under the constraints of the basic quantization parameters or quantization parameter callback results. These can include prediction modes, reference relationships, motion information, transform partitions, and local quantization parameter adjustment terms. The coding reconstruction result is the local reconstruction result formed by the encoder using the current candidate coding parameters through prediction, transform, quantization, dequantization, and reconstruction processes. The basic distortion is the difference between the original spatial unit and the corresponding coding reconstruction result. It can preferably be the mean squared error, or it can be the structural distortion obtained by subtracting the structural similarity. Unless otherwise stated, all types of distortion, structural similarity, and just-perceptible difference comparison quantities involved in this application can be determined according to the above-mentioned criteria during the encoder's rate-distortion optimization and candidate mode decision-making process. Figure 2 This is a flowchart of a binocular panoramic video compression coding method provided in this application. Figure 3 This is another flowchart illustrating the binocular panoramic video compression coding method involved in this application, specifically a flowchart used to demonstrate the overall coding timing. Please refer to... Figure 2 and Figure 3 In some embodiments, the binocular panoramic video compression encoding method provided in this application includes the following steps.

[0020] Step 210: Obtain the binocular panoramic video to be encoded and the corresponding gaze direction information.

[0021] To ensure that the encoding quality configuration can adapt to the constantly changing field of view of the user in a head-mounted display device as the head moves, this application further specifies the continuous acquisition of head posture data, predicts the visual axis direction within a preset time window based on the head posture data, and determines the gaze weight based on the eccentricity angle between the spherical direction of the spatial unit and the predicted visual axis direction. Since the gaze weight decreases as the eccentricity angle increases, spatial units closer to the user's likely gaze direction can receive higher quality priority, thus at least partially solving the problem that fixed high-quality areas cannot change with the user's viewing direction. The binocular panoramic video to be encoded can include left-eye video and right-eye video, both of which can be stored and encoded in ERP format. The gaze direction information is used to characterize the spherical direction that the user may focus on during the current or subsequent encoding decision time range.

[0022] Figure 4 This is a flowchart illustrating the joint weighting of head movement prediction and fixation level classification involved in this application, used to explain the principle of generating the joint weighting of head movement prediction and fixation level classification. Please refer to... Figure 4In one implementation, gaze direction information is obtained by continuously acquiring head posture data, predicting the visual axis direction within a preset time window based on the head posture data, and using the predicted visual axis direction as the gaze direction information. Specifically, this application continuously acquires head posture data returned by the head-mounted display device, for example, acquiring a head posture quaternion sequence at a sampling rate of not less than 90 Hz. A predictive model is established based on continuous head pose data, using head pose, angular velocity, and angular acceleration as state variables. An extended Kalman filter (EKF) is then used for state estimation and extrapolation to obtain the state within the time window. The predicted visual axis orientation. This predicted visual axis orientation is used to characterize the spherical orientation that the user is most likely to focus on within the subsequent encoding decision time window, and can serve as prior input for gaze priority determination. Prediction time window The preferred time window is 100ms to 500ms (ms, milliseconds), and its specific value can be determined by considering the video frame rate, image group length, and user head movement speed. In one preferred method, the prediction time window... Take 300ms. The predicted visual axis direction or predicted visual axis trajectory within the time window mentioned here can be understood as the sequence of changes in the direction of the user's line of sight on the sphere within the future time window.

[0023] A prediction confidence mechanism can be set for gaze direction prediction. During the encoding process, the predicted visual axis direction can be continuously compared with the subsequently transmitted true visual axis direction, and the prediction confidence is determined based on the deviation between the two. The prediction confidence can be determined based on one or more of the following: the prediction error of the visual axis angle, the magnitude of the change in angular velocity, and the degree of abrupt change in angular acceleration over several consecutive historical frames. When a sudden change in angular velocity is detected, the historical prediction error continues to increase, or the prediction confidence falls below a set threshold, the high-fidelity gaze region used subsequently can be expanded, or the gaze direction-based hierarchical processing can be temporarily disabled, and only spherical sampling weights can be used. The threshold can be determined based on offline training samples, statistical results of historical viewing data, or engineering calibration experiments, for example, set as a threshold that keeps the angular error between the predicted visual axis direction and the true visual axis direction within a predetermined comfortable range. Using the above method can reduce the possibility of image quality degradation in the gaze region due to over-reliance on prediction results when there is rapid head movement or unstable gaze prediction.

[0024] After obtaining the predicted visual axis direction, the spherical direction corresponding to each spatial unit can be determined, and the angle between this spherical direction and the predicted visual axis direction can be calculated. This angle is taken as the eccentricity angle. Based on the characteristic that visual acuity is higher near the fovea and decreases as it deviates from the fixation center, the fixation weight can be gradually decreased as the eccentricity angle increases. When the eccentricity angle is small, the spatial unit is closer to the predicted fixation center, so it can be assigned a relatively high fixation weight. When the eccentricity angle is large, the spatial unit is located outside the user's fixation direction, so it can be assigned a relatively low fixation weight. This fixation weight can then participate in the joint weight calculation and quantization parameter determination, enabling the bitrate configuration to adjust according to changes in the user's focus direction.

[0025] Head movement trajectory prediction is not limited to extended Kalman filtering. Lightweight Long Short-Term Memory (LSTM) networks or Gated Recurrent Unit (GRU) networks can also be used to predict future visual axis directions. Alternatively, a Transformer prediction model trained on multi-user viewing data can be employed. In computationally limited edge devices, linear extrapolation based on the current angular velocity can be used to obtain short-term predicted visual axis directions. All of these prediction methods can be used to obtain gaze direction information. By continuously acquiring head pose data instead of using a fixed gaze region and predicting the visual axis direction within a subsequent preset time window, gaze weights can reflect potential changes in the user's gaze direction in advance. Furthermore, considering the relationship that gaze weights decrease with increasing eccentricity, higher quality priorities can be assigned to spatial units closer to the predicted gaze center under limited bitrate conditions, while lower priorities can be assigned to regions farther from the predicted gaze center. This at least partially improves the encoding quality preservation and bitrate utilization efficiency of the actual viewing area in dynamic field-of-view scenarios.

[0026] Step 220: For spatial units in binocular panoramic video, determine spherical sampling weights based on the spherical orientation corresponding to the spatial unit, determine gaze weights based on the spherical orientation and gaze direction information corresponding to the spatial unit, and determine the joint weights of the spatial units based on the spherical sampling weights and gaze weights.

[0027] The reason for setting spherical sampling weights in step 220 is that although ERP unfolds spherical content according to latitude and longitude rules on a two-dimensional screen, the actual spherical sampling area represented by a unit pixel in different latitude zones is not the same. If the same distortion weight is applied to all pixels on the two-dimensional plane, the contribution of the polar and equatorial regions to the true spherical quality cannot be accurately represented. By determining the spherical sampling weights based on the spherical orientation corresponding to the spatial unit, the subsequent distortion statistics and quantization parameter allocation can reflect the actual sampling proportion of the spatial unit on the sphere.

[0028] Specifically, for an ERP image with a width of W and a height of H, the spherical latitude corresponding to each pixel can be determined first based on the pixel center position. and longitude For the image located at the first position on the projection screen... The spatial unit within the latitude zone will be the first The latitude corresponding to the center of each latitude zone is determined as follows: The corresponding longitude is determined as follows: .in, W represents the height of the projected image, and W represents the width of the projected image.

[0029] The above mapping uses the pixel center position for sampling, therefore when or At that time, the corresponding position corresponds to the center of the uppermost or lowermost latitude zone, not the South Pole or North Pole of the sphere itself. Latitude It can be directly used to determine the latitude-related spherical sampling weights, longitude It can be used together with latitude to determine the complete spherical orientation of a spatial unit, and further used for calculating the gaze eccentricity angle, geometric mapping of the left and right eye cross-eye regions, and determining the continuity of the ERP left and right meridian joints.

[0030] The solid angle corresponding to a unit pixel on a sphere has a corresponding relationship with the cosine of the latitude. Therefore, the cosine value of the center latitude of the j-th latitude band can be used as the unnormalized weight of that latitude band, and then normalized using the sum of the cosine values ​​of the center latitudes of all latitude bands. Thus, the weight of the j-th latitude band can be determined. The spherical sampling weights of spatial units within each latitudinal zone are determined as follows: In the formula, This represents the current latitude zone index, and its value is determined by... to When summing, it is possible to traverse all latitude zones, that is... Indicates the first Each latitude zone corresponds to a latitude value. Since latitude is determined by the pixel center position rather than directly by the North and South Poles of a sphere, therefore, for any... All Correspondingly During encoder initialization, a one-dimensional latitude weight lookup table can be pre-generated based on the height of the projected image. After determining the latitude zone of the current spatial unit during encoding, the corresponding spherical sampling weight can be directly queried, eliminating the need to repeatedly perform complex calculations for each spatial unit. This results in lower computational overhead for spherical perception distortion evaluation.

[0031] After obtaining the spherical sampling weights, a spherical solid angle weighted distortion metric model can be constructed using them. After the encoder generates the encoded reconstruction result for a specific region to be encoded, it can first calculate the basic distortion of each spatial unit within that region, and then aggregate them according to latitude zones. Specifically, this can be done for the... The average fundamental distortion of each spatial unit participating in the statistics within a latitudinal zone is calculated by averaging the fundamental distortion of that latitudinal zone. Then, the average fundamental distortion is weighted using the normalized spherical sampling weights of the corresponding latitudinal zone. Finally, the weighted results of all latitudinal zones are summed to obtain the spherically weighted global distortion. The specific calculation process can be as follows: .in, Indicates the first Within the first latitude zone The fundamental distortion of each spatial unit under the current candidate coding decision; Indicates the first The number of spatial units participating in the statistics within each latitudinal zone; Indicates the first The average value of the fundamental distortion within each latitudinal band. The fundamental distortion is obtained by comparing the original spatial cells with the corresponding candidate reconstruction results during rate-distortion optimization. Using the above calculation method, it can be obtained in subsequent encoding processes. In other alternative implementations, the basic distortion Structural distortion can also be used. Replacement, among which This represents the structural similarity between the original image and the corresponding candidate reconstructed image within this spatial unit. Step 220 completes the establishment of the model and its evaluation rules. The specific value is calculated during the subsequent encoding process as the candidate reconstruction results are generated. It reflects the user's true perceptual distortion in spherical space and eliminates bitrate mismatch in extreme regions.

[0032] The fundamental distortion corresponding to each spatial unit is weighted based on spherical sampling weights. Specifically, this can be achieved by first aggregating the fundamental distortions within the same latitude band (e.g., averaging the fundamental distortions of the spatial units participating in the statistics within that latitude band), then weighting them using the spherical sampling weights of the corresponding latitude band, and finally aggregating the weighted results from different latitude bands to obtain the spherical solid angle weighted distortion (DSWD). The fundamental distortion can be calculated using mean squared error or other methods. The resulting structure is distorted.

[0033] Obviously, this application is not limited to the ERP format. When using the cubemap projection format (CMP), the spherical sampling weights can be determined according to the following formula. When using other spherical projection formats such as Equi-Angular Cubemap (EAC), the spherical sampling weight function can be replaced with the local sampling density function corresponding to the respective projection format. This allows the basic idea of ​​spherical sampling weights to be applied to different spherical projection formats. Since the actual sampling area of ​​pixels closer to the ERP polar region differs significantly from that of pixels near the equator, using spherical sampling weights that vary with latitude cosine can reduce the impact of 2D projection sampling non-uniformity during quality assessment and subsequent encoding resource allocation, making the overall distortion assessment closer to the true contribution of spherical sampling. Simultaneously, since the weights can be pre-generated using a lookup table based on latitude zones, it can also reduce redundant computations during the real-time encoding stage while achieving spherical perception assessment.

[0034] In step 220, this application further determines the gaze weight based on the spherical orientation of the spatial cell and the gaze direction information obtained in step 210. Since human visual acuity typically decreases as the spatial position deviates from the gaze center, the eccentricity angle can be determined based on the angle between the spherical orientation of the spatial cell and the predicted visual axis direction. The weighting function for quality requirement attenuation is determined based on the eccentricity angle. The quality demand decay weighting function can be set as follows: ; Central concave high-resolution region half-angle The preferred angle is 15°, and the attenuation order k is preferably 2 to 4, used to control the rate of quality demand attenuation from the fixation center to the peripheral region. For any spatial unit, the fixation weight is determined based on the eccentricity angle between the spherical direction and the visual axis direction corresponding to the spatial unit, and the fixation weight decreases as the eccentricity angle increases. Specifically, when the eccentricity angle is small, the fixation weight is close to one, and the corresponding spatial unit has a high quality priority. As the eccentricity angle gradually increases, the fixation weight gradually decreases, and the corresponding spatial unit can have a lower quality priority.

[0035] Based on this, the joint weight can be obtained by multiplying the spherical sampling weight corresponding to each spatial unit with the gaze weight. The latitude of the spatial unit is determined by the mapping relationship between pixel coordinates and spherical orientation, while the eccentricity of the spatial unit is calculated based on the spherical orientation of the spatial unit and the predicted visual axis orientation. The joint weight of spatial unit u... It can be obtained by multiplying its spherical sampling weight and gaze weight, specifically determined by the following formula: .in, Represents a spatial unit. Represents the spherical sampling weight. This represents the gaze weight corresponding to the spatial unit under the gaze classification mechanism. Since both the spherical sampling weight and the gaze weight are positive, the joint weight is also positive. The joint weight reflects both the actual sampling percentage of the spatial unit on the sphere and the user's actual attention to that spatial direction, and can serve as the quantization basis for subsequent basic quantization parameter mapping and bitrate allocation.

[0036] To mitigate the adverse effects of prediction errors on image quality in the gaze area, a prediction confidence mechanism can be implemented. This mechanism continuously compares the predicted visual axis direction with the subsequently transmitted true visual axis direction, determining the prediction confidence based on one or more of the following: visual axis angle prediction error, angular velocity variation, and angular acceleration abrupt change over several consecutive historical frames. When a sudden change in user angular velocity is detected, historical prediction errors continue to increase, or the prediction confidence falls below a set threshold, the concave half-angle can be appropriately increased. To expand the high-quality area, fixation grading can be temporarily disabled, in which case the joint weights degenerate to be determined solely by spherical sampling weights. The prediction confidence threshold can be set based on offline training samples, historical viewing data statistics, or engineering calibration experiments, and can preferably be a threshold that keeps the angular error between the predicted visual axis direction and the actual visual axis direction within a predetermined comfortable range. This degradation mechanism can reduce the possibility of excessive compression of the actual fixation area due to prediction errors when the head moves rapidly or the prediction results are unstable.

[0037] The gaze weighting function converts the angular relationship between spatial cells and the predicted visual axis into a continuous quality priority, while the joint weighting further integrates this quality priority with the spherical sampling contribution. Therefore, the base quantization parameters and bitrate allocation can be adjusted simultaneously around the user's spherical space and the direction the user might actually be viewing, thus achieving at least partial coordination between maintaining quality in the gaze region and saving bitrate in the peripheral region.

[0038] Step 230: Determine the basic quantization parameters of the spatial units of the left-eye video and the right-eye video respectively according to the joint weights. Encode the left-eye video and the right-eye video respectively based on the basic quantization parameters. Based on the encoding and reconstruction results of the spatial units of the left-eye video and the right-eye video, determine the compression distortion of the spatial units of the left-eye video, the compression distortion of the spatial units of the right-eye video, and the cross-eye distortion difference between the two.

[0039] Specifically, a baseline quantization parameter can be set. and bitrate sensitivity The basic quantization parameters of the spatial unit u can be in the following form: ;in, Used as the baseline quantization parameter. The joint weights are the values ​​corresponding to spatial unit u. The average of the joint weights of multiple spatial units. It is the bitrate sensitivity coefficient, and .

[0040] In one example, the rate sensitivity coefficient β is preferably 2 to 4, and its value can be determined based on offline rate distortion calibration experiments, fitting results of the relationship between quantization parameters and rate response on typical sample sets, or engineering parameter tuning results. In a preferred embodiment, Let's take 3. When the joint weight of a spatial cell is greater than the mean joint weight, it indicates that the spatial cell has high spherical perception importance or high gaze priority. Therefore, the coding quality can be improved by reducing the basic quantization parameter. When the joint weight of a spatial cell is lower than the mean joint weight, the basic quantization parameter can be appropriately increased to reduce the bitrate consumed in that region. The above mapping is an implementation method for determining the first-round basic quantization parameter based on rate-distortion optimization principles. It does not require that it must be a unique closed-form solution obtained by strictly analytically differentiating a certain objective function.

[0041] In a binocular scene, both the spatial units of the left-eye and right-eye videos can use their respective basic quantization parameters as initial quantization parameters during the first round of encoding. Under the constraints of these basic quantization parameters, the encoder can evaluate candidate encoding parameters for the spatial units, including prediction modes, reference relationships, motion information, transform partitioning, and local quantization parameter adjustments, and determine the corresponding encoding decision based on rate-distortion optimization. The candidate encoding parameters are then processed through prediction, transform, quantization, dequantization, and reconstruction to form the encoded reconstruction result. Comparing the original spatial units of the left eye with its encoded reconstruction result yields the compression distortion of the spatial units in the left-eye video. By comparing the original spatial units of the right eye with its encoded reconstruction result, the compression distortion of the spatial units in the right eye video can be obtained. Compression distortion can be measured using mean square error or structural distortion obtained from 1-SSIM.

[0042] To ensure geometrically accurate comparison of compression quality between corresponding regions of the left and right eyes, and to allow permissible differences in quality between the left and right eyes to adapt to the human eye's perceptual characteristics across regions of varying brightness and texture, this application further defines the spherical correspondence of spatial units in left and right eye videos, the method for calculating cross-eye distortion difference, and the method for determining a perceptible difference threshold based on brightness and texture complexity. By first establishing the left-right eye correspondence within the same spherical region, then evaluating the compression distortion difference of the corresponding region, and setting a perceptible threshold according to the region's visual characteristics, the problems of misalignment between left and right eye comparison regions and the inability to adapt to different image content using a fixed threshold can be at least partially resolved.

[0043] Specifically, establishing a correspondence between spatial units in the left-eye and right-eye views can be achieved by determining the directional positions of these spatial units on the sphere, respectively. For binocular panoramic videos using the ERP format, the spherical orientation of the spatial units can be obtained first based on the mapping relationship between ERP pixel coordinates and spherical orientation. Then, a cross-eye region mapping can be established by combining the disparity estimation results or spherical reprojection alignment to determine a pair of spatial units representing the same spherical region in the left-eye and right-eye videos. Using this cross-eye region mapping allows subsequent left-eye distortion comparisons to be based on geometrically comparable spatial positions, and provides a correspondence for cross-eye distortion difference calculation, just-perceptible difference comparison, and subsequent cross-eye bitrate adjustment.

[0044] For a pair of spatial units of the left-eye video and the right-eye video that have established a correspondence, the absolute value of the difference in compression distortion between the two can be defined as the cross-eye distortion difference, that is, the cross-eye distortion difference between the two is: For example, when there is left-eye compression distortion in a corresponding spatial unit... Right eye compression distortion At that time, the trans-eye distortion difference on the spatial unit can be obtained. Therefore, step 230 not only obtains the encoding reconstruction quality of each eye, but also converts the compression quality difference between corresponding regions of the left and right eyes into cross-eye distortion difference that can be directly used for subsequent threshold determination.

[0045] In a standard-compliant implementation, the reconstruction results or mapping information of the corresponding region for the left eye can be used internally within the encoder to assist the right-eye encoding. For example, the right-eye image can form a prediction reference based on the corresponding region for the left eye after disparity mapping, and the corresponding disparity residual can be encoded. The corresponding residual quantization can still be subject to cross-eye consistency constraints. This auxiliary reference can occur only within the encoder, and the output still forms a standard-compliant left and right-eye bitstream, thus eliminating the need for the decoder to execute custom cross-eye prediction logic.

[0046] Step 240: Determine the just-perceptible difference threshold corresponding to the spatial unit of the left eye video and the spatial unit of the right eye video, and determine the current quantization parameters of the spatial unit of the left eye video and the spatial unit of the right eye video based on the comparison result of the cross-eye distortion difference and the just-perceptible difference threshold.

[0047] The reason for setting the just-perceptible difference threshold in step 240 is that the human eye's perception of quality differences between the left and right eyes is not constant. When the cross-eye distortion difference is below the range that the human eye can perceive, the impact on the viewing experience is small. However, when the cross-eye distortion difference exceeds the corresponding perceptible range, it may lead to binocular competition and increase the possibility of eye strain or dizziness. Therefore, this application uses the just-perceptible difference threshold as the basis for determining whether further coordination of the encoding quality of the left and right eyes is needed.

[0048] Figure 5 This is a flowchart illustrating the binocular just-perceptible difference consistency constraint and cross-eye bitrate adjustment involved in this application, used to demonstrate the principles of binocular just-perceptible difference consistency constraint and cross-eye bitrate adjustment. Please refer to... Figure 5 In one implementation, an adaptive just-perceptible difference threshold can be determined based on the brightness characteristics and texture complexity characteristics of the corresponding spatial unit. Specifically, the perceptible difference threshold can be determined based on the brightness characteristics and texture complexity characteristics of the corresponding spatial units, and satisfies: .in, The difference can be detected precisely based on the foundation. For the brightness statistics of a spatial unit, This is a statistical measure of the texture complexity of a spatial unit. Let be the adjustment function for brightness with respect to the threshold of just-perceptible difference. This is the adjustment function for texture complexity on the just-perceptible difference threshold. This threshold can be understood as the result of the combined effects of the basic just-perceptible difference, brightness adjustment, and texture complexity adjustment. The basic just-perceptible difference can be calibrated through binocular subjective experiments. The brightness adjustment function describes the influence of spatial unit brightness on the human eye's perceptible difference threshold; darker areas correspond to a larger threshold. The texture complexity adjustment function describes the influence of local texture masking on the perceptible difference threshold; more complex textures correspond to a larger threshold.

[0049] The luminance statistics L of spatial cell u can be: The texture complexity statistic C of spatial unit u is calculated as follows: .in, Represents the pixel within spatial cell u The brightness component, Representing spatial units Number of inner pixels, Represents pixels The local gradient of the brightness component.

[0050] The brightness adjustment function can be a monotonic function that increases as brightness decreases, and the texture complexity adjustment function can be a monotonic function that increases as texture complexity increases. These functions can be implemented using piecewise linear functions, exponential functions, or lookup tables. In a preferred implementation, the brightness adjustment function... This can be in this form: The texture complexity adjustment function can be of this form: .in, This indicates the maximum value of the luminance component; for an 8-bit luminance signal, it can be 255. and The adjustment coefficient is preferably set within the range of 0.1 to 1.0. This is a texture complexity smoothing constant used to avoid overly steep function changes in low-texture regions. For example, when... , , , , , , At that time, it can be obtained , Further obtained The above function forms and parameters are merely preferred implementations and do not constitute a limitation on the scope of protection of this application. The above function forms and parameter values ​​are preferred implementations and do not require that identical parameters be used in all applications.

[0051] As can be understood from the technical features defined in this application, by utilizing spherical orientation, disparity estimation, or spherical reprojection to make the comparison objects for the left and right eyes correspond to the same spherical region, the geometric effectiveness of cross-eye distortion comparison can be improved. Furthermore, by adjusting the just-perceptible difference threshold based on brightness and texture complexity features, the left-right eye consistency constraint can be adapted to the visual sensitivity of different regions of the human eye, rather than applying a fixed threshold to all regions. This at least partially reduces the possibility of bitrate waste due to an overly strict fixed threshold or perceptible binocular differences due to an overly broad fixed threshold.

[0052] In one implementation, the current quantization parameter of the left eye can be represented as the result of the combined effect of the base quantization parameter and the left eye cross-eye callback amount, and the current quantization parameter of the right eye can be represented as the result of the combined effect of the base quantization parameter and the right eye cross-eye callback amount. Wherein, the current quantization parameter of the left eye... Current quantization parameters of the right eye They can be represented in the following forms.

[0053]

[0054] in, , This represents the amount of cross-eye callbacks triggered by cross-eye consistency constraints; both are initialized to 0 before the first round of coding. Used to reflect spherical perception and gaze priority. , Used to reflect the binocular consistency compensation triggered after encoding evaluation. Basic quantization parameters. This primarily reflects the importance of spherical perception and fixation priority, while the cross-eye regression amount between the left and right eyes... , This primarily reflects the quality coordination requirements arising from binocular consistency testing after the initial coding reconstruction. Current quantization parameters. , It can also be truncated according to preset upper and lower limits, such as limiting it to between 12 and 37, in order to avoid excessive changes in local quantization parameters that cause sudden changes in spatial image quality.

[0055] Before the first round of encoding, both cross-eye callbacks can be initialized to 0. When the cross-eye distortion difference exceeds the corresponding just-perceptible difference threshold, the quantization parameters of the degraded and non-degraded sides are adjusted respectively. For example, when the cross-eye distortion difference exceeds the just-perceptible difference threshold, and the compression distortion of the spatial unit u of the left eye video... Compression distortion greater than the spatial unit u of the right eye video At that time, the current quantization parameter of the spatial unit of the left eye video is determined to be less than its basic quantization parameter. The current quantization parameter of the spatial unit u of the right eye video is determined to be greater than or equal to its fundamental quantization parameter. When the cross-eye distortion difference exceeds the just-perceptible difference threshold, and the compression distortion of the spatial unit u of the right eye video... Compression distortion greater than the spatial unit u of the left eye video In this process, the current quantization parameter of the spatial unit of the right eye video is set to be less than its baseline quantization parameter, while the current quantization parameter of the spatial unit of the left eye video is set to be greater than or equal to its baseline quantization parameter. By doing so, the quality of the side with greater compression distortion can be prioritized while keeping the total bitrate budget under control. If necessary, a small reverse adjustment can be made to the side with better quality, thus gradually bringing the difference in distortion between the two eyes back to within a perceptible threshold, rather than simply increasing the overall bitrate of both channels.

[0056] In this way, by identifying the side with greater compression distortion and reducing the quantization parameter on that side, the coding quality of the degraded side can be directly improved, while ensuring that the quantization parameter on the other side is not lower than the basic quantization parameter can reduce unnecessary increases in dual-path synchronization bitrate. Therefore, it is possible to at least partially reduce the compression quality difference between the corresponding regions of the left and right eyes with a relatively small bitrate adjustment cost, and reduce the possibility of binocular competition, eye fatigue, and dizziness.

[0057] Step 250: Perform constraint verification on the encoding result obtained based on the current quantization parameters.

[0058] In step 250, the constraint verification includes at least a spherical global distortion constraint determined based on spherical sampling weights, cross-eye consistency constraints for corresponding spatial units of the left-eye and right-eye videos, and gaze region distortion constraints determined based on gaze direction information. In one implementation, the optimization objective is to reduce the total bitrate of all spatial units, while requiring that the spherical global weighted distortion does not exceed a spherical global distortion threshold, that the cross-eye distortion difference of each corresponding spatial unit does not exceed its respective just-perceptible difference threshold, and that the gaze center region distortion does not exceed a gaze region distortion threshold. The optimization variables can essentially include the quantization parameters of each spatial unit and the resulting bitrate allocation. Specifically, they can be set in the following form.

[0059] ; ; ; ; By setting the above constraints simultaneously, it is possible to prevent the encoder from optimizing only a single metric, which could lead to a significant decline in other quality metrics related to the user's viewing experience. The spherical global distortion constraint can be based on the spherical global weighted distortion obtained in step 220. Execute. A global spherical distortion threshold can be set. Furthermore, it is required that the spherical global weighted distortion does not exceed this threshold. In a preferred embodiment, the spherical global distortion threshold can correspond to a spherical peak signal-to-noise ratio (S-PSNR) ≥ 40 dB. Since the spherical global weighted distortion is statistically calculated using spherical sampling weights, this constraint can reflect the actual sampling contribution of different spatial units on the sphere when evaluating overall image quality.

[0060] Cross-eye consistency constraints require that the cross-eye distortion difference between corresponding spatial units of the left-eye video and the right-eye video does not exceed the corresponding just-perceptible difference threshold. Therefore, binocular coding quality control does not merely require each eye to meet a certain absolute quality standard, but further limits the perceptible quality difference between the two eyes, helping to reduce binocular competition caused by significant inconsistencies in compression quality between the left and right eyes.

[0061] The gaze region distortion constraint can be implemented by determining the gaze center region based on the gaze direction information obtained in step 210, and then statistically analyzing the basic distortion of each spatial unit within the gaze center region. Specifically, the eccentricity angle between the gaze center region and the predicted visual axis direction can be less than half an angle of the gaze center region. Spatial units constitute the collection of spatial units at the gaze center , The preferred setting is 10°. This can be applied to sets. The distortion of the gaze center region is obtained by averaging the spatial unit basic distortion. , ,in, Indicates the spatial unit of the gaze center area The fundamental distortion in the fixation center region can be calculated using mean squared error or alternative methods. The obtained structural distortion. When using structural distortion for evaluation, the gaze region distortion threshold. It can be set to the average structural similarity of the corresponding gaze center region. This constraint can prevent a significant degradation in the image quality of the area the user is currently focusing on when the overall bitrate is low or the peripheral areas are heavily compressed.

[0062] Furthermore, the Lagrange relaxation method can be used to transform the above constraints into penalty terms in the rate-distortion optimization process. The penalty term corresponding to spherical global distortion can be determined based on the spatial cell bit rate, spherical sampling weight, and fundamental distortion. The penalty term corresponding to cross-eye consistency can be determined based on the degree to which the cross-eye distortion difference exceeds the just-perceptible difference threshold. The penalty term corresponding to the fixation region can be determined based on the degree to which the distortion in the fixation center region exceeds the fixation region distortion threshold. The specific forms are as follows.

[0063]

[0064] in, , and These can represent the penalty weights corresponding to the spherical global constraint, the trans-eye consistency constraint, and the fixation center region constraint, respectively. The spherical global distortion penalty term continues to use the spherical sampling weights defined in step 220. To avoid statistical distortion when making coding decisions, and to ensure that the statistical caliber used is consistent with the spherical global weighting. The evaluation criteria remain consistent. The joint weights are used for the first round of basic quantization parameter mapping to reflect the combined effect of spherical sampling characteristics and gaze priority. Binocular consistency and gaze region constraints can primarily be applied to the re-encoding stage through over-limit penalties and subsequent closed-loop adjustments, rather than being fully reflected in the first round of basic quantization parameter mapping. Based on this, the basic quantization parameter mapping formula corresponding to the joint weights is first derived: The mapping formula has already been explained above, and will not be repeated here.

[0065] Step 260: When at least one constraint does not meet the corresponding constraint condition, adjust the quantization parameters of the corresponding spatial unit according to the unmet constraint, and encode again based on the adjusted quantization parameters.

[0066] Figure 6 This is a flowchart illustrating the multi-objective constraint closed-loop quantization control process involved in this application. Please refer to [link / reference]. Figure 6 In step 260, to form a stable and reproducible constraint control process, image groups can be used as the unit for closed-loop verification and re-encoding. After encoding each image group, it can be determined whether the spherical global distortion of the image group satisfies the spherical global distortion constraint, whether there are spatial units within the image group whose cross-eye distortion difference does not satisfy the cross-eye consistency constraint, and whether the gaze region distortion of the image group satisfies the gaze region distortion constraint. If all constraints satisfy the corresponding conditions, the encoding result of the current image group can be output. If at least one constraint does not satisfy the corresponding condition, the spatial units corresponding to the unsatisfied constraints are identified, the quantization parameters of these spatial units are adjusted, and the current image group is re-encoded.

[0067] When cross-eye consistency constraints are not met, the quantization parameters of spatial units corresponding to the side with greater compression distortion can be reduced first to improve the coding quality on that side. The quantization parameters of spatial units corresponding to the other side can then be appropriately increased based on the total bitrate budget. When fixation region distortion constraints are not met, the quantization parameters of relevant spatial units within the fixation region can be reduced to improve the coding quality of the user's currently focused region. When spherical global distortion constraints are not met, the quantization parameters of relevant spatial units can be adjusted based on the spherical sampling weights and the contribution of each spatial unit to the overall spherical distortion. If multiple constraints are simultaneously not met, the process can proceed step-by-step, prioritizing cross-eye consistency constraints over fixation region distortion constraints, and then further prioritizing fixation region distortion constraints over spherical global distortion constraints, until all constraints meet their corresponding conditions.

[0068] The above verification process can be implemented as a deterministic algorithm. The term "identical input" here can include the same original video frame sequences for the left and right eyes' ERP systems, the same head pose input sequence, the same image grouping method, and the same global initial configuration parameters (including...). , , , , The global initial configuration parameters include the same initial state of the prediction model and the same quantization parameter callback rules. These parameters can include the baseline quantization parameters, bitrate sensitivity coefficient, spherical global constraint penalty weight, cross-eye consistency constraint penalty weight, gaze region constraint penalty weight, and various constraint thresholds. These global initial configuration parameters are used to initiate the first round of basic quantization parameter mapping and closed-loop control, but do not include the per-spatial-unit basic quantization parameters further formed during the encoding process based on the current video content. Current quantization parameters for left and right eyes , And candidate coding parameter combinations. When the encoding process does not introduce random perturbations or random initialization, the same input conditions can produce consistent quantization parameter allocation results and final output bitstream, thereby improving the quantizability, reproducibility, and auditability of the compression process.

[0069] During encoding, cross-boundary reference protection can be implemented for the meridian seams of the ERP projection. The 0° and 360° meridians on the ERP sphere are continuous in spherical space, but they are located at the left and right edges of the two-dimensional ERP image, respectively. If treated independently as ordinary two-dimensional boundaries, breakage artifacts are easily generated at the seams. Therefore, a predetermined width of strip area can be mirrored and extended at the left and right edges of the image, allowing motion estimation and motion compensation to reference across the left and right image boundaries. The width of the mirrored strip is preferably 32 pixels. This allows the encoder to obtain cross-boundary reference information when processing meridian regions that are continuous on the sphere but separated in the plane, thereby reducing breakage artifacts at the seams.

[0070] The encoder can be implemented based on standard H.265 or H.266 bitstream syntax. Quantization parameters and inter-frame reference relationships can be carried through the corresponding standard syntax elements, while extended parameters such as joint weight parameters and prediction model versions can be provided with the bitstream through Supplemental Enhancement Information (SEI) or external metadata. In a preferred embodiment, the left-eye video and right-eye video are encoded as independent sub-bitstreams conforming to the corresponding standards. Even if the encoder uses the left-eye reconstruction result or its mapping information as an auxiliary reference when generating right-eye prediction values, residual estimations, or quantization parameter callback decisions, the output can still output only the right-eye bitstream conforming to the standard syntax or the standard-compatible dual-stream encapsulation result. Therefore, the decoder does not need to perform a custom cross-eye reference prediction process; it can be parsed and played according to the corresponding standard bitstream and encapsulation format. Existing extended reality playback terminals, immersive display screens, and edge distribution devices can directly decode if they support the corresponding standards and encapsulation formats.

[0071] As a specific implementation of this application, offline compression can be performed on 8K resolution binocular panoramic content in an extended reality immersive exhibition hall. The input video is a binocular 8K resolution ERP panoramic video, with a single-eye resolution of 7680×3840, a frame rate of 30fps (frames per second), and 10-bit sampling in a 4:2:0 ratio. The head pose sequence can be generated from typical viewing trajectories recorded by the content acquisition terminal or from multi-user viewing heatmap statistics, and the sampling rate can be set to 120Hz. The prediction time window is set to 300ms, and the half-angle of the foveal high-resolution region is set to... =15°, eccentricity attenuation order k=3, bit rate sensitivity coefficient β=3, baseline quantization parameters The quantization parameter truncation range is set to [12, 37], the image group length is set to 15 frames, and the seam mirror strip width is set to 32 pixels. Spherical global distortion threshold. The threshold can be set to the corresponding spherical peak signal-to-noise ratio (S-PSNR) ≥ 40 dB, the gaze region distortion threshold can be set to the average structural similarity (SSIM) of the gaze center region with an eccentricity angle of less than 10° ≥ 0.97, and the basic just-perceptible difference can be calibrated through binocular subjective evaluation experiments.

[0072] Under the conditions of 30 frames per second and an image group length of 15 frames, the duration of an image group is approximately 500ms. This application sets the prediction time window to 300ms to cover the beginning of the current image group or the main gaze change interval, so as to balance gaze prediction stability and coding real-time performance, without requiring the prediction time window to be exactly the same as the duration of the entire image group. In actual processing, a latitudinal spherical sampling weight lookup table can be generated first, and then a joint weight per image group can be generated by combining the head pose prediction results. Cross-eye correspondence can be established based on the spherical direction of the left and right eyes and the disparity estimation or spherical reprojection results, and a just-perceptible difference threshold can be generated. Then, the basic quantization parameters are obtained based on the joint weight and encoded. Then, the spherical global weighted distortion, the compression distortion of the left and right eyes and the cross-eye distortion difference are calculated based on the encoding reconstruction results of the left and right eyes. The current quantization parameters of the left and right eyes are adjusted as needed and re-encoded. Finally, the three types of constraints are backtested for each image group. For image groups that exceed the limits, the quantization parameters are called back and re-encoded according to the predetermined constraint priority until the constraint conditions are met, and then the binocular bitstream and corresponding metadata are output.

[0073] In summary, the spherical solid angle weighted distortion metric in this application can determine the distortion weight according to the actual spherical sampling area corresponding to the spatial unit, and the computational burden in the encoding stage can be reduced by pre-generating a weight lookup table. Head motion prediction and gaze point classification can combine the predicted visual axis direction with the spherical sampling weight to form a joint weight, and set a prediction confidence degradation mechanism. Binocular just-perceptible difference consistency control can be achieved through left and right eye corresponding region mapping, cross-eye distortion difference calculation, just-perceptible difference threshold adaptive to brightness and texture complexity, and quantization parameter adjustment for the deteriorated side. Multi-objective constrained closed-loop quantization control can unify spherical global distortion, cross-eye consistency, and gaze area distortion for encoding result verification, and form a deterministic closed loop by adjusting and re-encoding the quantization parameters for each image group. For ERP projection, the continuity of spherical meridian seams can also be protected by left and right boundary mirror strip extension and cross-boundary motion estimation and compensation. The aforementioned technical features work together to enable the allocation of encoding resources for binocular panoramic video to simultaneously consider the contribution of spherical real sampling, the user's attention area, and the quality consistency of the corresponding areas of the left and right eyes. Within a standard-compatible encoding framework, this results in high-ratio compression, high image quality in the gaze area, low binocular viewing discomfort, and reproducible and auditable encoding results.

[0074] This application also relates to an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the binocular panoramic video compression encoding method as described above.

[0075] Figure 7 This is a schematic diagram of the structure of the electronic device shown in this application, such as... Figure 7 As shown, the memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which also receives and transmits data to the processor. The processor manages the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor during operation.

[0076] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application.

Claims

1. A binocular panoramic video compression encoding method, characterized in that, include: Acquire the binocular panoramic video to be encoded and the corresponding gaze direction information, wherein the binocular panoramic video includes a left-eye video and a right-eye video; For the spatial unit in the binocular panoramic video, the spherical sampling weight is determined according to the spherical direction corresponding to the spatial unit, the gaze weight is determined according to the spherical direction corresponding to the spatial unit and the gaze direction information, and the joint weight of the spatial unit is determined based on the spherical sampling weight and the gaze weight. Based on the joint weights, the basic quantization parameters of the spatial units of the left-eye video and the right-eye video are determined respectively. Based on the basic quantization parameters, the left-eye video and the right-eye video are encoded respectively. Based on the encoding and reconstruction results of the spatial units of the left-eye video and the right-eye video, the compression distortion of the spatial units of the left-eye video, the compression distortion of the spatial units of the right-eye video, and the cross-eye distortion difference between them are determined. Determine the just-perceptible difference threshold corresponding to the spatial unit of the left-eye video and the spatial unit of the right-eye video, and determine the current quantization parameter of the spatial unit of the left-eye video and the spatial unit of the right-eye video based on the comparison result of the cross-eye distortion difference and the just-perceptible difference threshold; Perform constraint verification on the encoding result obtained according to the current quantization parameters. The constraint verification includes: spherical global distortion constraint determined based on the spherical sampling weight, cross-eye consistency constraint for the spatial units of the corresponding left-eye video and the spatial units of the right-eye video, and gaze region distortion constraint determined based on the gaze direction information. When at least one of the constraints fails to meet the corresponding constraint conditions, the quantization parameters of the corresponding spatial unit are adjusted according to the unmet constraint, and encoding is performed again based on the adjusted quantization parameters.

2. The binocular panoramic video compression coding method according to claim 1, characterized in that, The step of determining the spherical sampling weight based on the spherical orientation corresponding to the spatial unit includes: Based on the spherical orientation, determine the spherical sampling area corresponding to the spatial unit in the projected image; The spherical sampling weight of the spatial unit is determined based on the spherical sampling area, so that the spherical sampling weight represents the actual sampling proportion of the spatial unit in the spherical space; The basic distortion corresponding to each spatial unit is weighted based on the spherical sampling weight to determine the spherical weighted distortion corresponding to the spherical global distortion constraint.

3. The binocular panoramic video compression coding method according to claim 2, characterized in that, The binocular panoramic video uses equidistant cylindrical projection; For the image located at the first The spatial unit within the latitude zone will be the first Latitude corresponding to the center of each latitude zone Determined as: ； in, The height of the projected image; Based on the relationship between the spherical sampling area corresponding to the spatial unit and the cosine of latitude, the area located at the first... The spherical sampling weights of the spatial units within each latitudinal band Determined as: ； The weighting of the basic distortion corresponding to each spatial unit based on the spherical sampling weight includes: The spherical sampling weights corresponding to each latitude zone are used sequentially to weight the basic distortion corresponding to that latitude zone.

4. The binocular panoramic video compression encoding method according to claim 1, characterized in that, The gaze direction information is obtained through the following methods: Continuously acquire head posture data, predict the visual axis direction within a preset time window based on the head posture data, and use the predicted visual axis direction as the gaze direction information. The step of determining the gaze weight based on the spherical orientation corresponding to the spatial unit and the gaze direction information includes: For the spatial unit, the gaze weight is determined based on the eccentricity angle between the spherical direction corresponding to the spatial unit and the visual axis direction, wherein the gaze weight decreases as the eccentricity angle increases.

5. The binocular panoramic video compression coding method according to claim 4, characterized in that, The gaze weight is determined according to the following formula: ； in, For the gaze weight, The eccentricity angle between the spherical direction and the predicted visual axis direction is... The central concave half-angle corresponding to the high-resolution fixation region. The decay order; The joint weights are determined according to the following formula: ； in, Indicates the spherical direction corresponding to the spatial unit u. The corresponding spherical sampling weights, This indicates the eccentricity angle corresponding to the spatial unit u. The corresponding gaze weight.

6. The binocular panoramic video compression coding method according to claim 5, characterized in that, The step of determining the basic quantization parameters of the spatial units of the left-eye video and the right-eye video respectively based on the joint weights includes: in, As the baseline quantization parameter, The joint weights corresponding to the spatial unit u are: The average of the joint weights of the multiple spatial units. It is the bitrate sensitivity coefficient, and .

7. The binocular panoramic video compression coding method according to claim 1, characterized in that, The spatial units of the left-eye video and the spatial units of the right-eye video are determined in the following manner: The directional positions of the spatial units in the left-eye video and the right-eye video on the sphere are determined respectively. Based on the directional positions and combined with the disparity estimation results or the spherical reprojection alignment results, the spatial units of the left-eye video and the right-eye video corresponding to the same spherical region are determined. The cross-eye distortion difference is determined as follows: in, The compression distortion of the spatial unit u in the left-eye video. The compression distortion of the spatial unit u in the right eye video; The just-perceptible difference threshold is determined based on the brightness characteristics and texture complexity characteristics of the spatial unit, and satisfies: in, The difference can be detected precisely based on the foundation. The brightness statistics of the spatial unit u. Let be the texture complexity statistic for the spatial unit u. Let be the adjustment function for brightness with respect to the threshold of just-perceptible difference. This is a function that modulates texture complexity with respect to the threshold of just-perceptible differences.

8. The binocular panoramic video compression coding method according to claim 1, characterized in that, The step of determining the current quantization parameters of the spatial unit of the left-eye video and the spatial unit of the right-eye video based on the comparison result of the cross-eye distortion difference and the just-perceptible difference threshold includes: When the cross-eye distortion difference exceeds the just-perceptible difference threshold, and the compression distortion of the spatial unit of the left-eye video is greater than the compression distortion of the spatial unit of the right-eye video, a quantization parameter less than the basic quantization parameter is determined as the current quantization parameter of the spatial unit of the left-eye video, and a quantization parameter greater than or equal to the basic quantization parameter is determined as the current quantization parameter of the spatial unit of the right-eye video. When the cross-eye distortion difference exceeds the just-perceptible difference threshold, and the compression distortion of the spatial unit of the right-eye video is greater than the compression distortion of the spatial unit of the left-eye video, a quantization parameter less than the basic quantization parameter is determined as the current quantization parameter of the spatial unit of the right-eye video, and a quantization parameter greater than or equal to the basic quantization parameter is determined as the current quantization parameter of the spatial unit of the left-eye video.

9. The binocular panoramic video compression coding method according to claim 1, characterized in that, The constraint verification is performed on a group of images as the processing unit. After encoding a group of images, it is determined whether the spherical global distortion of the group of images satisfies the spherical global distortion constraint, whether there are spatial units in the group of images where the cross-eye distortion difference does not satisfy the cross-eye consistency constraint, and whether the gaze region distortion of the group of images satisfies the gaze region distortion constraint. When at least one of the constraints is not satisfied, the quantization parameters of the spatial units corresponding to the unsatisfied constraints are adjusted in the order of priority of cross-eye consistency constraints, gaze region distortion constraints, and spherical global distortion constraints, and the image group is re-encoded based on the adjusted quantization parameters.

10. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the binocular panoramic video compression coding method as described in any one of claims 1 to 9.

Images