Panoramic video encoding method, apparatus, and electronic device

Abstract

Embodiments of the present application relate to the technical field of panoramic video, and particularly relate to a panoramic video encoding method, device and electronic equipment. The method comprises: determining a first video frame group of a panoramic video stream according to a group of pictures (GOP) parameter; determining a motion vector of each coding unit of an i-th and an i+1-th video frame based on inter-frame prediction; determining a first coding unit group and a second coding unit group according to a motion vector variation of corresponding coding units in the i-th and the i+1-th video frames; determining whether a user FOV and the GOP parameter need to be adjusted according to a coding unit size of the first coding unit group and the second coding unit group; if the user FOV and the GOP parameter need to be adjusted, obtaining user head pose data, and adjusting the user FOV and the GOP parameter according to the head pose data. Embodiments of the present application can effectively improve the viewing quality of a user while changing the user's viewing angle.

Description

[Technical Field]

[0001] The present invention relates to the field of panoramic video technology, and in particular to a panoramic video encoding method, apparatus and electronic device. [Background Technology]

[0002] In virtual reality (VR) panoramic live streaming scenarios with variable field of view (FOV), for high-precision video stream encoding, a relatively large fixed group of pictures (GOP) length can be selected and used for encoding in subsequent live streams. However, this method incurs significant loading latency, impacting the user's viewing quality. Alternatively, a shorter GOP-length video stream can be added to the fixed GOP-length video stream, temporarily switching to the shorter GOP-length stream when the user's viewpoint changes. However, this method adds additional decoding complexity, resulting in reduced decoding efficiency.

[0003] Therefore, during panoramic video live streaming, how to ensure the viewing experience of users while encoding panoramic video streams has changed has become an urgent problem to be solved. [Summary of the Invention]

[0004] This invention provides a panoramic video encoding method, apparatus, and electronic device. This invention can effectively shorten the loading time of high-precision video segments after the user changes their viewing angle in dynamic scenes without increasing the complexity of multi-channel encoding and decoding, thus improving the viewing quality.

[0005] In a first aspect, embodiments of the present invention provide a panoramic video encoding method, including:

[0006] Based on the GOP parameter, the first video frame group of the panoramic video stream is determined. The first video frame group contains M video frames, and each video frame corresponds to N coding units. M and N are both integers greater than 2.

[0007] Based on inter-frame prediction, determine the motion vector of each coding unit in the i-th and (i+1)-th video frames of the M video frames, where i is greater than 2 and less than M;

[0008] Based on the change in motion vector of the coding unit corresponding to the position in the i-th and i+1-th video frames, determine the first coding unit group with the largest motion amplitude within the user's FOV range and the second coding unit group with the largest motion amplitude outside the user's FOV range among the N coding units.

[0009] Based on the coding unit sizes of the first coding unit group and the second coding unit group, determine whether the user FOV and the GOP parameters need to be adjusted;

[0010] If it is necessary to adjust the user's FOV and the GOP parameter, then obtain the user's head pose data and adjust the user's FOV and the GOP parameter according to the head pose data.

[0011] In one possible implementation, determining the motion vector of any coding unit in the i-th video frame based on inter-frame prediction includes:

[0012] Predict the first coding unit of the (i-1)th video frame from the starting unit corresponding to the i-th video frame;

[0013] Starting from the starting unit, the search unit is determined in the i-th video frame according to the search step size set by the diamond search template, and the search unit with the minimum rate distortion cost is determined as the target search unit.

[0014] If the search distance between the target search unit and the starting unit is one search step, then the two unsearched units with the closest search distance to the target search unit are determined; the unit with the lowest rate-distortion cost among the target search unit and the two unsearched units is determined as the candidate search unit.

[0015] If the search distance between the target search unit and the starting unit exceeds a preset number of search steps, then a global search is performed on the i-th video frame with the target search unit as the center, and the unit with the lowest rate-distortion cost is determined as the candidate search unit.

[0016] Starting from the candidate search unit, repeat the above search steps until the candidate search units determined in the previous two attempts are consistent, and determine the candidate search unit that is consistent in the previous two attempts as the prediction unit of the first encoding unit.

[0017] Based on the first encoding unit and the prediction unit, the motion vector corresponding to the prediction unit in the i-th video frame is determined.

[0018] In one possible implementation, determining the first group of coding units with the largest motion amplitude within the user's field of view (FOV) and the second group of coding units with the largest motion amplitude outside the user's FOV, based on the motion vector changes of the coding units corresponding to the positions in the i-th and i+1 video frames, includes:

[0019] Based on the change in motion vector of the coding unit corresponding to the position in the i-th video frame and the i+1 video frames, determine the first coding unit with the largest motion amplitude that is within the user's FOV range and the second coding unit with the largest motion amplitude that is outside the user's FOV range among the N coding units.

[0020] Based on the motion vectors of the first coding unit and the second coding unit, determine the first coding unit group with the largest motion amplitude within the user's FOV range and the second coding unit group with the largest motion amplitude outside the user's FOV range.

[0021] In one possible implementation, based on the change in motion vectors of the coding units corresponding to their positions in the i-th video frame and the i+1 video frames, a first coding unit with the largest motion amplitude located within the user's field of view (FOV) and a second coding unit with the largest motion amplitude located outside the user's FOV are determined from among the N coding units, including:

[0022] Calculate the mean of the motion vector magnitudes of N coding units in the i-th video frame and the mean of the motion vector magnitudes of N coding units in the (i+1)-th video frame;

[0023] Calculate the ratio of the magnitude of the N motion vectors in the (i+1)th video frame to the average magnitude of the motion vectors in the ith video frame, and obtain the N magnitude ratios and combine them into a ratio array;

[0024] The ratio array is divided into an array within the user's FOV range and an array outside the user's FOV range;

[0025] The largest ratio is obtained from the array within the user's FOV range and the array outside the user's FOV range, and the encoding unit corresponding to the largest ratio between the user's FOV range and the array outside the user's FOV range is determined as the first encoding unit and the second encoding unit.

[0026] In one possible implementation, determining the first group of coding units with the largest motion amplitude within the user's field of view (FOV) and the second group of coding units with the largest motion amplitude outside the user's FOV, based on the motion vectors of the first coding unit and the second coding unit, includes:

[0027] Determine at least one coding unit that has the same motion vector as the first coding unit, and form a first coding unit group with the first coding unit that has the largest motion amplitude within the user's field of view;

[0028] At least one coding unit with the same motion vector as the second coding unit is identified, and together with the second coding unit, they form a second coding unit group with the largest motion amplitude outside the user's FOV range.

[0029] In one possible implementation, determining whether the user FOV and GOP parameters need to be adjusted based on the coding unit sizes of the first coding unit group and the second coding unit group includes:

[0030] The coding unit size of the first coding unit group and the second coding unit group is determined based on the number of coding units contained in the first coding unit group and the second coding unit group;

[0031] Compare the coding unit sizes contained in the first coding unit group and the second coding unit group;

[0032] If the coding unit size of the first coding unit group is smaller than the coding unit size of the second coding unit group, then it is determined that the user needs to adjust the user FOV and the GOP parameter.

[0033] Otherwise, it is determined that the user does not need to adjust the user FOV and the GOP parameter.

[0034] In one possible implementation, if it is necessary to adjust the user's FOV and the GOP parameters, then obtaining the user's head pose data and adjusting the user's FOV and the GOP parameters based on the head pose data includes:

[0035] If it is necessary to adjust the user's FOV and the GOP parameters, then obtain the user's head attitude data, which includes roll angle, pitch angle and yaw angle.

[0036] The user's predicted viewpoint is determined based on the roll angle, the pitch angle, and the yaw angle;

[0037] Determine whether the first encoding unit falls within the visible range of the predicted viewpoint. If so, there is no need to adjust the user FOV and the GOP parameters.

[0038] Otherwise, the user's FOV is adjusted based on the predicted viewpoint, and the GOP parameters are also adjusted.

[0039] The embodiments of the present invention can effectively shorten the loading time of high-precision video segments after the user changes the viewpoint in a dynamic scene without increasing the complexity of multi-channel encoding and decoding, thereby improving the viewing quality.

[0040] In a second aspect, embodiments of the present invention provide a panoramic video encoding device, comprising:

[0041] The first determining module is used to determine the first video frame group of the panoramic video stream according to the GOP parameter. The first video frame group contains M video frames, and each video frame corresponds to N coding units. M and N are both integers greater than 2.

[0042] The second determining module is used to determine the motion vector of each coding unit in the i-th video frame and the (i+1)-th video frame among the M video frames based on inter-frame prediction, wherein i is greater than 2 and less than M;

[0043] The third determining module is used to determine, based on the change in motion vector of the coding unit corresponding to the position in the i-th and i+1 video frames, the first coding unit group that is within the user's FOV and has the largest motion amplitude, and the second coding unit group that is outside the user's FOV and has the largest motion amplitude among the N coding units.

[0044] The fourth determining module is used to determine whether the user FOV and the GOP parameters need to be adjusted based on the coding unit sizes of the first coding unit group and the second coding unit group.

[0045] The adjustment module is used to acquire user head pose data and adjust the user FOV and GOP parameters according to the head pose data if the user FOV and GOP parameters need to be adjusted.

[0046] In one possible implementation, the second determining module is used to predict the starting unit corresponding to the first coding unit of the (i-1)th video frame in the i-th video frame.

[0047] Starting from the starting unit, the search unit is determined in the i-th video frame according to the search step size set by the diamond search template, and the search unit with the minimum rate distortion cost is determined as the target search unit.

[0048] If the search distance between the target search unit and the starting unit is one search step, then the two unsearched units with the closest search distance to the target search unit are determined; the unit with the lowest rate-distortion cost among the target search unit and the two unsearched units is determined as the candidate search unit.

[0049] If the search distance between the target search unit and the starting unit exceeds a preset number of search steps, then a global search is performed on the i-th video frame with the target search unit as the center, and the unit with the lowest rate-distortion cost is determined as the candidate search unit.

[0050] Starting from the candidate search unit, repeat the above search steps until the candidate search units determined in the previous two attempts are consistent, and determine the candidate search unit that is consistent in the previous two attempts as the prediction unit of the first encoding unit.

[0051] Based on the first encoding unit and the prediction unit, the motion vector corresponding to the prediction unit in the i-th video frame is determined.

[0052] In one possible implementation, the third determining module is specifically used to determine, based on the change in motion vector of the coding units corresponding to the positions in the i-th video frame and the i+1 video frames, the first coding unit that is within the user's FOV and has the largest motion amplitude, and the second coding unit that is outside the user's FOV and has the largest motion amplitude among the N coding units.

[0053] Based on the motion vectors of the first coding unit and the second coding unit, determine the first coding unit group with the largest motion amplitude within the user's FOV range and the second coding unit group with the largest motion amplitude outside the user's FOV range.

[0054] In one possible implementation, the third determining module is further specifically used to calculate the average value of the motion vector magnitude of N coding units in the i-th video frame and the motion vector magnitude of N coding units in the (i+1)-th video frame.

[0055] Calculate the ratio of the magnitude of the N motion vectors in the (i+1)th video frame to the average magnitude of the motion vectors in the ith video frame, and obtain the N magnitude ratios and combine them into a ratio array;

[0056] The ratio array is divided into an array within the user's FOV range and an array outside the user's FOV range;

[0057] The largest ratio is obtained from the array within the user's FOV range and the array outside the user's FOV range, and the encoding unit corresponding to the largest ratio between the user's FOV range and the array outside the user's FOV range is determined as the first encoding unit and the second encoding unit.

[0058] In one possible implementation, the third determining module is further specifically used to determine at least one coding unit that has the same motion vector as the first coding unit, and to form a first coding unit group with the largest motion amplitude within the user's FOV range with the first coding unit;

[0059] At least one coding unit with the same motion vector as the second coding unit is identified, and together with the second coding unit, they form a second coding unit group with the largest motion amplitude outside the user's FOV range.

[0060] In one possible implementation, the fourth determining module is specifically used to determine the coding unit size of the first coding unit group and the second coding unit group based on the number of coding units contained in the first coding unit group and the second coding unit group;

[0061] Compare the coding unit sizes contained in the first coding unit group and the second coding unit group;

[0062] If the coding unit size of the first coding unit group is smaller than the coding unit size of the second coding unit group, then it is determined that the user needs to adjust the user FOV and the GOP parameter.

[0063] Otherwise, it is determined that the user does not need to adjust the user FOV and the GOP parameter.

[0064] In one possible implementation, the adjustment module is specifically used to obtain user head attitude data if it is necessary to adjust the user FOV and the GOP parameters. The user head attitude data includes roll angle, pitch angle and yaw angle.

[0065] The user's predicted viewpoint is determined based on the roll angle, the pitch angle, and the yaw angle;

[0066] Determine whether the first encoding unit falls within the visible range of the predicted viewpoint. If so, there is no need to adjust the user FOV and the GOP parameters.

[0067] Otherwise, the user's FOV is adjusted based on the predicted viewpoint, and the GOP parameters are also adjusted.

[0068] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0069] At least one processor; and

[0070] At least one memory communicatively connected to the processor, wherein:

[0071] The memory stores program instructions that can be executed by the processor, and the processor can execute the method provided in the first aspect by calling the program instructions.

[0072] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that cause the computer to perform the method provided in the first aspect.

[0073] It should be understood that the second to fourth aspects of this specification are consistent with the technical solutions of the first aspect of this specification, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, so they will not be described again. [Attached Image Description]

[0074] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A flowchart of a panoramic video encoding method provided in an embodiment of the present invention;

[0076] Figure 2 A schematic diagram of a motion estimation search algorithm provided in an embodiment of the present invention;

[0077] Figure 3 A schematic diagram of a first coding unit and a second coding unit provided for an embodiment of the present invention;

[0078] Figure 4 A schematic diagram of the first coding unit group and the second coding unit group provided in an embodiment of the present invention;

[0079] Figure 5 This is a schematic diagram of the structure of a panoramic video encoding device provided in an embodiment of the present invention;

[0080] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention.

Detailed Implementation Methods

[0081] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0082] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of this specification.

[0083] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0084] In VR panoramic live streaming scenarios with variable viewpoints, current technical solutions for high-precision video stream encoding have the following drawbacks: One is the use of a relatively large fixed GOP length for encoding. When the user changes their viewpoint, the high-precision video segments experience significant loading delays due to waiting for IDR frames, preventing the user from viewing the high-precision video segments in a timely manner and affecting viewing quality. Another issue is the need to encode two video streams simultaneously, increasing encoding coupling and reducing system reliability. Furthermore, decoding also requires decoding two video streams with different GOP lengths. Since the two video streams use different encoding parameters, this adds extra difficulty to decoding and reduces decoding efficiency. This invention provides a panoramic video encoding method that effectively solves the above problems.

[0085] Figure 1 A flowchart of a panoramic video encoding method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0086] Step 101: Determine the first video frame group of the panoramic video stream according to the image group GOP parameters. The first video frame group contains M video frames, and each video frame corresponds to N coding units. M and N are both integers greater than 2.

[0087] In some embodiments, during the panoramic video live streaming initialization process, an initial Group of Pictures (GOP) is first determined, and the first video frame group is determined based on the GOP length. To ensure controllable random access time in the worst-case scenario, the maximum GOP length is set to a value equivalent to the video frame rate (Frames per Second, FPS), guaranteeing at least one IDR frame per second, and limiting the frame type to I-frames and P-frames. Taking a sample video frame sequence with a length of 1 second and a frame rate of 30fps as an example, the frame sequence S encoded by the maximum GOP length... max Represented as: S max ={IPPPPPPPPPPPPPPPPPPP PP PP PPPPP}. When using the maximum GOP length encoding, the maximum random access loading time is network latency (Round-Trip Time, RTT) + 1 second, and the minimum is RTT. The minimum GOP length is 1, and its video sequence S min Represented as: S min = {II II IIIIIIIIIIIIIIIIIIIIIIII IIIII}, which is full IDR frame encoding. When using the minimum GOP length for encoding, the only remaining random access loading time is network latency (RTT). Since full I-frame encoding has a lower compression ratio, it is necessary to select an appropriate GOP length for encoding based on the actual situation of the video.

[0088] Step 102: Based on inter-frame prediction, determine the motion vector of each coding unit in the i-th video frame and the (i+1)-th video frame among the M video frames, where i is greater than 2 and less than M.

[0089] In some embodiments, after determining the first video frame group, inter-frame prediction is performed on the video frames of the first video group. Specifically, a motion estimation search algorithm is used to determine the motion vector of each coding unit of the video frame.

[0090] Specifically, the first coding unit of the (i-1)th video frame is predicted to start at the corresponding starting unit in the i-th video frame; starting from the starting unit, a search unit is determined in the i-th video frame according to the search step size set by the diamond search template, and the search unit with the minimum rate-distortion cost is determined as the target search unit; if the search distance between the target search unit and the starting unit is one search step, the two unsearched units closest to the target search unit are determined; the unit with the minimum rate-distortion cost is determined from the target search unit and the two unsearched units as the candidate search unit; if the search distance between the target search unit and the starting unit exceeds a preset number of search steps, a global search is performed in the i-th video frame centered on the target search unit, and the unit with the minimum rate-distortion cost is determined as the candidate search unit; the above search steps are repeated starting from the candidate search unit until the candidate search units determined in the previous two searches are consistent, and the candidate search units that are consistent in the previous two searches are determined as the prediction unit of the first coding unit; based on the first coding unit and the prediction unit, the motion vector corresponding to the prediction unit in the i-th video frame is determined.

[0091] Advanced Motion Vector Prediction (AMVP) is used to determine the starting unit corresponding to the first coding unit in the (i-1)th video frame within the i-th video frame. First, the reference block of the first coding unit in the (i-1)th video frame is determined. Then, the distance between the (i-1)th video frame containing the first coding unit and the i-th video frame, the distance between the video frame containing the reference block and the reference video frame, and the motion vector of the reference block are determined. The candidate motion vector of the first coding unit is calculated using the following formula:

[0092]

[0093] Where td is the distance from the (i-1)th video frame to the ith video frame, tb is the distance between the video frame containing the reference block and its reference video frame, colMV is the motion vector of the reference block, and curMV is the candidate motion vector of the first coding unit.

[0094] Finally, the starting unit on the i-th video frame is determined based on the candidate motion vector of the first coding unit.

[0095] Figure 2 This is a schematic diagram of a motion estimation search algorithm provided in an embodiment of the present invention. First, starting from the starting cell, a step size of 1 is used to... Figure 2The diamond-shaped template shown searches within a range, with step sizes increasing in powers of 2, selecting the search unit with the minimum rate-distortion cost as the target search unit. Then, the search distance between the target search unit and the starting unit is determined, and different search processes are performed based on the magnitude of the search distance. If the search distance is 1, the two unsearched units closest to the target search unit are selected, and their rate-distortion costs are compared with those of the target search unit; the unit with the minimum rate-distortion cost is selected as the candidate search unit. If the search distance exceeds a preset threshold, a global search is performed within the i-th video frame centered on the target search unit, and the unit with the minimum rate-distortion cost is determined as the candidate search unit.

[0096] Then, the search steps described above are repeated, centering on the candidate search unit, until the candidate search units obtained from two adjacent searches are consistent. The search is then stopped, and the candidate search units that are consistent in two adjacent searches are determined as the prediction units. Based on the first coding unit of the (i-1)th video frame and the prediction unit of the ith video frame, the final motion vector of the prediction unit in the ith video frame can be determined.

[0097] Step 103: Based on the change in motion vector of the coding unit corresponding to the position in the i-th and i+1 video frames, determine the first coding unit group with the largest motion amplitude that is located within the user's field of view (FOV) and the second coding unit group with the largest motion amplitude that is located outside the user's FOV.

[0098] In some embodiments, the motion vectors of all coding units within the i-th and (i+1)-th video frames are determined according to the motion estimation search algorithm of step 102. Then, quantization analysis is performed based on the changes in the motion vectors of the coding units corresponding to the two video frames and the spatial attributes of the coding units.

[0099] Specifically, based on the change in motion vectors of the coding units corresponding to the positions in the i-th video frame and the i+1 video frames, a first coding unit with the largest motion amplitude within the user's FOV and a second coding unit with the largest motion amplitude outside the user's FOV are determined from among the N coding units; based on the motion vectors of the first coding unit and the second coding unit, a first coding unit group with the largest motion amplitude within the user's FOV and a second coding unit group with the largest motion amplitude outside the user's FOV are determined.

[0100] The first step is to determine the coding units within the user's field of view (FOV) and the coding units within the user's FOV based on the motion vector changes of the corresponding coding units in the i-th and i+1-th video frames.

[0101] Specifically, the mean of the motion vector magnitudes of N coding units in the i-th video frame and the mean of the motion vector magnitudes of N coding units in the (i+1)-th video frame are calculated; the ratio of the N motion vector magnitudes in the (i+1)-th video frame to the mean of the motion vector magnitudes in the i-th video frame is calculated to obtain N magnitude ratios, which are then combined into a ratio array; the ratio array is divided into an array within the user's field of view (FOV) and an array outside the user's FOV; the largest ratio is obtained from the array within the user's FOV and the array outside the user's FOV, respectively, and the coding unit corresponding to the largest ratio within and outside the user's FOV is determined as the first coding unit and the second coding unit.

[0102] The magnitude V of the motion vector MV represents the distance of the temporal transform of the coding unit in the image, used to describe the motion amplitude of that coding unit. The total number of coding units in the i-th video frame is denoted as N, and the motion vector of any coding unit k is denoted as MV. k The average motion vector magnitude of all coding units within the i-th video frame is obtained by weighted averaging. The average magnitude of the motion vector is used to describe the overall motion of the frame. The calculation formula is as follows:

[0103]

[0104] Meanwhile, the (i+1)th video frame contains N coding units, and the MVP of coding unit k is... k After the vector is calculated, its magnitude VP is... k The average magnitude of the motion vector of the i-th intra-frame coding unit Open comparison yields VP k and The ratio RP k RP k The calculation formula is as follows:

[0105]

[0106] Calculate the RP of all coding units in the (i+1)th video frame in sequence. k Receive N RPs k Then, the above N RPs k The array RP is composed as follows:

[0107] RP = {RP1, RP2, RP3...RP} N}

[0108] The array RP is arranged according to the position of the encoding unit k relative to the user's FOV. kThey are divided into two groups: the user's FOV range outside the array RP. outer and the array RP within the user's FOV range inner The user's FOV boundaries are FOV. min and FOV max Let k1∈{1,N} and FOV k1 <FOV min k2∈{1,N} and FOV k2 >FOV max Then RP outer and RP inner The expression is as follows:

[0109] RP outer ={RP1...RP k1}∪{RP k2 ...RP N}

[0110] RP inner ={RP k1+1 ...RP k2-1}

[0111] For RP outer and RP inner Obtain the maximum value max(RP) respectively outer ) and max(RP inner ), where max(RP) outer ) and max(RP inner The expression for ) is as follows:

[0112] max(RP outer =argmax(argmax({RP1...RP k1}), argmax({RP k2 ...RP N}))

[0113] max(RP inner =argmax({RP k1+1 ...RP k2-1})

[0114] Where max(RP) inner ) and max(RP outer The corresponding coding unit is the first coding unit and the second coding unit. Figure 3 This is a schematic diagram of the first and second coding units provided in an embodiment of the present invention, max(RP) inner ) and max(RP outer The corresponding coding unit is as follows: Figure 3 As shown.

[0115] Due to perspective, objects closer to the camera appear larger than those farther away in a video frame (the law of perspective). In actual video footage, moving objects that are important subjects are often closer to the camera, thus covering more encoding units on the screen. For example... Figure 3 As shown, only the coding unit with the largest motion amplitude inside and outside the user's FOV is calculated. If the entire image range of the moving subject is not covered, it is insufficient to fully describe the changing trend of points of interest inside and outside the field of view. Therefore, the image space size dimension of the coding unit is introduced, and the first coding unit group and the second coding unit group are determined based on the first coding unit and the second coding unit.

[0116] Specifically, at least one coding unit with the same motion vector as the first coding unit is determined, and together with the first coding unit, they form a first coding unit group with the largest motion amplitude within the user's FOV range; at least one coding unit with the same motion vector as the second coding unit is determined, and together with the second coding unit, they form a second coding unit group with the largest motion amplitude outside the user's FOV range.

[0117] Because inter-frame coding uses a depth-first approach to encode coding units sequentially, and the main subject of the image also covers multiple consecutive coding units. Figure 4 This is a schematic diagram of the first coding unit group and the second coding unit provided in an embodiment of the present invention. Based on the above characteristics, we respectively use... Figure 4 Based on the first and second coding units shown, nearby coding units are searched to find at least one coding unit whose motion vector is equal to that of the first coding unit, and then a combination is formed. Figure 4 The first coding unit group and the second coding unit group in the.

[0118] Step 104: Determine whether the user FOV and the GOP parameters need to be adjusted based on the coding unit sizes of the first coding unit group and the second coding unit group.

[0119] In some embodiments, Figure 4 By summing the sizes of all coding units in the first and second coding unit groups, we can obtain the value used to describe max(RP). outer ) and max(RP inner The image spatial size relationship at () is determined. The sizes of the first and second coding unit groups are compared to determine if the user's FOV and GOP parameters need adjustment.

[0120] Specifically, based on the number of coding units contained in the first coding unit group and the second coding unit group, the coding unit sizes of the first coding unit group and the second coding unit group are determined; the coding unit sizes contained in the first coding unit group and the second coding unit group are compared; if the coding unit size of the first coding unit group is smaller than the coding unit size of the second coding unit group, it is determined that the user needs to adjust the user FOV and the GOP parameter; otherwise, it is determined that the user does not need to adjust the user FOV and the GOP parameter.

[0121] Compare Figure 4 The obtained max(RP) inner ) and max(RP outer ), combined with the spatial size relationship of its coding unit groups, if max(RP inner ) less than max(RP) outer ); or max(RP) inner The size of the first coding unit group corresponding to ) is less than max(RP) outer If the size of the second coding unit group corresponding to the current field of view is determined, then it is considered that a point of interest has appeared outside the user's current field of view. At this time, it is determined that the user may change their perspective to look for points of interest outside the FOV range, and the encoder will set the GOP parameter of the encoder to a smaller size for the next period of time.

[0122] Step 105: If it is necessary to adjust the user's FOV and the GOP parameter, then obtain the user's head pose data and adjust the user's FOV and the GOP parameter according to the head pose data.

[0123] In some embodiments, if it is determined that the user's FOV and GOP parameters need to be adjusted, the user's head pose data is analyzed, and the user's viewpoint is predicted. Calibration regression is performed based on the predicted viewpoint, and it is then determined whether the GOP parameters need to be adjusted.

[0124] Specifically, if it is necessary to adjust the user's FOV and the GOP parameter, then user head attitude data is obtained, including roll angle, pitch angle, and yaw angle; the user's predicted viewpoint is determined based on the roll angle, pitch angle, and yaw angle; it is determined whether the first encoding unit falls within the visible range of the predicted viewpoint; if so, it is not necessary to adjust the user's FOV and the GOP parameter; otherwise, the user's FOV and the GOP parameter are adjusted based on the predicted viewpoint.

[0125] The user's attitude data includes angles in three directions: roll, pitch, and yaw. These represent the angles rendered along the Z, X, and Y axes, respectively, with right-hand rotation around the axis being positive. In a head-mounted display, sensors can acquire the VR headset's angles, angular velocities, and angular accelerations. These three angles can be used to describe the user's current viewpoint. Taking the yaw angle as an example, with the current angle θ, angular velocity ω, angular acceleration α, and prediction duration t, the formula for calculating the yaw angle is as follows:

[0126]

[0127] Pitch and roll angles can be calculated using a similar method. The prediction viewpoint is determined based on the yaw, pitch, and roll angles. Figure 4 max(RP) outer If the second coding unit corresponding to the prediction viewpoint falls within the visible range of the prediction viewpoint, then the user's FOV and GOP parameters are adjusted; otherwise, the FOV and GOP parameters remain unchanged.

[0128] This invention utilizes the optimal motion vectors calculated during the encoding process. Based on their characteristics of describing screen motion and combined with the spatial attributes of the encoding units, quantitative analysis is performed, and viewpoint prediction is used to correct and regress the analysis results. This enables dynamic scaling of the GOP length in subsequent frame encoding. It effectively shortens the loading time of high-precision video segments after the user changes their viewpoint in dynamic scenes without increasing the complexity of multi-channel encoding and decoding, thus improving the viewing quality.

[0129] Figure 5 This is a schematic diagram of the structure of a panoramic video encoding device provided in an embodiment of the present invention, as shown below. Figure 5As shown, the panoramic video encoding device 200 described above may include: a first determining module 201, a second determining module 202, a third determining module 203, a fourth determining module 204, and an adjustment module 205. The first determining module 201 is used to determine a first video frame group of the panoramic video stream based on the Group of Pictures (GOP) parameters. The first video frame group contains M video frames, each video frame corresponding to N encoding units, where M and N are both integers greater than 2. The second determining module 202 is used to determine the motion vector of each encoding unit in the i-th and (i+1)-th video frames among the M video frames based on inter-frame prediction, where i is greater than 2 and less than M. The third determining module 203 is used to determine, based on the change in the motion vector of the encoding units corresponding to the positions in the i-th and (i+1)-th video frames, a first encoding unit group located within the user's field of view (FOV) and with the largest motion amplitude, and a second encoding unit group located outside the user's FOV and with the largest motion amplitude among the N encoding units. The fourth determining module 204 is used to determine whether the user FOV and the GOP parameters need to be adjusted based on the coding unit sizes of the first coding unit group and the second coding unit group. The adjusting module 205 is used to acquire user head pose data and adjust the user FOV and the GOP parameters based on the head pose data if the user FOV and the GOP parameters need to be adjusted.

[0130] In the above embodiments of the present invention, optionally, the second determining module 202 is used to predict the starting unit corresponding to the first coding unit of the (i-1)th video frame in the i-th video frame; using the starting unit as the starting point, determine the search unit in the i-th video frame according to the search step size set by the diamond search template, and determine the search unit with the minimum rate-distortion cost as the target search unit; if the search distance between the target search unit and the starting unit is one search step size, then determine the two unsearched units with the closest search distance to the target search unit; determine the rate-distortion cost from the target search unit and the two unsearched units. The unit with the lowest cost is selected as the candidate search unit. If the search distance between the target search unit and the starting unit exceeds a preset number of search steps, a global search is performed on the i-th video frame centered on the target search unit, and the unit with the lowest rate-distortion cost is selected as the candidate search unit. The above search steps are repeated starting from the candidate search unit until the candidate search units determined in the previous two searches are consistent, and the candidate search units that are consistent in the previous two searches are selected as the prediction unit of the first coding unit. Based on the first coding unit and the prediction unit, the motion vector corresponding to the prediction unit in the i-th video frame is determined. The first processing module is configured to encode the i-th frame image if the current i-th frame image is one of the first X predicted frames of the same group of images (GOP), and save the encoding residual information of the i-th frame image, where X is a preset known value; if the current i-th frame image is a predicted frame after one of the first X predicted frames of the same GOP, then the i-th frame image or the k-th future frame after the i-th frame image is determined as the target frame, where i takes the value 1, 2...Y, Y is the number of image frames contained in the GOP, and k takes the value 1, 2...Y-1.

[0131] In the above embodiments of the present invention, optionally, the third determining module 203 is specifically used to determine, based on the change in motion vectors of the corresponding coding units in the i-th video frame and the i+1 video frames, a first coding unit with the largest motion amplitude within the user's field of view (FOV) and a second coding unit with the largest motion amplitude outside the user's FOV; and based on the motion vectors of the first coding unit and the second coding unit, to determine a group of first coding units with the largest motion amplitude within the user's FOV and a group of second coding units with the largest motion amplitude outside the user's FOV. The second processing module is used to encode using the quantization parameter offset of each macroblock in the target frame; and to update the coding residual information of the m associated frames.

[0132] In the above embodiments of the present invention, optionally, the third determining module 203 is further configured to calculate the average value of the motion vector magnitude of N coding units in the i-th video frame and the motion vector magnitude of N coding units in the (i+1)-th video frame; calculate the ratio of the N motion vector magnitudes in the (i+1)-th video frame to the average value of the motion vector magnitude of the i-th video frame, obtain N magnitude ratios and combine them into a ratio array; divide the ratio array into an array within the user's FOV range and an array outside the user's FOV range; obtain the largest ratio from the array within the user's FOV range and the array outside the user's FOV range respectively, and determine the coding unit corresponding to the largest ratio within and outside the user's FOV range as the first coding unit and the second coding unit.

[0133] In the above embodiments of the present invention, optionally, the third determining module 203 is further configured to determine at least one coding unit with the same motion vector as the first coding unit, and form a first coding unit group with the largest motion amplitude within the user's FOV range with the first coding unit; and determine at least one coding unit with the same motion vector as the second coding unit, and form a second coding unit group with the largest motion amplitude outside the user's FOV range with the second coding unit.

[0134] In the above embodiments of the present invention, optionally, the fourth determining module 204 is specifically used to determine the coding unit size of the first coding unit group and the second coding unit group based on the number of coding units contained in the first coding unit group and the second coding unit group; compare the coding unit sizes contained in the first coding unit group and the second coding unit group; if the coding unit size of the first coding unit group is smaller than the coding unit size of the second coding unit group, then it is determined that the user needs to adjust the user FOV and the GOP parameter; otherwise, it is determined that the user does not need to adjust the user FOV and the GOP parameter.

[0135] In the above embodiments of the present invention, optionally, the adjustment module 205 is specifically configured to: if it is necessary to adjust the user's FOV and the GOP parameters, acquire user head posture data, the user's head posture data including roll angle, pitch angle and yaw angle; determine the user's predicted viewpoint based on the roll angle, the pitch angle and the yaw angle; determine whether the first encoding unit falls within the visible range of the predicted viewpoint; if so, it is not necessary to adjust the user's FOV and the GOP parameters; otherwise, adjust the user's FOV and the GOP parameters based on the predicted viewpoint.

[0136] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention.

[0137] like Figure 6As shown, the electronic device described above may include at least one processor; and at least one memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor can execute this specification by calling the program instructions. Figure 1 The panoramic video encoding method provided in the illustrated embodiment.

[0138] The aforementioned electronic device can be a device capable of recognizing user gestures, such as a cloud server. This specification does not limit the specific form of the electronic device in the embodiments. It is understood that the electronic device here refers to the machine mentioned in the method embodiments.

[0139] Figure 6 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present invention is shown. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0140] like Figure 6 As shown, the electronic device is represented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, a communication interface 420, a memory 430, and a communication bus 440 connecting different system components (including the memory 430 and the processing unit 410).

[0141] Communication bus 440 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0142] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0143] Memory 430 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0144] A program / utility having a set (at least one) of program modules can be stored in memory 430. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of the present invention.

[0145] Processor 410 executes various functional applications and data processing by running programs stored in memory 430, such as implementing the present invention. Figure 1 The panoramic video encoding method provided in the illustrated embodiment.

[0146] This invention provides a computer-readable storage medium storing computer instructions that cause a computer to execute the present specification. Figure 1 The panoramic video encoding method provided in the illustrated embodiment.

[0147] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0148] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0149] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0150] Computer program code for performing the operations described herein can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0152] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0153] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this specification, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0154] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this specification includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which the embodiments of this specification pertain.

[0155] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0156] It should be noted that the terminals involved in the embodiments of this specification may include, but are not limited to, personal computers (PCs), personal digital assistants (PDAs), wireless handheld devices, tablet computers, mobile phones, MP3 players, MP4 players, etc.

[0157] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0158] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0159] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A panoramic video encoding method, characterized in that, include: Based on the Group of Pictures (GOP) parameters, the first video frame group of the panoramic video stream is determined. The first video frame group contains M video frames, and each video frame corresponds to N coding units, where M and N are both integers greater than 2. Based on inter-frame prediction, determine the motion vector of each coding unit in the i-th and (i+1)-th video frames of the M video frames, where i is greater than 2 and less than M; Based on the change in motion vector of the coding unit corresponding to the position in the i-th and i+1-th video frames, determine the first coding unit group with the largest motion amplitude that is located within the user's field of view (FOV) and the second coding unit group with the largest motion amplitude that is located outside the user's FOV. Based on the coding unit sizes of the first coding unit group and the second coding unit group, determine whether the user FOV and the GOP parameters need to be adjusted; If it is necessary to adjust the user's FOV and the GOP parameter, then obtain the user's head pose data and adjust the user's FOV and the GOP parameter according to the head pose data; The step of determining, based on the change in motion vectors of the coding units corresponding to their positions in the i-th and i+1-th video frames, a first group of coding units with the largest motion amplitude located within the user's field of view (FOV) and a second group of coding units with the largest motion amplitude located outside the user's FOV, includes: Based on the change in motion vector of the coding unit corresponding to the position in the i-th video frame and the i+1 video frames, determine the first coding unit with the largest motion amplitude that is within the user's FOV range and the second coding unit with the largest motion amplitude that is outside the user's FOV range among the N coding units. Based on the motion vectors of the first coding unit and the second coding unit, determine the first coding unit group with the largest motion amplitude within the user's FOV range and the second coding unit group with the largest motion amplitude outside the user's FOV range; Based on the motion vector changes of the coding units corresponding to their positions in the i-th video frame and the i+1-th video frame, determine the first coding unit with the largest motion amplitude within the user's field of view (FOV) and the second coding unit with the largest motion amplitude outside the user's FOV, including: Calculate the mean of the motion vector magnitudes of N coding units in the i-th video frame and the mean of the motion vector magnitudes of N coding units in the (i+1)-th video frame; Calculate the ratio of the magnitude of the N motion vectors in the (i+1)th video frame to the average magnitude of the motion vectors in the ith video frame, and obtain the N magnitude ratios and combine them into a ratio array; The ratio array is divided into an array within the user's FOV range and an array outside the user's FOV range; The largest ratio is obtained from the array within the user's FOV range and the array outside the user's FOV range, and the encoding unit corresponding to the largest ratio between the user's FOV range and the array outside the user's FOV range is determined as the first encoding unit and the second encoding unit.

2. The method according to claim 1, characterized in that, The step of determining the motion vector of any coding unit in the i-th video frame based on inter-frame prediction includes: Predict the first coding unit of the (i-1)th video frame from the starting unit corresponding to the i-th video frame; Starting from the starting unit, the search unit is determined in the i-th video frame according to the search step size set by the diamond search template, and the search unit with the minimum rate distortion cost is determined as the target search unit. If the search distance between the target search unit and the starting unit is one search step, then the two unsearched units with the closest search distance to the target search unit are determined; the unit with the lowest rate-distortion cost among the target search unit and the two unsearched units is determined as the candidate search unit. If the search distance between the target search unit and the starting unit exceeds a preset number of search steps, then a global search is performed on the i-th video frame with the target search unit as the center, and the unit with the lowest rate-distortion cost is determined as the candidate search unit. Starting from the candidate search unit, repeat the above search steps until the candidate search units determined in the previous two attempts are consistent, and determine the candidate search unit that is consistent in the previous two attempts as the prediction unit of the first encoding unit. Based on the first encoding unit and the prediction unit, the motion vector corresponding to the prediction unit in the i-th video frame is determined.

3. The method according to claim 1, characterized in that, The step of determining the first group of coding units with the largest motion amplitude within the user's field of view (FOV) and the second group of coding units with the largest motion amplitude outside the user's FOV, based on the motion vectors of the first coding unit and the second coding unit, includes: Determine at least one coding unit that has the same motion vector as the first coding unit, and form a first coding unit group with the first coding unit that has the largest motion amplitude within the user's field of view; At least one coding unit with the same motion vector as the second coding unit is identified, and together with the second coding unit, they form a second coding unit group with the largest motion amplitude outside the user's FOV range.

4. The method according to claim 3, characterized in that, The step of determining whether to adjust the user FOV and GOP parameters based on the coding unit sizes of the first coding unit group and the second coding unit group includes: The coding unit size of the first coding unit group and the second coding unit group is determined based on the number of coding units contained in the first coding unit group and the second coding unit group; Compare the coding unit sizes contained in the first coding unit group and the second coding unit group; If the coding unit size of the first coding unit group is smaller than the coding unit size of the second coding unit group, then it is determined that the user needs to adjust the user FOV and the GOP parameter. Otherwise, it is determined that the user does not need to adjust the user FOV and the GOP parameter.

5. The method according to claim 1, characterized in that, If it is necessary to adjust the user's FOV and the GOP parameters, then the user's head pose data is obtained, and the user's FOV and the GOP parameters are adjusted according to the head pose data, including: If it is necessary to adjust the user's FOV and the GOP parameters, then obtain the user's head attitude data, which includes roll angle, pitch angle and yaw angle. The user's predicted viewpoint is determined based on the roll angle, the pitch angle, and the yaw angle; Determine whether the first encoding unit falls within the visible range of the predicted viewpoint. If so, there is no need to adjust the user FOV and the GOP parameters. Otherwise, the user's FOV is adjusted based on the predicted viewpoint, and the GOP parameters are also adjusted.

6. A panoramic video encoding device, characterized in that, include: The first determining module is used to determine the first video frame group of the panoramic video stream according to the image group GOP parameter. The first video frame group contains M video frames, and each video frame corresponds to N coding units. M and N are both integers greater than 2. The second determining module is used to determine the motion vector of each coding unit in the i-th video frame and the (i+1)-th video frame among the M video frames based on inter-frame prediction, wherein i is greater than 2 and less than M; The third determining module is used to determine, based on the change in motion vector of the coding unit corresponding to the position in the i-th and i+1 video frames, the first coding unit group that is within the user's field of view (FOV) and has the largest motion amplitude, and the second coding unit group that is outside the user's FOV and has the largest motion amplitude. The fourth determining module is used to determine whether the user FOV and the GOP parameters need to be adjusted based on the coding unit sizes of the first coding unit group and the second coding unit group. The adjustment module is used to acquire user head pose data and adjust user FOV and GOP parameters according to the head pose data if user FOV and GOP parameters need to be adjusted. The third determining module is specifically used to determine, based on the change in motion vectors of the corresponding coding units in the i-th video frame and the i+1 video frames, a first coding unit with the largest motion amplitude within the user's FOV and a second coding unit with the largest motion amplitude outside the user's FOV; and based on the motion vectors of the first coding unit and the second coding unit, to determine a group of first coding units with the largest motion amplitude within the user's FOV and a group of second coding units with the largest motion amplitude outside the user's FOV. The third determining module is further specifically used to calculate the average value of the motion vector magnitude of N coding units in the i-th video frame and the motion vector magnitude of N coding units in the (i+1)-th video frame; calculate the ratio of the N motion vector magnitudes in the (i+1)-th video frame to the average value of the motion vector magnitude of the i-th video frame, obtain N magnitude ratios and combine them into a ratio array; divide the ratio array into an array within the user's FOV and an array outside the user's FOV; obtain the largest ratio from the array within the user's FOV and the array outside the user's FOV respectively, and determine the coding unit corresponding to the largest ratio within and outside the user's FOV as the first coding unit and the second coding unit.

7. An electronic device, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 5 by calling the program instructions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1 to 5.

Images