Method and device for improving panoramic video user experience

Abstract

A method for improving the user experience of panoramic videos is applied to a server, and the server is used to encode the video data of the panoramic videos. The method includes: obtaining the frame rate, the duration of freeze when the viewing angle is switched, and the field of view coordinates fed back by the user end, and determining the field of view switching speed based on the field of view coordinates; determining the encoding strategy based on the frame rate, the duration of freeze and the field of view switching speed, and the encoding strategy includes key frame interval and quantization parameters; encoding the video data based on the encoding strategy to determine the panoramic frame; cropping the panoramic frame based on the field of view coordinates to determine the field of view frame under the user's current viewing angle; and sending the field of view frame to the user end. This method can improve the user viewing experience when using panoramic videos.

Description

Technical Field

[0001] The present invention relates to the field of panoramic video technology, and in particular to a method and device for improving panoramic video user experience. Background Art

[0002] Panoramic video is a new type of multimedia application that allows users to stand at the camera's position and look around at pictures from all angles. With virtual reality head-mounted devices, users can get an immersive audio-visual experience that is different from flat videos. Considering that users can only see pictures within a certain angle at any point in time, downloading the complete panoramic picture from the client will cause unnecessary waste of network bandwidth and additional decoding overhead. Therefore, a perspective-based transmission solution can be used to only transmit the picture at the angle the user is viewing. For an example, please refer to Figure 1 , Figure 1 is a schematic diagram of a transmission scheme based on viewing angle. Figure 1 As shown in the figure, the client transmits its own viewing angle information to the server. The server obtains the video frame corresponding to the user's viewing angle according to the user's viewing angle, and returns the obtained video frame to the client. This method can avoid unnecessary overhead. However, this method faces the challenge of quickly responding to changes in user viewing angles. After the user switches the viewing angle, if the content within the new viewing angle range is not loaded in time, it will cause the user to feel uncomfortable or even dizzy. This situation seriously affects the user's viewing experience. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing device, computer storage medium and product including a computer program for improving the user experience of panoramic videos, which can enhance the user experience when watching panoramic videos.

[0004] In the first aspect, an embodiment of the present application provides a method for improving the user experience of panoramic videos, which is applied to a server, and the server is used to encode video data of the panoramic video. The method includes: obtaining the frame rate, the duration of the freeze when the viewing angle is switched, and the field of view coordinates fed back by the user end, and determining the field of view switching speed based on the field of view coordinates; determining the encoding strategy based on the frame rate, the duration of the freeze and the field of view switching speed, and the encoding strategy includes a key frame interval and a quantization parameter; encoding the video data based on the encoding strategy to determine the panoramic frame; cropping the panoramic frame based on the field of view coordinates to determine the field of view frame under the user's current viewing angle; and sending the field of view frame to the user end.

[0005] In some possible implementations, the server is also used to obtain bandwidth estimation, system resource utilization, and the bit rate of panoramic video played by the user, and determine the encoding strategy specifically: determining the encoding strategy based on bandwidth estimation, system resource utilization, frame rate, freeze duration and field of view switching speed.

[0006] In some possible implementations, the server is also used to obtain the video quality and bit rate of the panoramic video when the user plays the panoramic video. The method also includes: adjusting the encoding strategy based on the video quality, bit rate, frame rate, and freeze duration.

[0007] In some possible implementations, the encoding strategy is determined according to the following formula:

[0008]

[0009] 0≤k≤k max

[0010] 0≤q≤q max

[0011] stU(k,q)≤U threshold

[0012] EFR(k,q)≥25

[0013] R(k,q)≤E

[0014] Among them, k represents the key frame interval, q represents the quantization parameter, w represents the weight, F represents the frame rate, Q represents the image quality, T represents the freeze duration, U represents the viewpoint information update time interval, and U threshold represents the threshold of the viewpoint information update time interval, EFR represents the encoder output frame rate, R represents the encoder output bit rate, E represents the available bandwidth estimate, and st represents the constraint condition.

[0015] In some possible implementations, the weights are adjusted according to preferences set by the user.

[0016] In the second aspect, an embodiment of the present application provides a device for improving the user experience of panoramic videos, which is deployed on a server side. The server side is used to encode video data of panoramic videos. The device includes: an acquisition module, which is used to acquire the frame rate, the duration of freeze when the viewing angle is switched, and the field of view coordinates fed back by the user side, and determine the field of view switching speed based on the field of view coordinates; a processing module, which is used to determine the encoding strategy based on the frame rate, the duration of freeze and the field of view switching speed, and the encoding strategy includes a key frame interval and a quantization parameter; the processing module is also used to encode the video data based on the encoding strategy to determine the panoramic frame; the processing module is also used to crop the panoramic frame based on the field of view coordinates to determine the field of view frame under the user's current viewing angle; a communication module is used to send the field of view frame to the user side.

[0017] In some possible implementations, the acquisition module is also used to obtain bandwidth estimation, system resource utilization, and the bit rate of panoramic video played by the user end; the processing module is also used to determine the encoding strategy based on the bandwidth estimation, system resource utilization, frame rate, freeze duration and field of view switching speed.

[0018] In some possible implementations, the acquisition module is also used to obtain the video quality and bit rate of the panoramic video played by the user end; the processing module is also used to adjust the encoding strategy based on the video quality, bit rate, frame rate, and freeze duration.

[0019] In some possible implementations, the encoding strategy is determined according to the following formula:

[0020]

[0021] 0≤k≤k max

[0022] 0≤q≤q max

[0023] stU(k,q)≤U threshold

[0024] EFR(k,q)≥25

[0025] R(k,q)≤E

[0026] Among them, k represents the key frame interval, q represents the quantization parameter, w represents the weight, F represents the frame rate, Q represents the image quality, T represents the freeze duration, U represents the viewpoint information update time interval, and U threshold represents the threshold of the viewpoint information update time interval, EFR represents the encoder output frame rate, R represents the encoder output bit rate, E represents the available bandwidth estimate, and st represents the constraint condition.

[0027] In some possible implementations, the weights are adjusted according to preferences set by the user.

[0028] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method described in any one of the first aspects.

[0029] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method described in any one of the first aspects is executed.

[0030] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program. When the computer program product runs on a processor, the processor executes the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0032] Figure 1 is a schematic diagram of a perspective-based transmission solution provided in an embodiment of the present application;

[0033] Figure 2 This is a schematic diagram of a panoramic video user experience improvement system provided by an embodiment of the present application;

[0034] Figure 3 It is a flowchart of a method for improving user experience of panoramic video provided by an embodiment of the present application;

[0035] Figure 4 It is a structural schematic diagram of a device for improving user experience of panoramic video provided in an embodiment of the present application;

[0036] Figure 5 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0039] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0040] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0041] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0042] To facilitate the understanding of the embodiments of the present application, the following will further explain and illustrate specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0043] First, the technical terms involved in the embodiments of the present application are introduced:

[0044] 1. Video bit rate, or bit rate, refers to the number of data bits transmitted per unit time during video data transmission, and can also be called the video bit rate.

[0045] Next, the technical solution provided by the embodiments of the present application is introduced.

[0046] In some known methods, the perspective-based video transmission scheme can be divided into two methods: pre-segmentation and real-time capture. For the pre-segmentation method, in the pre-segmentation perspective transmission scheme, the original video frame is spatially divided into several tiles in the preprocessing stage, and is packaged into several video chunks in time, and stored in the server in this way. The user downloads the video chunk containing the corresponding tile on demand based on the current viewpoint information. Since video chunk packaging will introduce additional cache delay, the pre-segmentation perspective transmission scheme is only suitable for low real-time scenarios such as panoramic video on demand.

[0047] As for the real-time interception method, in the real-time interception perspective transmission scheme, the server extracts the pixels of the field of view area from the original video frame in real time according to the perspective information fed back by the user, and sends them frame by frame immediately. The real-time interception method can effectively reduce the processing delay, but it consumes a lot of computing resources. When encoding the video, the original video frame will be encoded as a key frame or a non-key frame. It is worth noting that in the real-time interception scheme, the server only accepts the user's viewpoint information update when the encoder outputs a key frame. Because the video key frame carries the spatial order relationship between different tiles, if the server also accepts the viewpoint information update in the non-key frame, the client may make errors during decoding or the tile order may be disordered during display, which seriously affects the user experience. Therefore, the key frame interval directly determines the update frequency of the viewpoint information.

[0048] In existing real-time interception-type perspective transmission schemes, the encoder outputs a key frame interval that is usually a fixed value (e.g., 1 key frame is output after every 30 non-key frames). When the user quickly switches viewpoints, if the key frame interval is large, the server cannot perceive the viewpoint information changes in time, and still sends the content around the old viewpoint to the client, resulting in the loss of high-definition content within the user's field of view, and the appearance of low-definition background or even black pixels. If the key frame interval is small, although the server can perceive the viewpoint information changes in time, due to the high complexity of key frame encoding and the limitation of the server's computing power, the encoder output frame rate will be reduced, sacrificing the smoothness of the picture. In addition, since the key frames carry richer picture information, more network transmission bandwidth will be consumed after the key frame interval is reduced, and the average time consumed by the client in receiving, assembling and decoding video frames will also increase significantly, thereby affecting the smoothness of the picture.

[0049] In view of this, in order to overcome the adverse effect of reducing the key frame interval on the smoothness of the picture, the embodiment of the present application provides a method for improving the user experience of panoramic video. The server jointly regulates the encoder key frame interval and picture quality based on the user's most recent viewpoint movement speed, server computing resource usage and available network bandwidth. Synchronously adjusting the picture quality during the reduction of the key frame interval can ensure smoothness and improve user experience.

[0050] For example, Figure 2 A schematic diagram of a panoramic video user experience improvement system provided by an embodiment of the present application is shown. Figure 2 As shown, the panoramic video user experience improvement system includes a server 210 and a user 220 .

[0051] Among them, the server 210 includes a controller 2101, a decoder 2102, an encoder 2103, a cropping module 2104, and a transmission module 2105. Among them, the controller 2101 is used to determine the encoder parameters based on the acquired parameter information. Specifically, the controller 2101 can obtain the frame rate of the user end 220 received by the transmission module 2105, and the motion freeze duration of the panoramic video, and obtain the FOV switching speed sent by the cropping module 2104. Based on the acquired frame rate, freeze duration and FOV switching speed, the encoding strategy is determined by calculation, and the encoding strategy includes a key frame interval and a quantization parameter (quantization parameter, QP). Then the calculated encoding strategy is sent to the encoder so that the encoder encodes the video according to the key frame interval and QP. Among them, the quantization parameter is used to control the quality of the panoramic frame output by the encoder.

[0052] In some possible embodiments, the controller 2101 may also be used to receive bandwidth information estimated by the transmission module 2105. The controller 2101 may also be used to read the log information of the encoder 2103, calculate the video bit rate, video quality (natural image quality evaluator, NIQE) and encoding frame rate information. The controller 2101 may also be used to monitor system resources and calculate the system resource utilization. When calculating QP and key frame interval, the acquired bandwidth information, video bit rate, video quality, encoding frame rate information and system resource utilization information may also be considered, and the encoding strategy may be updated based on this information.

[0053] Decoder 2102 is used to decode panoramic video files or real-time video streams. Panoramic video files are video files stored locally on server 210, and real-time video streams are video streams obtained by server 210 from a third party. By decoding decoder 2102, the original data of the video file or real-time video stream can be obtained. The original video data is sent to encoder 2103.

[0054] The encoder 2103 is used to receive the QP and key frame interval sent by the controller 2101, encode the original video data based on the QP and the key frame interval, and obtain the panoramic frame of the original video data. The obtained panoramic frame is sent to the cropping module 2104. The encoder 2103 is also used to send the bit rate, video quality, and frame rate of the video encoding of the panoramic frame to the controller 2101, so that the controller 2101 updates the encoding strategy according to the bit rate, frame rate, and NIQE.

[0055] The cropping module 2104 is used to receive the FOV coordinate information sent by the transmission module, calculate the switching speed of the user's viewpoint according to the FOV coordinate information, and send the switching speed to the controller 2101. The cropping module 2104 is also used to receive the panoramic frame sent by the encoder 2103, crop the panoramic frame, and obtain the FOV frame of the user at the current viewing angle.

[0056] The transmission module 2105 is used to interact with the transmission module 2201 of the user terminal 220 through the network, send the cropped FOV frame to the client 220, and receive the FOV coordinates, freeze duration and frame rate sent by the user terminal 220. The transmission module 2105 is also used to send the received FOV coordinates to the cropping module 2104, and receive the cropped FOV video frame sent by the cropping module 2104.

[0057] In some possible embodiments, the transmission module 2105 is further configured to estimate the available bandwidth according to information such as packet loss and RTT, and send the bandwidth estimation result to the controller 2101 through a callback function.

[0058] The user end 220 includes a transmission module 2201, a decoder 2202, a playback module 2203, and a sampling module 2204. The transmission module 2201 is used to interact with the transmission module 2105 of the server end 210 to obtain the cropped FOV frame sent by the server end 210. After receiving the FOV video frame, the transmission module 2201 sends the received FOV frame to the decoder 2202. The transmission module 2201 is also used to send the FOV coordinates of the user end 220, the frame rate of the video played by the user end 220, and the duration of the panoramic video motion freeze to the transmission module 2105 of the server end 210. The duration of the panoramic video motion freeze refers to the time from when the user changes the viewing angle to when the panoramic video is completely loaded and successfully stopped at the converted viewing angle.

[0059] The decoder 2202 is used to receive the FOV frame sent by the transmission module 2201, decode the received FOV frame, and send the decoded FOV frame to the playback module 2203.

[0060] The playback module 2203 is used to play the decoded FOV frame, determine the frame rate and the motion freeze duration when switching viewpoints, and send the frame rate and freeze duration to the transmission module 2201.

[0061] The sampling module 2204 is used to obtain the current FOV coordinates of the user terminal and send the obtained FOV coordinates to the transmission module 2201.

[0062] In an embodiment of the present application, the server 210 can adaptively adjust the key frame interval according to the user's viewpoint movement speed, the network bandwidth estimation of the transmission layer and the computing resource occupancy of the server, so as to improve the server 210's sensing speed of changes in the user's viewpoint coordinates. Taking into account the limitations of the computing resources in the server 210, the unsmooth playback caused by the increase in the number of key frames is avoided by jointly regulating the key frame interval and the quantization parameters. And based on the most recent decision results and actual experience, the decision logic can be updated to adapt to the impact of changes in network bandwidth and video content complexity on encoding. Based on the panoramic video user experience improvement system, the embodiment of the present application also provides a panoramic video user experience improvement method. Exemplarily, Figure 3 is a flow chart of a method for improving user experience of panoramic video provided by an embodiment of the present application. The method can be applied to a server, and the server can be as follows: Figure 2 The server 210 shown. Figure 3 As shown, the method for improving panoramic video user experience may include the following steps:

[0063] S31: Obtain the frame rate, freeze duration when the viewing angle is switched, and FOV coordinates fed back by the user end, and determine the FOV switching speed based on the FOV coordinates.

[0064] In this embodiment, the server can Figure 2 The transmission module 2105 shown receives user data transmitted by the user end, including the frame rate of the panoramic video when the user watches the panoramic video, the motion freeze duration when the user changes the viewing angle, and the FOV coordinates.

[0065] In some possible embodiments, the server may also obtain available bandwidth estimation value, video bit rate, video quality, encoding frame rate, and system resource utilization.

[0066] In this embodiment, the server can calculate the encoded bit rate, NIQE, and encoding frame rate information based on the encoder log information. The server can also monitor its own system resources and calculate the system resource utilization. Based on the congestion control mechanism, the available bandwidth is estimated based on information such as packet loss and round-trip time (RTT) in the network.

[0067] S32: Determine a coding strategy based on the frame rate, motion freeze duration, and FOV switching speed. The coding strategy includes key frame interval and QP.

[0068] In this embodiment, after obtaining the frame rate, motion freeze duration and FOV switching speed, the key frame interval and QP required when encoding the video data can be determined based on the obtained frame rate, motion freeze duration and FOV switching speed.

[0069] In some possible embodiments, the key frame interval and QP required for encoding may be determined based on the acquired frame rate, motion freeze duration and FOV switching speed, system resource utilization, and bandwidth estimation value.

[0070] In some possible embodiments, the encoding strategy (ie, key frame interval and QP) may be adjusted according to the bit rate, NIQE, encoding frame rate, and motion freeze duration of the panoramic video.

[0071] In some possible embodiments, the coding strategy is determined according to the following formula:

[0072]

[0073] 0≤k≤k max

[0074] 0≤q≤q max

[0075] stU(k,q)≤U threshold

[0076] EFR(k,q)≥25

[0077] R(k,q)≤E

[0078] Among them, k represents the key frame interval, q represents the quantization parameter, w represents the weight, F represents the frame rate, Q represents the image quality, T represents the freeze duration, U represents the viewpoint information update time interval, and U threshold The threshold value representing the viewpoint information update time interval (in milliseconds), EFR represents the encoder output frame rate, R represents the encoder output bit rate, E represents the available bandwidth estimate, and st represents the constraint condition.

[0079] The image quality is measured by the natural image quality evaluation index NIQE, and the view switching freeze duration records the duration of low-definition or blank pixels after the user switches the view. The weight w can be adjusted according to the user's preference. For example, if the user prefers smoothness, the value of the frame rate weight w1 can be increased. In the constraints, k max Characterizes the maximum value of the key frame interval, q max Characterizes the maximum value of the quantization parameter. These two values ​​can usually be determined during the encoder design phase. The threshold of the viewpoint information update time interval can be determined based on the user's viewpoint movement speed. The encoder's output frame rate usually needs to be greater than 25fps to ensure smooth playback of the picture.

[0080] Since the mathematical relationship between key frame interval and quantization parameters and frame rate, image quality, freeze duration when switching viewpoints, viewpoint information update interval, encoder output frame rate, and encoder output bit rate is not clear, and these mathematical relationships are also affected by video complexity, before making encoding decisions, we can train and update a set of decision tree regression models based on the decision results and experience indicators of the latest several historical sessions to fit these mathematical relationships. In the process of searching for the optimal encoder parameters, it is found that the size of the complete solution space of these mathematical relationships is k max ×q max , and k max and q max The value of is usually small. After obtaining the decision tree regression model of the mathematical relationship, the solution space can be further narrowed by using the constraints, and then the maximum value of the objective function can be searched by enumeration. This method is called the key frame interval and quantization parameter joint control algorithm. In this algorithm, the input of the algorithm is the historical key frame interval k τ , quantization parameter q τ , frame rate F τ , picture quality τ 、Stuttering duration T τ , viewpoint information update time interval U τ , encoder output frame rate EFR τ , the encoder's output bit rate R τ , the threshold of the viewpoint information update time interval Available bandwidth estimate E, output is the current key frame interval k t and the quantization parameter q t Specifically, the input parameter k τ ,q τ , F τ , Q τ , T τ , U τ , EFR τ , R τ ,E, Among them, τ∈[t-1-(hw-1),t-1], hw represents the number of historical sessions, and t represents the current time. τ ,q τ ],F τ},τ∈[t-1-(hw-1),t-1] fits the decision tree regression model F(k,q), using {[k τ ,q τ ],Q τ},τ∈[t-1-(hw-1),t-1] fits the decision tree regression model Q(k,q), using

[0081] {[k τ ,qτ ],T τ},τ∈[t-1-(hw-1),t-1] fits the decision tree regression model T(k,q), using

[0082] {[k τ ,q τ ],U τ},τ∈[t-1-(hw-1),t-1] fits the decision tree regression model U(k,q), using

[0083] {[k τ ,q τ ],EFR τ},τ∈[t-1-(hw-1),t-1] fits the decision tree regression model EFR(k,q), using

[0084] {[k τ ,q τ ],R τ},τ∈[t-1-(hw-1),t-1] fits the decision tree regression model R(k,q). After fitting the decision tree regression model, the following code can be used for calculation

[0085]

[0086] Through multiple cycles, the maximum value of the objective function is enumerated, and Max_k and Max_q at this time are used as the current key frame interval k t and the quantization parameter q t .

[0087] S33: Encode the video data based on the encoding strategy to determine the panoramic frame.

[0088] In this embodiment, after the encoding strategy is determined, the encoder at the server can encode the video data according to the encoding strategy to obtain a panoramic frame of the video data. A panoramic frame is a complete 360-degree video frame. In a panoramic frame, each frame contains a horizontal 360-degree and vertical 180-degree (or more) viewing angle, so that the user can freely rotate the viewing angle when watching and experience a full range of scenes.

[0089] S34: Based on the FOV coordinates, the panoramic frame is cropped to determine the FOV frame under the user's current viewing angle.

[0090] In this embodiment, the user end feeds back the user's FOV coordinates to the server end, and the server end can crop the panoramic frame according to the FOV coordinates to obtain the FOV frame under the user's current viewing angle.

[0091] S35: Send the FOV frame to the user end.

[0092] In this embodiment, after the FOV frame is determined, the server may send the FOV frame of the current viewing angle to the client, so that the client displays and plays the FOV frame.

[0093] The above is an introduction to the method for improving the user experience of panoramic videos provided in the embodiment of the present application. By obtaining the FOV coordinates of the user's viewpoint fed back by the client, the frame rate of the panoramic video, and the duration of freeze when switching the viewpoint, a coding strategy is formulated based on the data fed back by the client, the system resource occupancy of the server, and the available bandwidth to adjust the key frame interval and quantization parameters, and encode the video data based on the formulated coding strategy so that the encoded panoramic frame can meet the user's usage, especially the situation where the user's viewpoint switches quickly. This method can significantly reduce the delay when switching the viewpoint, and at the same time, there is almost no loss of image quality and smoothness, which can improve the user's viewing experience.

[0094] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any feature of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0095] Based on the method in the above embodiment, the embodiment of the present application also provides a device for improving the user experience of panoramic video. Figure 4 The present invention provides a schematic diagram of a panoramic video user experience improvement device, which is deployed on a server and is used to encode the video data of the panoramic video. The panoramic video user experience improvement device 400 includes: an acquisition module 401, a processing module 402, and a communication module 403.

[0096] The acquisition module 401 is used to acquire the frame rate, the freeze duration when the viewing angle is switched, and the field of view coordinates fed back by the user end, and determine the field of view switching speed based on the field of view coordinates.

[0097] The processing module 402 is used to determine the encoding strategy based on the frame rate, the freeze duration and the field of view switching speed. The encoding strategy includes a key frame interval and a quantization parameter.

[0098] The processing module 402 is also used to encode the video data based on the encoding strategy to determine the panoramic frame.

[0099] The processing module 402 is also used to crop the panoramic frame based on the field of view coordinates to determine the field of view frame at the user's current viewing angle.

[0100] The communication module 403 is used to send the field of view frame to the user end.

[0101] In some possible embodiments, the acquisition module 401 is also used to obtain bandwidth estimation, system resource utilization, and the bit rate of panoramic video played by the user end; the processing module 402 is also used to determine the encoding strategy based on the bandwidth estimation, system resource utilization, frame rate, freeze duration and field of view switching speed.

[0102] In some possible embodiments, the acquisition module 401 is also used to obtain the video quality and bit rate of the panoramic video of the server; the processing module 402 is also used to adjust the encoding strategy based on the video quality, bit rate, frame rate, and freeze duration.

[0103] In some possible embodiments, the encoding strategy is determined according to the following formula:

[0104]

[0105] 0≤k≤k max

[0106] 0≤q≤q max

[0107] stU(k,q)≤U threshold

[0108] EFR(k,q)≥25

[0109] R(k,q)≤E

[0110] Among them, k represents the key frame interval, q represents the quantization parameter, w represents the weight, F represents the frame rate, Q represents the image quality, T represents the freeze duration, U represents the viewpoint information update time interval, and U threshold represents the threshold of the viewpoint information update time interval, EFR represents the encoder output frame rate, R represents the encoder output bit rate, E represents the available bandwidth estimate, and st represents the constraint condition.

[0111] In some possible embodiments, the weights are adjusted according to preferences set by the user.

[0112] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0113] The present application also provides a computing device 500. Figure 5 As shown, computing device 500 includes: bus 502, processor 504, memory 506 and communication interface 508. Processor 504, memory 506 and communication interface 508 communicate through bus 502. Computing device 500 refers to a server. It should be understood that the present application does not limit the number of processors and memories in computing device 500.

[0114] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus 504 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 504 may include a path for transmitting information between various components of the computing device 500 (eg, the memory 506, the processor 504, and the communication interface 508).

[0115] The processor 504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0116] The memory 506 may include a volatile memory, such as a random access memory (RAM). The processor 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0117] The memory 506 stores executable program codes, and the processor 504 executes the executable program codes to respectively implement the functions of the aforementioned receiving module 401 and the processing module 402, thereby implementing all or part of the steps of the method in the above embodiment. That is, the memory 506 stores instructions for executing all or part of the steps in the method in the above embodiment.

[0118] Alternatively, the memory 506 stores executable codes, and the processor 504 executes the executable codes to respectively implement the functions of the aforementioned panoramic video user experience improvement device 400, thereby implementing all or part of the steps in the above-mentioned embodiment method. That is, the memory 506 stores instructions for executing all or part of the steps in the above-mentioned embodiment method.

[0119] The communication interface 508 is used to implement the functions of the aforementioned communication module 403, and may use, but is not limited to, a transceiver module such as a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0120] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0121] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0122] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0123] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0124] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0125] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. A method for improving user experience of panoramic video, characterized in that: Applied to a server, the server is used to encode video data of a panoramic video, and the method includes: Obtaining the frame rate, the duration of freeze when the viewing angle is switched, and the field of view coordinates fed back by the user end, and determining the field of view switching speed based on the field of view coordinates; Based on the frame rate, the freeze duration and the field of view switching speed, a coding strategy is determined, the coding strategy including a key frame interval and a quantization parameter; wherein the coding strategy is determined according to the following formula: 0≤k≤k max 0≤q≤q max s.t.U(k,q)≤U threshold EFR(k,q)≥25 R(k,q)≤E Among them, k represents the key frame interval, q represents the quantization parameter, w represents the weight, F represents the frame rate, Q represents the image quality, T represents the freeze duration, st represents the constraint condition, and k max Characterizes the maximum value of the key frame interval, q max represents the maximum value of the quantization parameter, U represents the viewpoint information update time interval, and U threshold The threshold representing the viewpoint information update time interval, EFR representing the encoder output frame rate, R representing the encoder output bit rate, and E representing the available bandwidth estimation value; Based on the encoding strategy, the video data is encoded to determine a panoramic frame; Based on the field of view coordinates, the panoramic frame is cropped to determine the field of view frame at the user's current viewing angle; The field of view frame is sent to the user terminal.

2. The method according to claim 1, characterized in that The server is also used to obtain a bandwidth estimation value, a system resource utilization rate, and a bit rate of the panoramic video played by the user. The encoding strategy is specifically determined as follows: The encoding strategy is determined based on the bandwidth estimation value, the system resource utilization, the frame rate, the freeze duration and the field of view switching speed.

3. The method according to claim 2, characterized in that The server is also used to obtain the video quality and bit rate of the panoramic video played by the user, and the method further includes: The encoding strategy is adjusted based on the video quality, the bit rate, the frame rate, and the freeze duration.

4. The method according to claim 1, characterized in that The weights are adjusted according to preferences set by the user.

5. A device for improving user experience of panoramic video, characterized in that: Deployed on a server, the server is used to encode video data of a panoramic video, and the device includes: An acquisition module, used to acquire the frame rate, the duration of freeze when the viewing angle is switched, and the field of view coordinates fed back by the user end, and determine the field of view switching speed based on the field of view coordinates; A processing module is used to determine a coding strategy based on the frame rate, the freeze duration and the field of view switching speed, wherein the coding strategy includes a key frame interval and a quantization parameter; the coding strategy is determined according to the following formula: 0≤k≤k max 0≤q≤q max s.t.U(k,q)≤U threshold EFR(k,q)≥25 R(k,q)≤E Among them, k represents the key frame interval, q represents the quantization parameter, w represents the weight, F represents the frame rate, Q represents the image quality, T represents the freeze duration, st represents the constraint condition, and k max Characterizes the maximum value of the key frame interval, q max represents the maximum value of the quantization parameter, U represents the viewpoint information update time interval, and U threshold The threshold representing the viewpoint information update time interval, EFR representing the encoder output frame rate, R representing the encoder output bit rate, and E representing the available bandwidth estimation value; The processing module is further used to encode the video data based on the encoding strategy to determine a panoramic frame; The processing module is further used to crop the panoramic frame based on the field of view coordinates to determine the field of view frame under the user's current viewing angle; A communication module is used to send the field of view frame to the user terminal.

6. The device according to claim 5, characterized in that The acquisition module is also used to obtain a bandwidth estimation value, a system resource utilization rate, and a bit rate of the panoramic video played by the user end; The processing module is further used to determine the encoding strategy based on the bandwidth estimation value, the system resource utilization, the frame rate, the freeze duration and the field of view switching speed.

7. The device according to claim 6, characterized in that The acquisition module is also used to obtain the video quality and bit rate of the panoramic video played by the user end; The processing module is further used to adjust the encoding strategy based on the video quality, the bit rate, the frame rate, and the freeze duration.

8. The device according to claim 5, characterized in that The weights are adjusted according to preferences set by the user.

Images