A method, apparatus and device for transmitting a block panorama video

Abstract

The application discloses a kind of block panoramic video transmission method, device and equipment, the method comprises: first when the user viewport prediction result obtained is inconsistent with real user viewport, using the way of buffer replacement, according to the intersection of user viewport prediction result and real user viewport, determine the resource that can be reused in current buffer, and determine the set of video block that needs to be requested, then according to the time length of video block in buffer and the network bandwidth prediction result obtained, the code rate of video block to be requested is calculated, and the video block and its code rate currently required are calculated according to the preset evaluation index of QoE, so that the QoE of panoramic video is maximum, and then according to the set of video block that needs to be requested, the code rate of video block to be requested, the video block and its code rate currently required are calculated, to realize the transmission of block panoramic video.So as to improve the transmission effect of block panoramic video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a block panoramic video transmission method, device and equipment. BACKGROUND

[0002] In recent years, with the development of bank digitization, in order to provide personalized and immersive financial services for customers, virtual business halls emerge as the times require. The virtual business hall mainly uses panoramic video technology to simulate a real business hall environment, so that users can perform online bank services in a virtual environment. At present, the block-based panoramic video adaptive transmission method has become mainstream due to its high transmission efficiency.

[0003] At present, in the adaptive transmission process of the block panoramic video, the system needs to predict the user viewport and network status in real time, so as to avoid video black blocks or interruptions when the prediction fails. Specifically, for the case of insufficient bandwidth or network fluctuation, the existing technology reduces video interruption by increasing the buffer length. However, this method not only reduces the video quality, but also wastes part of the bandwidth resources. For the case of user viewport prediction failure, the existing technology reduces video black blocks or interruptions by using a double-layer cache or a method of emptying and re-requesting. However, this method also wastes part of the bandwidth resources and reduces the quality of the viewport area. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a block panoramic video transmission method, device and equipment, which can determine the total code rate of the requested video blocks and the set of video blocks that need to be re-requested by using a code rate adjustment function and a buffer replacement method, thereby guaranteeing the quality of experience (QoE) of the panoramic video when the prediction fails, and improving the transmission effect of the block panoramic video.

[0005] In a first aspect, the embodiments of the present application provide a block panoramic video transmission method, comprising:

[0006] obtaining a network bandwidth prediction result and a user viewport prediction result;

[0007] when the user viewport prediction result is inconsistent with the real user viewport, using a buffer replacement method, determining the reusable resources in the current buffer according to the intersection of the user viewport prediction result and the real user viewport, and determining the set of video blocks that need to be re-requested;

[0008] calculating the code rate of the video blocks to be requested according to the network bandwidth prediction result and the time length of the video blocks in the buffer;

[0009] According to a preset evaluation index of quality of experience (QoE), a video block currently required to be requested and a code rate thereof are calculated, so that the QoE of the panoramic video is maximized.

[0010] According to the set of video blocks required to be re-requested, the code rate of the video block to be requested, the video block currently required to be requested and the code rate thereof, a final video block required to be requested and a corresponding code rate thereof are calculated, so that the chunked panoramic video transmission is implemented.

[0011] Optionally, the calculation of the code rate of the video block to be requested according to the network bandwidth prediction result and the time length of the video blocks in the buffer includes:

[0012] A predefined code rate adjustment function is used to dynamically adjust the code rate of the video block to be requested according to the time length of the video blocks currently stored in the buffer.

[0013] Optionally, the preset evaluation index of QoE includes a user viewport quality evaluation index, a user viewport spatial quality smoothness evaluation index, a user viewport time quality smoothness evaluation index and a video interruption time evaluation index.

[0014] Optionally, the calculation of the final video block required to be requested and the corresponding code rate thereof according to the set of video blocks required to be re-requested, the code rate of the video block to be requested, the video block currently required to be requested and the code rate thereof includes:

[0015] According to the set of video blocks required to be re-requested, the code rate of the video block to be requested, the video block currently required to be requested and the code rate thereof, a final video block required to be requested and a corresponding code rate thereof are calculated, so that the chunked panoramic video transmission is implemented.

[0016] In a second aspect, an embodiment of the present application further provides a chunked panoramic video transmission device, including:

[0017] An acquisition unit is configured to acquire a network bandwidth prediction result and a user viewport prediction result.

[0018] A determination unit is configured to, when the user viewport prediction result is inconsistent with an actual user viewport, determine, by using a buffer replacement manner, a reusable resource in a current buffer according to an intersection of the user viewport prediction result and the actual user viewport, and determine a set of video blocks required to be re-requested.

[0019] A first calculation unit is configured to calculate a code rate of a video block to be requested according to the network bandwidth prediction result and a time length of the video blocks in the buffer.

[0020] The second computing unit is configured to calculate the video block currently required to be requested and the code rate of the video block according to a preset evaluation index of quality of experience (QoE) so as to maximize the QoE of the panoramic video.

[0021] The third computing unit is configured to calculate the video block finally required to be requested and the code rate of the video block according to the set of video blocks required to be re-requested, the code rate of the video block to be requested, the video block currently required to be requested and the code rate of the video block, so as to realize the transmission of the chunked panoramic video.

[0022] Optionally, the first computing unit is specifically configured to:

[0023] The code rate of the video block to be requested is dynamically adjusted according to the time length of the video block currently stored in the buffer by using a predefined code rate adjustment function.

[0024] Optionally, the preset evaluation index of QoE includes a user viewport quality evaluation index, a user viewport spatial quality smoothness evaluation index, a user viewport time quality smoothness evaluation index and a video interruption time evaluation index.

[0025] Optionally, the third computing unit is specifically configured to:

[0026] The video block finally required to be requested and the code rate of the video block are calculated by using a heuristic search algorithm BeamSearch according to the set of video blocks required to be re-requested, the code rate of the video block to be requested, the video block currently required to be requested and the code rate of the video block, so as to realize the transmission of the chunked panoramic video.

[0027] Embodiments of the present application further provide a chunked panoramic video transmission device, which comprises a processor, a memory and a system bus.

[0028] The processor and the memory are connected through the system bus.

[0029] The memory is configured to store one or more programs, and the one or more programs comprise instructions which, when executed by the processor, cause the processor to execute any one of the implementation manners of the chunked panoramic video transmission method.

[0030] Embodiments of the present application further provide a computer readable storage medium, which stores instructions, and when the instructions run on a terminal device, cause the terminal device to execute any one of the implementation manners of the chunked panoramic video transmission method.

[0031] The embodiment of the present application provides a kind of block panoramic video transmission method, device and equipment, first acquire network bandwidth prediction result and user viewport prediction result, then, when user viewport prediction result is inconsistent with real user viewport, using the way of buffer replacement, according to the intersection of user viewport prediction result and real user viewport, determine the resource that can be reused in current buffer, and determine the set of video block that needs to be requested, then, according to network bandwidth prediction result and the time length of video block in buffer, the code rate of video block to be requested is calculated, and then according to the preset evaluation index of quality of experience QoE, the video block and its code rate currently required are calculated, to make the QoE of panoramic video maximum, and then the set of video block that needs to be requested, the code rate of video block to be requested, the video block and its code rate currently required are calculated to realize the transmission of block panoramic video. So as to improve the transmission effect of block panoramic video. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0033] Figure 1 The process schematic diagram of block panoramic video transmission provided by the embodiment of the present application is shown in the figure.

[0034] Figure 2 The flowchart of a block panoramic video transmission method provided by the embodiment of the present application is shown in the figure.

[0035] Figure 3 The comparative example diagram of buffer content processing method when user viewport prediction fails provided by the embodiment of the present application is shown in the figure.

[0036] Figure 4 The schematic diagram of code rate adjustment function provided by the embodiment of the present application is shown in the figure.

[0037] Figure 5 The composition schematic diagram of a block panoramic video transmission device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] At present, the process of block panoramic video transmission is as follows Figure 1As shown, first, the original panoramic video is divided into equal video segments in time, then each video segment is divided into equal video blocks in space, and then each video block is encoded into different code rates and stored in the server. During user viewing, the client requests the corresponding media file from the server according to the current user's viewing angle and network environment, and after processing, the video is played.

[0039] Moreover, in the process of adaptive transmission of panoramic video, the system needs to predict the user viewport and network conditions in real time, so as to avoid video blackouts or interruptions when prediction fails. Specifically, the following two processing methods (which can also be understood as fault tolerance mechanisms) are proposed for prediction failure:

[0040] (1) For the case of insufficient bandwidth or network fluctuation, increase the buffer length to reduce video interruption.

[0041] When watching online videos, to prevent video blocks from not arriving at the client in time under network fluctuations, causing video interruption, existing methods usually use a buffer to cache part of the video resources. The client sends a request to the server, and the media resources arrive at the client first and are stored in the buffer queue, waiting for the previous video segment to be played, and then the next video segment is taken out from the head of the buffer queue for playback. With the buffer, the system can fully utilize the bandwidth resources to request video segments.

[0042] (2) For the case where the user's viewing angle prediction deviation is large, a double-layer buffer or empty re-request method is used.

[0043] In the process of rate adaptation, the code rate allocated to the non-viewport area is relatively low, and in poor network conditions, video blocks in the non-viewport area may not be requested. When the user's viewing angle prediction deviates, there may be inconsistent video quality or video blackouts in the user's field of view, affecting the user's viewing experience. Existing research has proposed some methods to address this issue. Among them, the method using double-layer buffering is the most common, which requests the entire panoramic video to the client as a base layer at a low code rate to ensure that the user's field of view does not have video blackouts, and then requests video blocks in the viewport area according to the user's viewing angle prediction. The viewport area is displayed as an enhancement layer at a high code rate. User viewing angle prediction is a continuous process, and prediction failure at a certain time will affect the user's viewing angle at subsequent times based on the user's viewing angle at that time. Therefore, the video segments cached in the buffer will no longer be suitable for subsequent playback. To address the chain reaction caused by prediction failure, existing research methods use the method of deleting the existing video segments in the buffer and re-requesting subsequent video segments to solve this problem.

[0044] It should be noted that in the adaptive transmission process, the longer the buffer is, the stronger the ability to resist network fluctuations is, and the less likely the video interruption problem is. Therefore, the existing method usually increases the upper limit of the buffer length to improve the fault tolerance of the system. However, this way sacrifices video quality in actual use, increases the buffer length by reducing the video code rate, so that the input and output ratio of the buffer is greater than 1. In addition, when the buffer reaches the upper limit, the client will no longer send a request to the server until the next video segment in the buffer is consumed, and the request will be sent again. In this case, there will be a waiting request time, and the bandwidth resources in this part of the time will be wasted.

[0045] (1) The double-layer cache method effectively solves the problem of video black blocks, but the overlapping part of the base layer and the enhancement layer will be requested twice, and the base layer of the overlapping part will not be displayed as a redundant part. The repeated request will waste part of the bandwidth resources, thereby reducing the quality of the viewport area.

[0046] (2) The empty cache method uses the method of deleting the existing video segments in the buffer and re-requesting subsequent video segments to solve the chain reaction caused by prediction failure. However, this method does not consider the video interruption that may be caused by deleting video segments in the buffer and the resource waste caused by blind deletion.

[0047] It can be seen that the above existing processing method will cause poor transmission effect of the block panoramic video, therefore, how to effectively improve the transmission effect of the block panoramic video is a technical problem to be solved at present.

[0048] To solve the above defects, the embodiment of the present application provides a block panoramic video transmission method, which first acquires a network bandwidth prediction result and a user viewport prediction result, then when the user viewport prediction result is inconsistent with the real user viewport, uses a buffer replacement method, determines the reusable resources in the current buffer according to the intersection of the user viewport prediction result and the real user viewport, and determines a set of video blocks that need to be re-requested, then calculates the code rate of the to-be-requested video blocks according to the network bandwidth prediction result and the time length of the video blocks in the buffer, and calculates the video blocks and their code rates that need to be requested currently according to the preset evaluation index of the quality of experience (QoE), so as to maximize the QoE of the panoramic video, and then the final video blocks and their corresponding code rates that need to be requested can be calculated according to the set of video blocks that need to be re-requested, the code rate of the to-be-requested video blocks, the video blocks and their code rates that need to be requested currently, so as to realize the transmission of the block panoramic video. Therefore, the transmission effect of the block panoramic video can be improved.

[0049] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0050] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0051] A panoramic video is a video that is shot by a professional camera in all directions of 360 degrees. A user can freely switch the viewing angle to watch, thereby obtaining an immersive experience. The video content can be a real scene spliced from different angles recorded by multiple cameras or shot by a professional panoramic camera, or a 3D panoramic video generated by modeling. Compared with a traditional video, a panoramic video has stronger interactivity and a stronger sense of immersion. At present, panoramic videos are widely used in virtual tourism, virtual education, live broadcasting of events and real estate showrooms and other three-dimensional e-commerce fields. The requirement for resolution of a panoramic video is also extremely high. The resolution of a panoramic video needs to reach 4K or above to provide a better viewing experience for a user. The resolution of most panoramic videos usually reaches 6K-8K to provide a more immersive experience for a user, which makes the network bandwidth required for online viewing of a panoramic video 4-6 times that of an ordinary video.

[0052] Quality of Experience (QoE) refers to the subjective feeling of a user on the quality and performance of a device, a network and a system, an application or a service. QoE refers to the degree of difficulty that a user feels in completing the entire process.

[0053] User viewport, the viewport is a video area that can be seen by a user. The viewport of a panoramic video is only a part of the entire video.

[0054] (1) Modeling of a block panoramic video transmission system: in the block panoramic video transmission method proposed in the present application, a model of a block panoramic video adaptive transmission system is first defined to execute the block panoramic video transmission method proposed in the present application (only an example, and the block panoramic video transmission method can also be executed by other computer devices as an execution subject without the model). A server end divides an original panoramic video into J equal time segments, wherein the duration of each time segment is t sThen, each time segment is divided into I equal video blocks in space, and each video block is encoded into K quality versions. These video blocks are stored on the server waiting for the client to request. T(i, j, k) represents a video block of the time segment sequence number j, spatial position sequence number i, and quality version k. Where i ∈ [1, …, I], j ∈ [1, …, J], k ∈ [1, …, K]. j c represents the sequence number of the time segment currently being played. At the client, in order to shorten the playback waiting time, the next time segment that needs to be played needs to be requested in advance. For this purpose, the system uses a user perspective prediction algorithm to predict the user viewport area according to the historical user perspective trajectory. The model uses to represent the video block set in the user viewport area after n*t s time units. At the same time, this method also needs a network prediction mechanism to predict the next network bandwidth, C pre (t) represents the network bandwidth after t time units. The client sends a request to the server and stores the obtained video blocks in the buffer. Assuming that L complete video segments have been stored in the buffer, that is, the video resources stored in the buffer can be played by the player for L*t s time units before the next video segment arrives at the client, the video block set in the user viewport area of the lth time segment stored in the buffer is represented by .

[0055] (2) Define the chunked panoramic video adaptive transmission mechanism: First, the chunked panoramic video transmission method proposed in this application uses C max to represent the maximum rate that can be requested in the next t s time units. As shown in equation (1), L is the length of the time segment stored in the current buffer, and QoE(n) represents the user experience quality of the panoramic video in the nth time segment. This method defines the video rate vector R in the nth time segment as shown in equation (2).

[0056]

[0057] R = <r1, r2, …, r N > (2)

[0058] The goal of the chunked panoramic video transmission method proposed in this application is to find a video rate vector R that maximizes the QoE of the nth time segment. Equations (3) and (4) below define this process.

[0059] arg max R QoE(n) (3)

[0060] ∑r∈R r≤C max (4)

[0061] First embodiment

[0062] Referring to Figure 1 A flowchart of a block panoramic video transmission method provided in the embodiment is shown in the figure. The method comprises the following steps:

[0063] S201: Obtain network bandwidth prediction results and user viewport prediction results.

[0064] In the embodiment, when the block panoramic video transmission is performed, when the user starts watching, the system needs to predict the next viewing angle of the user and predict the next network condition, and calculate the corresponding video block of the required code rate according to the predicted viewing angle and network condition. Therefore, in order to improve the transmission effect of the block panoramic video, first, the network bandwidth prediction results and the user viewport prediction results are obtained, so as to determine whether the prediction error occurs, and the video block and the corresponding reasonable code rate (i.e. the reasonable allocation of the video transmission code rate) required to be requested are calculated through subsequent steps S202-S205, so as to improve the transmission effect of the block panoramic video.

[0065] S202: When the user viewport prediction results are inconsistent with the real user viewport, the buffer replacement method is used to determine the reusable resources in the current buffer according to the intersection of the user viewport prediction results and the real user viewport, and determine the set of video blocks that need to be requested again.

[0066] In the embodiment, after the user viewport prediction results are obtained through step S201, further, when it is judged that the user viewport prediction results are inconsistent with the real user viewport, in order to solve the bandwidth waste caused by directly emptying the buffer when the viewport prediction fails in the prior art, the buffer replacement method is used to determine the reusable resources in the current buffer according to the intersection of the user viewport prediction results and the real user viewport, and determine the set of video blocks that need to be requested again.

[0067] Specifically, the buffer replacement method can be used to retain the available video blocks in the buffer, such as Figure 3The intersection of the video block set in the previous predicted user viewport region stored in the current buffer and the video block set in the newly predicted user viewport region is obtained first when the user viewport prediction failure occurs. Then the intersection is removed from the video block set in the newly predicted user viewport region to obtain the set of all video blocks that need to be re-requested. The system will compare each time segment in the buffer in turn and derive the video blocks that need to be re-requested in the next time segment. Finally, the video blocks that need to be re-requested are added to each time segment to obtain the set of all re-requested video blocks. This process can be represented by the following formula (5):

[0068]

[0069] wherein, represents the video block set in the user viewport region in the lth time segment stored in the buffer; represents the video block set in the user viewport region after the l time segments when the prediction fails. The method of buffer replacement is used in the present application to retain as many video blocks as possible in the current buffer. When the user viewport region prediction fails, the quality of the video blocks in the user viewport region can be adjusted in time, so that the bandwidth resources can be saved to request the video blocks in the new video segment by making full use of the currently available resources in the buffer, thereby improving the video quality of the user viewport region in the new time segment.

[0070] S203: Calculate the code rate of the video blocks to be requested according to the network bandwidth prediction result and the time length of the video blocks in the buffer.

[0071] It should be noted that although the video interruption problem caused by network fluctuation can be prevented by increasing the buffer length in the prior art, the video quality of the user viewport region will be affected and the bandwidth resources will be wasted. In the present embodiment, to solve the problem of video interruption that may occur when the network bandwidth prediction is inaccurate, the time of the video segments stored in the buffer (i.e. the time length of the video blocks in the buffer) is taken as a reference together with the predicted network bandwidth (i.e. the network bandwidth prediction result) as a reference standard for determining the total code rate of the requested blocks. In specific implementation, a predefined code rate adjustment function can be used to dynamically adjust the code rate of the video blocks to be requested according to the time length of the video blocks currently stored in the buffer, i.e. the calculation formula of the total code rate of the video blocks to be requested can be as follows:

[0072]

[0073] wherein, f(t buffer) represents a predefined bitrate adjustment factor, which dynamically adjusts the total bitrate of the requested video block based on the duration of the video block currently stored in the buffer. In this function, t buffer =L*t s .like Figure 4 As shown, when the length of the video segment stored in the buffer is less than L b When, f(t) buffer The function value should be less than 1 to reduce the total bitrate of the requested blocks. This prevents video interruptions caused by network bandwidth prediction failures and insufficient buffer length, which could result in the requested blocks not reaching the player before the buffer runs out. Simultaneously, as the buffer length decreases, the bitrate adjustment factor should be reduced to lower the total bitrate, increasing the buffer's input-output ratio and preventing video interruptions due to buffer exhaustion. When the length of the video segment stored in the buffer is greater than L... b When the buffer length is long enough, the total bitrate of the requested block can be increased to improve the video quality of the user viewport area. At the same time, the bitrate adjustment factor should also increase with the increase of the buffer length to increase the total bitrate and reduce the input-output ratio of the buffer to avoid wasting bandwidth resources due to excessive buffer length. The bitrate adjustment function is defined by a linear function (as shown in formula (7) below) and can dynamically adjust the total bitrate of the requested block.

[0074] f(t buffer )=k*t buffer +b (7)

[0075] Thus, in the buffer-controlled panoramic video bitrate allocation fault-tolerance mechanism proposed in this application, firstly, a buffer function is defined to determine the total bitrate of the requested video block based on the duration of the video block in the buffer. Secondly, when viewport prediction fails, a buffer replacement mechanism is used to determine the reusable resources in the current buffer and the set of video blocks that need to be re-requested. Finally, the two fault-tolerance mechanisms are integrated into the adaptive transmission process of panoramic video to avoid a decrease in the QoE of the panoramic video when prediction fails. This solves the problems of video interruption caused by prediction failure and video quality degradation in the viewport area during bitrate allocation.

[0076] S204: Calculate the video blocks and their bitrates that need to be requested based on the preset evaluation index of user experience quality (QoE) to maximize the QoE of the panoramic video.

[0077] It should be noted that, in order to improve the transmission effect of segmented panoramic video, this application pre-defines a QoE evaluation model and uses a bitrate adjustment function and a buffer replacement method to determine the total bitrate of the requested block and the set of video blocks that need to be re-requested, so as to ensure that the QoE of the panoramic video is maximized when the prediction fails.

[0078] where QoE refers to the quality of user experience. In the proposed chunked panoramic video transmission method, the system needs to assign appropriate bit rate to each requested video chunk to maximize the quality of user experience. QoE is a measure of user experience quality. The present application uses pre-defined evaluation metrics to define QoE, including user viewport quality evaluation metric, user viewport spatial quality smoothness evaluation metric, user viewport temporal quality smoothness evaluation metric, video interruption time evaluation metric. The viewport area video quality (user viewport quality evaluation metric) is crucial in QoE evaluation, because the higher it is, the better the user experience of the panoramic video. The temporal quality smoothness (user viewport temporal quality smoothness evaluation metric) and spatial quality smoothness (user viewport spatial quality smoothness evaluation metric) are unique metrics for chunked panoramic video evaluation, because panoramic video is processed in time slices, when the video quality between adjacent time slices varies greatly, it will cause users to have a clear perception of video quality switching, resulting in viewing discomfort. Therefore, the present application introduces temporal quality smoothness and spatial quality smoothness as indicators for panoramic video QoE evaluation. Video interruption is caused by video pause waiting for buffering due to buffer depletion and the arrival of new video chunks at the client, which will affect the user's viewing experience. Each metric is explained as follows:

[0079] (1) User viewport quality evaluation metric

[0080] In the chunked panoramic video, the present application uses video bit rate to quantify the video quality of the user viewport area. Because the higher the video bit rate, the better the video quality. Since the user viewport area may contain multiple video chunks, the method uses the average bit rate of all video chunks within the user viewport to represent the video quality within the user viewport, which uses f1(j) to evaluate the video quality of the user viewport area in the jth time slice, as shown in the following formula (8).

[0081]

[0082] (2) User viewport spatial quality smoothness evaluation metric

[0083] The method obtains the average bit rate of the video chunks within the user viewport by calculating the average of the bit rates of all video chunks in the area. Then by calculating the variance of the bit rate of each video chunk from the average bit rate, the dispersion degree of each video chunk from the average bit rate is obtained. The average of the variance of each video chunk is obtained to get the average dispersion degree of the bit rate of the video chunks within the viewport. Since the higher the dispersion degree indicates the lower the quality smoothness, the inverse of this average dispersion degree is used to quantify the smoothness of the spatial quality of the user viewport. The following formula (9) defines the solving process of the spatial quality smoothness f2(j).

[0084]

[0085] (3) User viewport time quality smoothness evaluation index

[0086] When the playing time segment is played, the client takes the next time segment from the buffer for playing. Since the two video segments are not a complete continuous video segment, the video quality may change during the transition. The method uses the absolute value of the difference between the average code rate of the video in the adjacent two time segments to represent the time smoothness of the video quality in the user viewport. The greater the absolute value, the worse the smoothness, and the worse the user experience. Therefore, the time smoothness needs to take the opposite value of the absolute value to represent. The following formula (10) defines the solving process of the time quality smoothness f3(j).

[0087] f3(j) = |f1(j) - f1(j-1)| (10)

[0088] (4) Video interruption time evaluation index

[0089] Video interruption time. That is, the video interruption duration in the corresponding time segment. Before the video blocks stored in the buffer are exhausted, the next time segment to be played does not arrive in time, which will cause video interruption. Therefore, the method uses the time difference between the request time of the video block and the duration of the video block stored in the buffer to represent. When the duration of the time segment stored in the buffer is greater than the request time, no video interruption occurs. That is, the video interruption time is 0. When the duration of the time segment stored in the buffer is less than the request time, the video interruption time is the difference between the two.

[0090]

[0091] In the above evaluation indexes, the video quality in the user viewport area, the time smoothness of the video quality in the user viewport, and the spatial smoothness of the video quality in the user viewport are all evaluation values. The greater the evaluation value, the higher the QoE of the corresponding panoramic video. And the video interruption time is the opposite, the longer the interruption duration, the lower the QoE of the panoramic video. Therefore, when calculating the QoE of the panoramic video, the video interruption time needs to be taken as the opposite value, so the QoE is represented by formula (12):

[0092] QoE(j) = αf1(j) + βf2(j) + γf3(j) - δf4(j) (12)

[0093] Wherein, α, β, γ, δ respectively represent the weight of the corresponding index in the whole QoE model. In different types of panoramic video, the value of the weight value will also be different, the specific value is not limited, which can be set according to the actual situation and experience value.

[0094] Further, in the adaptive transmission of panoramic video, video blocks need to be prefetched into the buffer through prediction. In this process, the bit rate of the prefetched video resource needs to be determined through user viewpoint prediction and network to maximize the QoE of the panoramic video at a future time. However, since the application adopts a cache replacement mechanism to solve the problem of deviation of video content stored in the buffer when user viewpoint prediction fails, the video blocks that need to be requested at the same time may cover multiple time segments. Therefore, the sum of the QoEs of each time segment needs to be calculated to select a bit rate allocation scheme. Since the result of user viewpoint prediction becomes more and more unstable over time, the calculation of QoE will also be affected. Therefore, this method uses the QoE calculation result of each time segment multiplied by the weight of the accuracy of user viewpoint prediction to distinguish the degree of influence of the QoE calculation result of different time segments on the bit rate allocation decision. The QoE expectation of the panoramic video can be calculated by the following formula (13).

[0095] E(QoE) = ω1QoE(j c +1) + ω2QoE(j c +2) + … + ω L QoE(j c +L) + ω L+1 QoE(j c +L+1) (13)

[0096] wherein QoE(j c +n) represents the value of QoE of the nth time segment after the current time segment. ω n represents the weight of QoE of the nth time segment after the current time segment. The definition of the QoE weight of each time segment depends on the accuracy of user viewpoint prediction in the nth time segment after the current time segment. The higher the accuracy, the greater the weight. That is, as time increases, the accuracy of user viewpoint prediction gradually decreases, that is, the QoE weight of the corresponding time segment is lower. It can be seen that in this formula, the time segment is at most L+1. Because in the adaptive transmission system, the number of time segments stored in the buffer is L, if cache replacement is needed, the number of time segments covered by replacement is at most L, and in addition to the failed resource, new time segment video resources also need to be requested, so the number of time segments covered by the QoE expectation is at most L+1.

[0097] S205: According to the set of video blocks that need to be re-requested, the bit rate of the video blocks to be requested, the current video blocks that need to be requested and their bit rates, the final video blocks that need to be requested and their corresponding bit rates are calculated to realize the transmission of the blocked panoramic video.

[0098] In the embodiment, the set of video blocks that need to be re-requested is determined by step S202, the code rate of the video blocks to be requested is calculated by step S203, and the video blocks currently needed to be requested and their code rates are calculated by step S204 so as to maximize the QoE of the panoramic video. After that, the video blocks needed to be requested and their corresponding code rates can be calculated by using the heuristic search algorithm Beam Search according to the factors such as the predicted viewing angle and the predicted network condition, and the video blocks are requested and stored in the buffer for the user to watch.

[0099] Beam Search is a heuristic search algorithm that uses breadth-first search to build a search tree, which can reduce the memory requirement but is not necessarily the global optimal solution.

[0100] Specifically, in the adaptive panoramic video streaming process, a specific code rate is allocated to each video block to be requested according to the predicted user viewport region and the predicted network bandwidth, and a request is sent to the server. In the current environment, a code rate allocation scheme that can maximize the QoE of the panoramic video stream needs to be found. That is, the goal of the system is Max E(QoE). Since the method of re-requesting the video blocks that cannot be reused in the buffer when the user viewport prediction fails is adopted in the present application, the total number of video blocks needed to be requested will include the video blocks needed to be re-requested and the video blocks needed to be newly requested, obtaining a set of T(i, j, k). The video blocks in the user's non-viewport region in all newly requested video segments are always allocated with the lowest code rate, and then a request is sent to the server for these video blocks. This process is shown in the following formulas (14)-(16).

[0101]

[0102] R = <r1, r2, r3,...> (15)

[0103] ∑ r∈R r≤R 总 (16)

[0104] Since the optimal request scheme needs to be quickly determined in the transmission of the panoramic video, Beam Search is used to solve a relatively large space graph, reducing the space consumption and time efficiency. In each step of depth expansion, the nodes with lower quality are pruned, and the nodes with higher quality are retained. Each video block is regarded as a node for depth expansion. For each video block, the higher the allocated code rate, the higher the QoE, but the more bandwidth resources it occupies. Therefore, the QoE is not the only scoring standard, and the bandwidth resources occupied by the video blocks also need to be considered.

[0105] Score = (1 - r i / R 总)*E(QoE) (17)

[0106] Taking the above formula (17) as the scoring standard, the application scores and sorts the nodes of each layer, selects the top B nodes, and then classifies them, finally obtaining the optimal code rate allocation sequence. While ensuring the QoE of panoramic video, the code rate proportion is also reasonably allocated.

[0107] Beam Search adopts a breadth-first strategy, builds a search tree at each layer of the tree, sorts the nodes according to the heuristic cost, and then only leaves a predetermined number of nodes (n, Beam Width-Bundle Width). Only these nodes continue to expand at the next level, and other nodes are cut off. Algorithm 1 is a specific implementation of the rate allocation fault-tolerant algorithm based on buffer control. The steps of using Beam Search to select the code rate allocation scheme are as follows:

[0108] 1. Insert the initial node (different code rate versions of the requested block) into the list.

[0109] 2. Select the top n nodes from the heap according to the score. If the node is the target node, the algorithm ends.

[0110] 3. Otherwise, expand the node, list the different code rate versions of the next node, select the score combined with the current node, then sort the expanded nodes according to the heuristic cost, and take the top n nodes into the heap. Then continue step 2.

[0111] 4. The end condition of the algorithm is to find the optimal solution or the heap is empty.

[0112] In this way, by using the defined QoE evaluation model, the video code rate, viewport area time / space quality change, and video interruption time are considered comprehensively. The BeamSearch algorithm is used to screen the optimal allocation scheme, and according to this scheme, the corresponding code rate is allocated for each video block to be requested. The code rate allocation mechanism with fault-tolerant control based on buffer is realized, and the user experience quality of panoramic video is improved.

[0113] In conclusion, the block panoramic video transmission method provided in the embodiment first acquires a network bandwidth prediction result and a user viewport prediction result, then when the user viewport prediction result is inconsistent with the real user viewport, determines the reusable resources in the current buffer and the set of video blocks that need to be requested again by using a buffer replacement method according to the intersection of the user viewport prediction result and the real user viewport, next calculates the code rate of the video blocks to be requested according to the network bandwidth prediction result and the time length of the video blocks in the buffer, and then calculates the video blocks and their code rates that need to be requested currently according to preset evaluation indexes of quality of experience (QoE) of the panoramic video, so that the QoE of the panoramic video is maximized, and finally calculates the video blocks and their corresponding code rates that need to be requested finally according to the set of video blocks that need to be requested again, the code rate of the video blocks to be requested, the video blocks and their code rates that need to be requested currently, so as to realize block panoramic video transmission. Therefore, the transmission effect of the block panoramic video can be improved.

[0114] Second embodiment

[0115] The embodiment will introduce a block panoramic video transmission device, and the related content can be referred to the method embodiment.

[0116] Referring to Figure 5 The block panoramic video transmission device provided in the embodiment includes:

[0117] The acquisition unit 501 is configured to acquire a network bandwidth prediction result and a user viewport prediction result.

[0118] The determination unit 502 is configured to, when the user viewport prediction result is inconsistent with the real user viewport, determine the reusable resources in the current buffer and the set of video blocks that need to be requested again by using a buffer replacement method according to the intersection of the user viewport prediction result and the real user viewport.

[0119] The first calculation unit 503 is configured to calculate the code rate of the video blocks to be requested according to the network bandwidth prediction result and the time length of the video blocks in the buffer.

[0120] The second calculation unit 504 is configured to calculate the video blocks and their code rates that need to be requested currently according to preset evaluation indexes of quality of experience (QoE) of the panoramic video, so that the QoE of the panoramic video is maximized.

[0121] The third calculation unit 505 is configured to calculate the video blocks and their corresponding code rates that need to be requested finally according to the set of video blocks that need to be requested again, the code rate of the video blocks to be requested, the video blocks and their code rates that need to be requested currently, so as to realize block panoramic video transmission.

[0122] In an implementation form of the embodiment, the first calculating unit 503 is specifically configured to:

[0123] The code rate of the video block to be requested is dynamically adjusted according to the time length of the video block currently stored in the buffer by using a predefined code rate adjustment function.

[0124] In an implementation form of the embodiment, the preset evaluation indexes of the QoE include a user viewport quality evaluation index, a user viewport spatial quality smoothness evaluation index, a user viewport temporal quality smoothness evaluation index, and a video interruption time evaluation index.

[0125] In an implementation form of the embodiment, the third calculating unit 505 is specifically configured to:

[0126] The final video block to be requested and the corresponding code rate are calculated by using a heuristic search algorithm BeamSearch according to the set of video blocks to be re-requested, the code rate of the video block to be requested, and the video block to be currently requested and the code rate thereof, so as to implement the transmission of the block-based panoramic video.

[0127] To sum up, the block-based panoramic video transmission device provided by the embodiment first acquires a network bandwidth prediction result and a user viewport prediction result, then when the user viewport prediction result is inconsistent with the real user viewport, the intersection of the user viewport prediction result and the real user viewport is determined by using a buffer replacement method, the set of video blocks to be re-requested is determined according to the intersection, then the code rate of the video block to be requested is calculated according to the network bandwidth prediction result and the time length of the video block in the buffer, the video block to be currently requested and the code rate thereof are calculated according to the preset evaluation indexes of the quality of experience QoE, so as to maximize the QoE of the panoramic video, and finally the final video block to be requested and the corresponding code rate are calculated according to the set of video blocks to be re-requested, the code rate of the video block to be requested, and the video block to be currently requested and the code rate thereof, so as to implement the transmission of the block-based panoramic video. Therefore, the transmission effect of the block-based panoramic video can be improved.

[0128] Further, the embodiment of the application further provides a block-based panoramic video transmission device, which comprises a processor, a memory, and a system bus.

[0129] The processor and the memory are connected through the system bus.

[0130] The memory is used to store one or more programs, and the one or more programs comprise instructions which, when executed by the processor, cause the processor to execute any implementation method of the block-based panoramic video transmission method.

[0131] Further, the embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions run on a terminal device, the terminal device executes any one of the above-mentioned implementation methods of the block panoramic video transmission method.

[0132] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software and necessary universal hardware platforms. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0133] It should be noted that the various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be mutually referred to. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts are referred to the method part.

[0134] It should also be noted that the terms such as first and second in the present document are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0135] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for transmitting a tiled panoramic video, the method comprising: The method comprises the following steps: obtaining a network bandwidth prediction result and a user viewport prediction result; when the user viewport prediction result is inconsistent with a real user viewport, determining a set of video blocks that need to be re-requested and resources that can be reused in a current buffer according to an intersection of the user viewport prediction result and the real user viewport by using a buffer replacement method; calculating a code rate of a video block to be requested according to the network bandwidth prediction result and a time length of a video block in the buffer; calculating the video block to be requested and the code rate of the video block to be requested according to a preset evaluation index of a quality of experience (QoE) so as to maximize the QoE of the panoramic video; calculating a final video block to be requested and a corresponding code rate of the final video block to be requested according to the set of video blocks that need to be re-requested, the code rate of the video block to be requested and the video block to be requested so as to realize transmission of the block panoramic video.

2. The method of claim 1, wherein, The method further comprises the following steps: dynamically adjusting the code rate of the video block to be requested according to the time length of the video block currently stored in the buffer by using a predefined code rate adjustment function.

3. The method of claim 1, wherein, The preset evaluation index of the QoE comprises a user viewport quality evaluation index, a user viewport spatial quality smoothness evaluation index, a user viewport time quality smoothness evaluation index and a video interruption time evaluation index.

4. The method of claim 1, wherein, The method further comprises the following steps: calculating the final video block to be requested and the corresponding code rate of the final video block to be requested by using a heuristic search algorithm (Beam Search) according to the set of video blocks that need to be re-requested, the code rate of the video block to be requested and the video block to be requested so as to realize the transmission of the block panoramic video.

5. A patchwork panoramic video transmission apparatus characterized by comprising: The method comprises the following steps: an obtaining unit, configured to obtain a network bandwidth prediction result and a user viewport prediction result; a determining unit, configured to, when the user viewport prediction result is inconsistent with a real user viewport, determine a set of video blocks that need to be re-requested and resources that can be reused in a current buffer according to an intersection of the user viewport prediction result and the real user viewport by using a buffer replacement method; a first calculating unit, configured to calculate a code rate of a video block to be requested according to the network bandwidth prediction result and a time length of a video block in the buffer; a second calculating unit, configured to calculate the video block to be requested and the code rate of the video block to be requested according to a preset evaluation index of a quality of experience (QoE) so as to maximize the QoE of the panoramic video; a third calculating unit, configured to calculate a final video block to be requested and a corresponding code rate of the final video block to be requested according to the set of video blocks that need to be re-requested, the code rate of the video block to be requested and the video block to be requested so as to realize transmission of the block panoramic video.

6. The apparatus of claim 5, wherein, The first calculating unit is specifically configured to: A predefined rate adjustment function is used to dynamically adjust the rate of the video block to be requested according to the time length of the video block currently stored in the buffer.

7. The apparatus of claim 5, wherein, The preset evaluation index of the QoE includes a user viewport quality evaluation index, a user viewport spatial quality smoothness evaluation index, a user viewport time quality smoothness evaluation index, and a video interruption time evaluation index.

8. The apparatus of claim 5, wherein, The third calculation unit is specifically configured to: According to the set of video blocks that need to be re-requested, the rate of the video block to be requested, the current video block that needs to be requested and the rate thereof, a heuristic search algorithm Beam Search is used to calculate the final video block that needs to be requested and the corresponding rate thereof, so as to realize the transmission of the block panoramic video.

9. A tiled panoramic video transmission apparatus, characterized by, Comprise: A processor, a memory, a system bus; The processor and the memory are connected through the system bus; The memory is used for storing one or more programs, the one or more programs include instructions, the instructions when being executed by the processor make the processor execute the method in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium has instructions stored therein, when the instructions run on the terminal equipment, make the terminal equipment execute the method in any one of claims 1-4.

Images