Method and system for transmitting content in immersive live stream, and storage medium

Abstract

The invention provides a method and a system for transmitting content in an immersive live broadcast stream and a storage medium. The method comprises the following steps: acquiring head motion data of a user in a preorder time period of watching live broadcast; calculating a dynamic vision field range of the user in a subsequent time period according to the motion data; establishing an adaptive model of a dynamic relationship between the block duration and the dynamic vision field range, inputting a real-time network bandwidth and the dynamic vision field range into the adaptive model, and generating the dynamic relationship between the dynamic vision field range and the block duration; and introducing the dynamic vision field range and the block duration as optimization variables into a target function, generating a relative weight between the dynamic vision field range and the block duration, and performing content transmission of the live broadcast stream according to the relative weight. The problems of black edges, jamming or redundant broadband consumption in the process of watching the immersive live broadcast are solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to methods, systems and storage media for transmitting content in immersive live streaming. Background Technology

[0002] As a disruptive technology in next-generation multimedia applications, immersive media constructs a three-dimensional, surround-like environment, enabling viewers to interact naturally and immersively, rather than through linear narratives. This paradigm shift has fundamentally changed the sense of presence in media consumption, and is rapidly entering the mainstream, widely used in fields such as sports broadcasting and virtual concerts. However, this trend also places dual stringent demands on transmission infrastructure: systems must maintain millisecond-level end-to-end latency while ensuring a smooth and seamless immersive experience. Achieving this goal faces fundamental technical challenges.

[0003] Specifically, the main challenges currently faced by immersive live streaming include: fluctuations in network bandwidth and user head movements and scene complexity affecting the viewing experience; the real-time generation of data in live streaming makes the future system state unpredictable and highly uncertain; and insufficient dynamism during the viewing process leads to black borders and stuttering, resulting in a poor viewing experience. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a method, system and storage medium for transmitting content in an immersive live stream, which solves the problems of black borders, stuttering or redundant bandwidth consumption during the viewing of immersive live streams.

[0005] To achieve one of the aforementioned objectives, this application provides a method for transmitting content in an immersive live stream, the method comprising: Acquire head movement data of users during the time period preceding their live stream viewing; Based on the motion data, calculate the user's dynamic field of view range in the subsequent time period; An adaptation model is established to show the dynamic relationship between the block duration of a video block and the dynamic field of view. The real-time network bandwidth and the dynamic field of view are input into the adaptation model to generate the dynamic relationship between the dynamic field of view and the block duration. The dynamic view range and the block duration are introduced as optimization variables into the objective function to generate a relative weight between the dynamic view range and the block duration, and the content transmission of the live stream is performed based on the relative weight.

[0006] As a further improvement to one embodiment of this application, the step of calculating the user's dynamic field of view range in a subsequent time period based on the motion data includes: The normal distribution of the user's head displacement in subsequent time periods is calculated using the following formula: ; ; in, The displacement is in the first direction. L represents the displacement in the second direction, where the first and second directions are perpendicular to each other. t v is the duration of the subsequent time period. x，t-1 v is the velocity in the first direction during the next time interval. y，t-1 The velocity in the first direction during the next time period. and It is the variance of the normal distribution.

[0007] As a further improvement to one embodiment of this application, the step of inputting the real-time network bandwidth and the dynamic field of view into the adaptation model to generate a dynamic relationship between the dynamic field of view and the block duration includes: Define a single transmission unit, which is a coded block located at a tile position within a time length; The data volume of the transmission unit is calculated according to the following formula: ; Among them, S t M represents the amount of data. t Here, m and n represent the width and height of the user's current field of view, respectively, and k is the bitrate. t L represents the number of tile layers extending in each direction. t The duration is the specified time. The number of tile layers extended in each direction is the quantized value of the dynamic field of view, and the time length is the quantized value of the block duration, using k. t and L t The coupling represents the dynamic relationship between the dynamic view range and the block duration.

[0008] As a further improvement to one embodiment of this application, the objective function is expressed as: ; Where η is the balance coefficient, L stall Let L be the expected value of the stuttering loss. black This represents the expected value of the black border loss.

[0009] As a further improvement of one embodiment of this application, the user's head movement state includes: a stable head or frequent head movements; The duration of time when the head is stable is greater than the duration of time when the head moves frequently. The number of tile layers expanding in each direction when the head is stable is less than the number of tile layers expanding in each direction when the head moves frequently.

[0010] As a further improvement to one embodiment of this application, based on the adaptation model, the expected loss expression for when stuttering occurs is obtained: ; Where Φ() is the cumulative distribution function of the standard normal distribution, and μ V With σ V Let B be the log-normal distribution parameter of the bandwidth. t S is the buffer size when the t-th video block is downloaded. t The amount of data mentioned above; Based on the aforementioned adaptation model, the expected loss expression for the black border is generated as follows: ; Among them, S i With P i These represent the black border area and corresponding probability under different motion states, I i This is a scene indicator function.

[0011] As a further improvement to one embodiment of this application, before introducing the dynamic field of view and the block duration as optimization variables into the objective function, the following steps are included: By introducing a Bayesian optimization engine based on residual Gaussian processes, the model L(L) in the objective function is analyzed. t k t The residual term is used to characterize the difference between the theoretical and actual results, and an optimized objective function is generated.

[0012] As a further improvement to one embodiment of this application, the model L(L) in the objective function is analyzed. t k t )include: The correlation between the block duration and the dynamic view range is characterized by the following formula: ; ; ; Where x is the input variable, and the kernel function K(x, x') adopts the radial basis function K corresponding to the continuous variable. L (L, L'), and the Hamming kernel K corresponding to the discrete variables. k (k, k`); During the optimization process, the user's interaction speed and network conditions within the most recent d block periods are continuously monitored to form a dataset, represented as: ; Among them, D t For the dataset; The following formula describes the difference between theoretical and actual results: For the current candidate points The posterior distribution of the residual process can be expressed as: ; ; ; Where, μ(x) t ) represents the predicted mean, σ 2 (x) t ) represents the prediction variance, |X| represents the search space size, and δ represents the confidence threshold.

[0013] Based on the same inventive concept, this application also provides a system for transmitting content in an immersive live stream, comprising: The acquisition module is used to acquire head movement data of users in the time period preceding the live stream viewing. The first calculation module is used to calculate the user's dynamic field of view range in a subsequent time period based on the motion data. The generation module establishes an adaptation model, inputs the real-time network bandwidth and the dynamic field of view into the adaptation model, and generates a dynamic relationship between the dynamic field of view and the block duration. The transmission module is used to introduce the dynamic view range and the block duration as optimization variables into the objective function, generate the relative weight between the dynamic view range and the block duration, and transmit the live stream according to the relative weight.

[0014] Based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to execute any of the above-described methods for transmitting content in an immersive live stream.

[0015] Compared to existing technologies, the technical advantages of this invention are as follows: When targeting immersive live streaming, it utilizes the inherent correlation between user head movement patterns, block duration, and dynamic field of view. Based on the prediction of head movement, the dynamic field of view is obtained, and a dynamic relationship between block duration and dynamic field of view is established in conjunction with network bandwidth fluctuations. By leveraging the information decision-making capabilities of the adaptation model and optimizing the objective function, a dynamic solution is found among network bandwidth fluctuations, the dynamic field of view corresponding to user head movement, and block duration, optimizing the transmission efficiency of the live stream and improving the quality of the user's immersive experience. This solves the problem of maximizing transmission efficiency and minimizing latency and black border occurrence rates in immersive live streaming under limited bandwidth and terminal computing resources, based on precise analysis of user interaction dynamics and network status, ensuring the continuity and smoothness of the immersive experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the implementation methods or related technologies will be briefly introduced below. Obviously, the drawings described below are only the implementation methods of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for transmitting content in an immersive live stream, provided as an embodiment of this application; Figure 2 A framework diagram of a method for transmitting content in an immersive live stream, provided for one embodiment of this application; Figure 3 A schematic diagram of a content transmission system in an immersive live stream provided for an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided for an embodiment of this application. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by those skilled in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects.

[0020] The development of existing immersive live streaming places dual stringent demands on transmission infrastructure: the system must maintain millisecond-level end-to-end latency while ensuring a smooth and seamless immersive experience. Achieving this goal faces fundamental technical challenges. According to the ITU-T P.1203 standard, raw 8K panoramic video requires a bitrate of 1.5–2 Gbps. This contradiction between ultra-high bandwidth requirements and strict real-time constraints has become a key bottleneck limiting the large-scale deployment of immersive streaming media systems.

[0021] To address these challenges, the industry has widely adopted the View-Related Streaming (VDS) strategy. This strategy leverages the fact that the human visual system primarily perceives a limited area (typically only covering 12%-20% of the entire scene), selectively streaming the area currently observable by the user (often called the field of view, FoV), thereby reducing bandwidth usage. However, in practical deployments, VDS has two significant limitations: (i) visual discontinuities due to prediction errors at the boundaries of the field of view; and (ii) bandwidth waste caused by redundant streaming. Another strategy, called Extended Field of View Streaming, introduces a static spatial buffer to avoid excessive data waste in potentially observable areas. This is a valuable trade-off that helps alleviate streaming problems. However, empirical evidence suggests that black borders can occur up to 70% of the time when the user rapidly turns their head (more than 50 degrees per second). The fixed extended field of view and the duration of individual data blocks severely hinder the system's efficiency in dynamic environments.

[0022] Specifically, current immersive live streaming content delivery faces the challenge of providing smooth, black-bar-free content without wasting bandwidth. When providing a live stream with a sufficiently large field of view, content outside the user's field of vision wastes network bandwidth, and in reality, existing infrastructure struggles to support the smoothness of massive data volumes, resulting in stuttering during viewing. When the field of view is narrowed based on actual bandwidth, it becomes difficult to track changes in the user's head, causing black bars to appear within the user's field of vision, impacting the immersive experience.

[0023] This application provides a method for transmitting content in an immersive live stream, such as... Figure 1 and Figure 2 As shown, it includes the following steps: Step S100: Obtain head movement data of the user during the period preceding the live stream.

[0024] Specifically, when watching immersive live streams, the user's field of vision changes are mainly driven by head movements. Based on the statistical relationship between the user's visual displacement and time during the viewing process, head movements are regarded as a continuous random process. By jointly modeling the head velocity and acceleration in the preceding time period, the dynamic estimation of the future field of vision range can be achieved.

[0025] Step S200: Calculate the user's dynamic field of view range in the subsequent time period based on the motion data.

[0026] Specifically, based on the pattern of user head movement, the range of user head movement in subsequent time periods can be predicted, thereby predicting the boundary of the potential user field of view within a block duration. The image within the boundary is the dynamic field of view.

[0027] It should be noted that the subsequent time period here is calculated based on the head velocity and acceleration in the preceding time period. Therefore, it predicts the user's head movement in an instant and does not predict subjective changes made by the user due to viewing intentions or live content.

[0028] In one possible implementation of this application, step S200 includes: The normal distribution of the user's head displacement in subsequent time periods can be calculated using the following formula: ; ; in, The displacement is in the first direction. L represents the displacement in the second direction, where the first and second directions are perpendicular to each other. t v represents the duration of the subsequent time interval, i.e., the duration of the block. x，t-1 v is the velocity in the first direction during the next time interval. y，t-1 The velocity in the first direction during the next time period. and It is the variance of the normal distribution.

[0029] Specifically, we first define the average head movement velocity of the user during the preceding time as v. t It is decomposed into horizontal components v xt With the vertical component v ytTo establish an interpretable and computationally inexpensive predictive mathematical model, the model assumes that the acceleration follows a zero-mean Gaussian distribution and is independent in the horizontal and vertical directions, as expressed by the formula: ; Based on this, by integrating the change in velocity over time, we can obtain the velocity over time L. t The expected displacement of the viewpoint within the field of view. Since acceleration and velocity have a first-order derivative relationship, and velocity and displacement also have a first-order derivative relationship, the user's horizontal and vertical displacements follow a normal distribution. This mathematical model effectively characterizes the influence of velocity fluctuations during head rotation by introducing a random perturbation term, and can predict the spatial range that the future viewpoint may cover, i.e., the dynamic field of view, given the current velocity.

[0030] During operation, the quaternion representation of the user's head pose is collected in real time and updated with a sampling frequency. The duration L of the next video segment is estimated using the velocity and acceleration distribution at the current moment. t The potential field of view shift for users within the site. Combined with the spatially discrete division of the area, the range of the prefetching area is dynamically adjusted at the transmission end to cover as much of the area the user might view in the next moment as possible, under bandwidth-constrained conditions.

[0031] Predicting head movements based on physical laws is lightweight, highly interpretable, and computable in real time. It can work independently or in combination with deep learning vision prediction models. By working in conjunction with subsequent adaptation models and objective functions, it can balance the probability of black borders and playback latency in both spatial and temporal dimensions, thereby significantly improving the stability of immersive live streaming experience and network bandwidth utilization efficiency.

[0032] Step S300: Establish an adaptation model for the dynamic relationship between the block duration of a video block and the dynamic field of view. Input the real-time network bandwidth and the dynamic field of view into the adaptation model to generate the dynamic relationship between the dynamic field of view and the block duration.

[0033] Specifically, block duration refers to a unit of video length. Within this unit, the dynamic field of view is fixed. Therefore, based on the determined pattern of the user's head movement, the longer the block duration, the larger the dynamic field of view. The dynamic field of view must satisfy the range of head movement of the user within the block duration. However, the larger the dynamic field of view, the higher the network bandwidth requirements. Therefore, it is necessary to find a dynamic relationship between block duration and dynamic field of view in the context of fluctuating network bandwidth and the pattern of the user's head movement to ensure smooth playback and no black borders in the user's field of view.

[0034] Based on user head motion prediction, an adaptation model is established to dynamically determine the relationship between block duration and dynamic field of view. This model aims to determine the optimal coding block length L within a given time window based on the joint dynamics of the user's field of view change rate and network bandwidth conditions. t With the number of extended tile layers k t This balances the risk of black borders and playback latency in both spatial and temporal dimensions, ensuring the stability and efficiency of immersive video transmission.

[0035] In one possible implementation of this application, step S300 includes: Step S310: Define a single transmission unit, which is a coded block located at a tile position within a time length.

[0036] Specifically, the video stream is divided into many small regions, which are called tiles. The coded blocks at the tile positions are usually represented by a two-dimensional coordinate (w, h).

[0037] Step S320: Calculate the data volume of the transmission unit according to the following formula: ; Among them, S t For the amount of data, M t Here, m and n represent the width and height of the user's current field of view, respectively, and k is the bitrate. t L represents the number of tile layers extending in each direction. t The duration; The number of tile layers extended in each direction is the quantized value of the dynamic view range, and the time length is the quantized value of the block duration, using k. t and L t The coupling represents the dynamic relationship between the dynamic view range and the block duration.

[0038] Specifically, with the length of time L t Increasing k enhances continuity over time, but also leads to increased redundant data and slower interactive responses; while increasing k... t This provides redundancy in the spatial dimension to reduce the probability of black borders appearing, but it increases bandwidth consumption. Therefore, L t With k t The choice involves a coupling trade-off.

[0039] In the specific implementable method of this application, S t Indicates the data size, in MB.

[0040] In one possible implementation of this application, the duration of time when the head is stable is greater than the duration of time when the head moves frequently. The number of tile layers that expand in each direction when the head is stable is less than the number of tile layers that expand in each direction when the head moves frequently.

[0041] Specifically, when the user's head movement is stable, a longer block duration and a smaller dynamic field of view are preferred to reduce transmission redundancy and improve coding efficiency; when high-speed or unstable head movement is detected, the block duration is shortened and the dynamic field of view is increased to improve the inter-frame update frequency and expand the field of view, thereby maintaining a continuous viewing experience under uncertain conditions.

[0042] In one possible implementation of this application, based on the adaptation model, the expected loss expression for when stuttering occurs is obtained: ; Where Φ() is the cumulative distribution function of the standard normal distribution, and μ V With σ V Let B be the log-normal distribution parameter of the bandwidth. t Let S be the buffer size when the t-th video block is downloaded. t For data volume.

[0043] Specifically, during user viewing, playback stuttering mainly occurs when the download time exceeds the available buffer. Modeling this stuttering event, its probability of occurrence can be expressed as: ; Based on the probability of stuttering, combined with download time, data volume, and buffer status, the expected loss expression for stuttering is obtained. The expected loss expression for stuttering reflects the transmission delay and buffer exhaustion risk caused by excessively long block durations. While increasing the dynamic field of view can alleviate the risk of black borders, it also increases the probability of stuttering due to the increased data volume.

[0044] Based on the adaptation model, the expected loss expression for generating black borders is as follows: ; Among them, S i With P i These represent the black border area and corresponding probability under different motion states, I i This is a scene indicator function.

[0045] Specifically, shorter block durations or larger dynamic field of view can effectively reduce the probability of black borders, but at the same time, they will increase system bandwidth and bandwidth overhead during processing.

[0046] The above-mentioned loss expectation expressions for stuttering and black borders are used to represent the dynamic relationship between block duration and dynamic view range.

[0047] It should be noted that the core function of the scene indicator function is to mark whether the user's head displacement falls into the i-th type of black border scene. It only takes a value of 1 when the corresponding scene occurs, and a value of 0 when it does not occur. It is the quantization black border loss L. black The key logical variable is 'i', which represents different scenarios. In the design, the following scenarios are defined as scenarios 1-5: the displacement is completely within the transmission area with no black border; the horizontal displacement exceeds the transmission area, but the vertical displacement is within the range; the vertical displacement exceeds the transmission area, but the horizontal displacement is within the range; both the horizontal and vertical displacements exceed the transmission area, but part of the transmission area is still within the field of view; and all remaining cases, such as the displacement completely exceeding the transmission area.

[0048] For example, when the user's head displacement belongs to Case 1 (displacement is entirely within the transmission area, with no black border), I1=1, while I2-I5=0, indicating that only the data information of this scenario is included when the first scenario occurs. I is essentially a "filter," using its 0 / 1 value to only include the black border area and the probability of the scenario occurrence corresponding to the current actual scenario in the calculation, ensuring that the black border loss only reflects the actual black border situation and avoiding interference from irrelevant scenarios on the results.

[0049] Step S400: The dynamic view range and block duration are introduced into the objective function as optimization variables to generate the relative weight between the dynamic view range and block duration, and the content transmission of the live stream is performed based on the relative weight.

[0050] Specifically, the dynamic relationships in the above steps illustrate that block duration and dynamic view range need to be continuously adjusted within the dynamics of the live stream to ensure a good viewing experience for users throughout the entire viewing process. The optimized objective function is used to weigh these dynamic relationships and make a definite decision instantaneously.

[0051] In one possible implementation of this application, the duration of time when the head is stable is greater than the duration of time when the head moves frequently. The number of tile layers that expand in each direction when the head is stable is less than the number of tile layers that expand in each direction when the head moves frequently.

[0052] Specifically, when the user's head movement is stable, a longer block duration and a smaller dynamic field of view are preferred to reduce transmission redundancy and improve coding efficiency; when high-speed or unstable head movement is detected, the block duration is shortened and the dynamic field of view is increased to improve the inter-frame update frequency and expand the field of view, thereby maintaining a continuous viewing experience under uncertain conditions.

[0053] In one possible implementation of this application, the objective function in step S400 is expressed as: ; Where η is the balance coefficient, Lstall Let L be the expected value of the stuttering loss. black This represents the expected value of the black border loss.

[0054] Specifically, a unified bivariate control mechanism is adopted, proposing a single optimization framework that integrates continuous block duration and discrete field of view radius. By treating block duration and dynamic field of view range as two core controllable variables, a dynamic balance between playback latency and the probability of black border appearance is constructed under a given environment, thereby achieving a stable and optimal immersive viewing experience. After considering the coupled effects of the temporal and spatial dimensions, block duration and expansion radius can be used as joint optimization variables and introduced into the objective function to reveal the dynamic relationship between the two.

[0055] In one possible implementation of this application, before introducing the dynamic view range and block duration as optimization variables into the objective function in step S400, it is necessary to solve the non-convex, stochastic optimization problem described in the objective function that simultaneously includes continuous and discrete variables, including: By introducing a Bayesian optimization engine based on residual Gaussian processes, the model L(L) in the objective function is analyzed. t k t The residual term is used to characterize the difference between the theoretical and actual results, and an optimized objective function is generated.

[0056] In one possible implementation of this application, the model L(L) in the analytical objective function is... t k t )include: The correlation between block duration and dynamic field of view range can be characterized by the following formula: ; ; ; Where x is the input variable, and the kernel function K(x, x') adopts the radial basis function K corresponding to the continuous variable. L (L, L'), and the Hamming kernel K corresponding to the discrete variables. k (k, k`).

[0057] Specifically, by using radial basis functions and Hamming kernels to characterize the correlation on time and spatial scales, this hybrid kernel structure ensures the smooth differentiability and learnability of the model in the continuous-discrete mixed input space, enabling the optimization framework that introduces the objective function to handle different types of control parameters in a unified form.

[0058] During the optimization process, the user's interaction speed and network conditions within the most recent d block periods are continuously monitored to form a dataset, represented as: ; Among them, D t For dataset; The following formula describes the difference between theoretical and actual results: For the current candidate points The posterior distribution of the residual process can be expressed as: ; ; ; Where, μ(x) t ) represents the predicted mean, σ 2 (x) t ) represents the prediction variance, |X| represents the search space size, and δ represents the confidence threshold.

[0059] Specifically, the upper confidence bound criterion is used as the acquisition function x. t Balance exploration and utilization, parameter β t Adaptive adjustments are made using the above formula. By incorporating the upper confidence bound criterion and introducing an optimization framework for the objective function, the space of positional parameters can be efficiently explored while ensuring stability, gradually converging to the optimal combination. .

[0060] The method for transmitting content in an immersive live stream disclosed in this application first determines the dynamic field of view within a block duration by calculating the user's head movement patterns. Then, considering the implemented broadband network conditions, the dynamic relationship between the block duration and the dynamic field of view is obtained through an adaptation model, enabling precise and real-time adjustment of transmission parameters. Furthermore, a residual Gaussian process Bayesian optimization engine is used to adjust the decision parameters in the objective function, balancing the weight relationship between the block duration and the dynamic field of view to achieve the most perceptible loss balance of "low stuttering, fewer black borders, and high bandwidth efficiency," thereby further maximizing the quality of the user's immersive experience.

[0061] Another embodiment of this application discloses a content transmission system for immersive live streaming, such as... Figure 3 As shown, it includes: The acquisition module is used to acquire head movement data of users in the time period preceding the live stream viewing. The first calculation module is used to calculate the user's dynamic field of view range in a subsequent time period based on motion data. The generation module establishes an adaptation model by inputting real-time network bandwidth and dynamic field of view into the adaptation model, thereby generating a dynamic relationship between the dynamic field of view and the block duration. The transmission module is used to introduce the dynamic view range and block duration as optimization variables into the objective function, generate the relative weight between the dynamic view range and block duration, and transmit the live stream according to the relative weight.

[0062] In one possible implementation of this application, the first computing module includes: The normal distribution of the user's head displacement in subsequent time periods can be calculated using the following formula: ; ; in, The displacement is in the first direction. L represents the displacement in the second direction, where the first and second directions are perpendicular to each other. t v represents the duration of the subsequent time period. x，t-1 v is the velocity in the first direction during the next time interval. y，t-1 The velocity in the first direction during the next time period. and It is the variance of the normal distribution.

[0063] In one possible implementation of this application, the generation module includes: Define a module to define a single transmission unit, which is a coded block located at a tile position within a time length; The second calculation module is used to calculate the amount of data in the transmission unit according to the following formula: ; Among them, S t For the amount of data, M t Here, m and n represent the width and height of the user's current field of view, respectively, and k is the bitrate. t L represents the number of tile layers extending in each direction. t The duration; The number of tile layers extended in each direction is the quantized value of the dynamic view range, and the time length is the quantized value of the block duration, using k. t and L t The coupling represents the dynamic relationship between the dynamic view range and the block duration.

[0064] In one possible implementation of this application, the objective function in the transmission module is expressed as: ; Where η is the balance coefficient, L stall Let L be the expected value of the stuttering loss. black This represents the expected value of the black border loss.

[0065] In one of the possible implementation methods of this application, the user's head movement state includes: head stability or frequent head movement; The length of time when the head is stable is greater than the length of time when the head moves frequently. The number of tile layers that expand in each direction when the head is stable is less than the number of tile layers that expand in each direction when the head moves frequently.

[0066] In one possible implementation of this application, the generation module includes: Based on the adaptive model, the expected loss expression for when stuttering occurs is obtained: ; Where Φ() is the cumulative distribution function of the standard normal distribution, and μ V With σ V Let B be the log-normal distribution parameter of the bandwidth. t Let S be the buffer size when the t-th video block is downloaded. t For data volume; Based on the adaptation model, the expected loss expression for generating black borders is as follows: ; Among them, S i With P i These represent the black border area and corresponding probability under different motion states, I i This is a scene indicator function.

[0067] The implementation method of this application also includes: The analysis module, by introducing a Bayesian optimization engine based on residual Gaussian processes, analyzes the model L(L) in the objective function. t k t The residual term is used to characterize the difference between the theoretical and actual results, and an optimized objective function is generated.

[0068] In one possible implementation of this application, the parsing module includes: The correlation between block duration and dynamic field of view range can be characterized by the following formula: ; ; ; Where x is the input variable, and the kernel function K(x, x') adopts the radial basis function K corresponding to the continuous variable. L (L, L'), and the Hamming kernel K corresponding to the discrete variables. k (k, k`); During the optimization process, the user's interaction speed and network conditions within the most recent d block periods are continuously monitored to form a dataset, represented as: ; Among them, D t For dataset; The following formula describes the difference between theoretical and actual results: For the current candidate points The posterior distribution of the residual process can be expressed as: ; ; ; Where, μ(x) t ) represents the predicted mean, σ 2 (x) t ) represents the prediction variance, |X| represents the search space size, and δ represents the confidence threshold.

[0069] Figure 4 This diagram illustrates a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0070] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0071] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0072] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0073] The communication interface 1040 is used to connect the communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, radio (shortwave / ultra-shortwave) communication, satellite communication, data link communication, etc.).

[0074] Bus 1050 includes pathways for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0075] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments described in this specification, and need not include all the components shown in the figures.

[0076] The electronic device described above is used to implement the method for transmitting content in the corresponding immersive live stream in any of the foregoing embodiments, and has the beneficial effects of the corresponding method implementation, which will not be elaborated here.

[0077] Based on the same inventive concept, corresponding to any of the above-described embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method for transmitting content in an immersive live stream as described in any of the above embodiments.

[0078] The computer-readable medium in this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0079] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method for transmitting content in an immersive live stream as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0080] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; this manner of description is merely for clarity, and those skilled in the art should consider the specification as a whole. Within the framework of this application, the above embodiments or the technical features of different embodiments can also be appropriately combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0081] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be entirely within the understanding of those skilled in the art). While specific details (e.g., circuits) are set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0082] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0083] The embodiments described herein are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and principles of the embodiments described herein should be included within the protection scope of this application.

Claims

1. A method for transmitting content in an immersive live stream, characterized in that, The method includes: Acquire head movement data of users during the time period preceding their live stream viewing; Based on the motion data, calculate the user's dynamic field of view range in the subsequent time period; An adaptation model is established to show the dynamic relationship between the block duration of a video block and the dynamic field of view. The real-time network bandwidth and the dynamic field of view are input into the adaptation model to generate the dynamic relationship between the dynamic field of view and the block duration. The dynamic view range and the block duration are introduced as optimization variables into the objective function to generate a relative weight between the dynamic view range and the block duration, and the content transmission of the live stream is performed based on the relative weight.

2. The method for transmitting content in an immersive live stream according to claim 1, characterized in that, The step of calculating the user's dynamic field of view range in a subsequent time period based on the motion data includes: The normal distribution of the user's head displacement in subsequent time periods is calculated using the following formula: ； ； in, The displacement is in the first direction. L represents the displacement in the second direction, where the first and second directions are perpendicular to each other. t v is the duration of the subsequent time period. x，t-1 v is the velocity in the first direction during the next time interval. y，t-1 The velocity in the first direction during the next time period. and It is the variance of the normal distribution.

3. The method for transmitting content in an immersive live stream according to claim 1, characterized in that, The step of inputting the real-time network bandwidth and the dynamic field of view into the adaptation model to generate a dynamic relationship between the dynamic field of view and the block duration includes: Define a single transmission unit, which is a coded block located at a tile position within a time length; The data volume of the transmission unit is calculated according to the following formula: ； Among them, S t M represents the amount of data. t Here, m and n represent the width and height of the user's current field of view, respectively, and k is the bitrate. t L represents the number of tile layers extending in each direction. t The duration is the specified time. The number of tile layers extended in each direction is the quantized value of the dynamic field of view, and the time length is the quantized value of the block duration, using k. t and L t The coupling represents the dynamic relationship between the dynamic view range and the block duration.

4. The method for transmitting content in an immersive live stream according to claim 1, characterized in that, The objective function is expressed as: ； Where η is the balance coefficient, L stall Let L be the expected value of the stuttering loss. black This represents the expected value of the black border loss.

5. The method for transmitting content in an immersive live stream according to claim 3, characterized in that, The user's head movement state includes: head steady or frequent head movement; The duration of time when the head is stable is greater than the duration of time when the head moves frequently. The number of tile layers expanding in each direction when the head is stable is less than the number of tile layers expanding in each direction when the head moves frequently.

6. The method for transmitting content in an immersive live stream according to claim 4, characterized in that, Based on the aforementioned adaptation model, the expected loss expression for when stuttering occurs is obtained: ； Where Φ() is the cumulative distribution function of the standard normal distribution, and μ V With σ V Let B be the log-normal distribution parameter of the bandwidth. t S is the buffer size when the t-th video block is downloaded. t The amount of data mentioned above; Based on the aforementioned adaptation model, the expected loss expression for the black border is generated as follows: ； Among them, S i With P i These represent the black border area and corresponding probability under different motion states, I i This is a scene indicator function.

7. The method for transmitting content in an immersive live stream according to claim 3, characterized in that, Before incorporating the dynamic view range and the block duration as optimization variables into the objective function, the following steps are included: By introducing a Bayesian optimization engine based on residual Gaussian processes, the model L(L) in the objective function is analyzed. t k t The residual term is used to characterize the difference between the theoretical and actual results, and an optimized objective function is generated.

8. The method for transmitting content in an immersive live stream according to claim 7, characterized in that, Analyze the model L(L) in the objective function t k t )include: The correlation between the block duration and the dynamic view range is characterized by the following formula: ； ； ； Where x is the input variable, and the kernel function K(x, x') adopts the radial basis function K corresponding to the continuous variable. L (L, L'), and the Hamming kernel K corresponding to the discrete variables. k (k, k`); During the optimization process, the user's interaction speed and network conditions within the most recent d block periods are continuously monitored to form a dataset, represented as: ； Among them, D t For the dataset; The following formula describes the difference between theoretical and actual results: For the current candidate points The posterior distribution of the residual process can be expressed as: ； ； ； Where, μ(x) t ) represents the predicted mean, σ 2 (x) t ) represents the prediction variance, |X| represents the search space size, and δ represents the confidence threshold.

9. A content transmission system for an immersive live stream, characterized in that, The system includes: The acquisition module is used to acquire head movement data of users in the time period preceding the live stream viewing. The first calculation module is used to calculate the user's dynamic field of view range in a subsequent time period based on the motion data. The generation module establishes an adaptation model, inputs the real-time network bandwidth and the dynamic field of view into the adaptation model, and generates a dynamic relationship between the dynamic field of view and the block duration. The transmission module is used to introduce the dynamic view range and the block duration as optimization variables into the objective function, generate the relative weight between the dynamic view range and the block duration, and transmit the live stream according to the relative weight.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method for transmitting content in any one of claims 1 to 8 of the immersive live stream.

Images