Panoramic video on-demand transmission method and device based on view prediction agent
By dynamically adjusting the frequency and selecting appropriate models through a vision prediction agent, the problem of inaccurate user vision prediction in on-demand panoramic video transmission is solved, achieving efficient and accurate video tile transmission, and improving user experience and resource utilization efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-01
- Publication Date
- 2026-04-14
AI Technical Summary
In existing technologies, on-demand transmission methods for panoramic video struggle to accurately predict a user's future field of view, leading to suboptimal resource allocation and a degraded user experience. In particular, the prediction accuracy is insufficient or performance is wasted in scenarios with drastic or gradual changes in field of view.
By using a vision prediction agent-based approach, the vision prediction frequency is dynamically adjusted, and an appropriate prediction model is selected in conjunction with the vision prediction frequency. Based on the degree of change in the user's viewing behavior, the user's vision trajectory data at the next moment is accurately predicted, thereby transmitting video tiles on demand.
It achieves efficient and accurate on-demand transmission in different scenarios, reduces bandwidth consumption of panoramic video, improves user experience, and enhances predictive performance to meet diverse needs.
Smart Images
Figure CN121865006A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video technology, and in particular to a method and apparatus for on-demand transmission of panoramic video based on a vision prediction agent. Background Technology
[0002] With the rapid development of panoramic video technology, its application areas are constantly expanding to virtual reality, video surveillance, remote conferencing, education, and entertainment. Panoramic video can provide a 360-degree immersive experience, significantly enhancing user engagement. The large amount of data that panoramic video requires processing and transmission increases bandwidth demands and also spurs the need for on-demand transmission, which transmits only the user's field of view (FoV), also known as the viewport. Therefore, accurately predicting the user's future field of view is a problem that needs to be solved to achieve on-demand transmission, optimize resource allocation, and ensure a good user experience. Summary of the Invention
[0003] In view of this, the purpose of this application is to provide a method and apparatus for on-demand transmission of panoramic video based on a vision prediction agent.
[0004] To achieve the above objectives, embodiments of this application provide a method for on-demand transmission of panoramic video based on a view prediction agent, including: Receive current field-of-view trajectory data sent by the client; Based on the current field of view trajectory data and the field of view trajectory data at the previous moment, calculate the trajectory change data; Update the field of view prediction frequency based on the trajectory change data; Based on the vision prediction frequency, a target vision trajectory prediction model is selected; The target view trajectory prediction model is used to predict the view trajectory data at the next moment; Based on the view trajectory data at the next moment, the corresponding video tile is searched from the pre-constructed video tile set; The found video tiles are sent to the client.
[0005] Optionally, based on the current field-of-view trajectory data and the field-of-view trajectory data from the previous moment, trajectory change data is calculated, including: Calculate the rotation angle of the gaze point based on the current visual field trajectory data and the visual field trajectory data at the previous moment; Calculate the angular velocity and the change in angular velocity based on the rotation angle of the gaze point.
[0006] Optionally, the visual field trajectory data includes a viewport quaternion and a corresponding timestamp; the fixation point rotation angle is calculated based on the current visual field trajectory data and the visual field trajectory data from the previous moment, using the following method: ; ; in, for t The viewport quaternion of time, for t- The viewport quaternion at time 1. The angle of rotation of the gaze point; calculate t The method for determining the angular velocity at a given time is as follows: ; in, From time t -1 to time t The time interval; The method for calculating the change in angular velocity is as follows: ; in, for t Angular velocity at time -1.
[0007] Optionally, updating the field-of-view prediction frequency based on the trajectory change data includes: The desired prediction frequency is calculated based on the change in angular velocity, the preset default prediction frequency, the maximum prediction frequency, the minimum prediction frequency, and the field of view prediction frequency at the previous moment. The field of view prediction frequency is calculated based on the expected prediction frequency and the field of view prediction frequency at the previous moment.
[0008] Optionally, the desired prediction frequency is calculated based on the change in angular velocity, the preset default prediction frequency, the maximum prediction frequency, the minimum prediction frequency, and the field-of-view prediction frequency of the previous moment. The method is as follows: ; in, F 0 represents the default prediction frequency. F min For the minimum prediction frequency, F max For the maximum prediction frequency, To control the parameters of response sensitivity, The change in angular velocity Angular velocity threshold F(t-1) for t The frequency of visual prediction at time -1 F desired This represents the expected predicted frequency.
[0009] Optionally, the visual field prediction frequency is calculated based on the expected prediction frequency and the visual field prediction frequency at the previous moment, using the following method: ; in, F(t) for t Frequency of visual prediction at any given time This is the smoothing coefficient.
[0010] Optionally, based on the vision prediction frequency, a target vision trajectory prediction model is selected, including: When the field of view prediction frequency is greater than a preset frequency threshold, the first field of view trajectory prediction model is selected. When the field of view prediction frequency is less than or equal to the frequency threshold, the second field of view trajectory prediction model is selected.
[0011] Optionally, selecting a target visual trajectory prediction model based on the visual field prediction frequency includes: The prediction time interval is determined based on the field of view prediction frequency and the length of the video data block; When transmitting video data blocks, a target visual trajectory prediction model is selected according to the predicted time interval.
[0012] Optionally, the method further includes: The panoramic video is preprocessed to generate a complete background video and a set of video tiles consisting of multiple video tiles, each with corresponding coordinate information. Based on the view trajectory data at the next moment, the corresponding video tile is searched from the pre-constructed video tile set, including: Calculate the coordinates of the viewport on the ERP projection plane based on the viewport quaternion at the next moment; Based on this coordinate and the coordinate information of each video tile, find the video tile that includes this coordinate.
[0013] This application also provides a panoramic video on-demand transmission device based on a view prediction agent, including: The receiving module is used to receive the current field-of-view trajectory data sent by the client; The calculation module is used to calculate trajectory change data based on the current field of view trajectory data and the field of view trajectory data at the previous moment; The update module is used to update the field of view prediction frequency based on the trajectory change data; The selection module is used to select a target visual trajectory prediction model based on the visual field prediction frequency. The prediction module is used to predict the vision trajectory data at the next moment using the target vision trajectory prediction model; The search module is used to search for the corresponding video tile from a pre-built set of video tiles based on the view trajectory data at the next moment; The sending module is used to send the found video tiles to the client.
[0014] As can be seen from the above description, the panoramic video on-demand transmission method and apparatus based on a field-of-view prediction intelligent agent provided in this application receives current field-of-view trajectory data sent by the client, calculates trajectory change data based on the current field-of-view trajectory data and the field-of-view trajectory data of the previous moment, updates the field-of-view prediction frequency based on the trajectory change data, selects a target field-of-view trajectory prediction model based on the field-of-view prediction frequency, predicts the field-of-view trajectory data of the next moment using the target field-of-view trajectory prediction model, searches for corresponding video tiles from a pre-built video tile set based on the field-of-view trajectory data of the next moment, and sends the found video tiles to the client. This application can efficiently and accurately transmit the required video tiles to the client. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic diagram of the method flow of an embodiment of this application; Figure 2 This is a schematic diagram of the system architecture of an embodiment of this application; Figure 3 This is a schematic diagram of fixed frequency prediction for some embodiments; Figure 4 This is a schematic diagram of dynamic frequency prediction in an embodiment of this application; Figure 5 This is a block diagram of the device structure according to an embodiment of this application; Figure 6 This is a block diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.
[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0019] In related technologies, the unpredictability of user viewing behavior and the stringent performance requirements of real-time systems present challenges for on-demand transmission. Most methods use fixed time intervals and single prediction models to predict future views, which are ill-suited to dynamic scenarios. For example... Figure 3 As shown, each video data block uses a fixed time interval to predict the video tiles that need to be transmitted for the next video data block. Predict For predicting timing, fixed-time predictions are insufficient in scenarios with drastic changes in field of view (such as sports competitions), while in scenarios with gradual changes in field of view (such as beach vacations), fixed-time predictions result in wasted performance due to frequent predictions. Moreover, relying on a single prediction model cannot effectively adapt to the diverse needs of different scenarios, thus limiting overall prediction performance.
[0020] In view of this, embodiments of this application provide a panoramic video on-demand transmission method based on a field-of-view prediction agent. The method dynamically adjusts the field-of-view prediction frequency based on the degree of change in user viewing behavior and selects a suitable field-of-view prediction model in combination with the field-of-view prediction frequency. This method can adapt to various scenarios, reduce prediction overhead while ensuring prediction accuracy, and achieve accurate and efficient on-demand transmission.
[0021] The technical solution of this application will be further described in detail below through specific embodiments.
[0022] like Figure 1 , 2 As shown in the figure, this application provides a method for on-demand transmission of panoramic video based on a view prediction agent, including: S101: Receive current field of view trajectory data sent by the client; The on-demand transmission method provided in this embodiment is based on a server-client architecture. The server provides panoramic video, and the user watches the video on the client. Since transmitting the complete panoramic video by the server consumes too much bandwidth resources, and the user's viewing behavior is constantly changing, it is advisable to consider transmitting only the field of view that the user is about to view, i.e., on-demand transmission rather than full transmission.
[0023] To achieve on-demand transmission, it's necessary to know the user's current viewing behavior in order to predict subsequent viewing areas. The client captures changes in the playback device's viewport orientation through a viewport data capture interface. For example, it calls an OpenVR-compliant API to obtain viewport quaternions, combines these quaternions with the corresponding timestamps to construct the current viewport trajectory data, and sends this data to the server. The spatial quaternion is in the form of... ,correspond The real part of the number, Quaternions represent unit vectors in space. This embodiment will not provide a detailed explanation of the specific meaning and principles of quaternions.
[0024] In some methods, the client encapsulates the current field-of-view trajectory data into a lightweight data packet based on the UDP protocol and sends the data packet to the server to reduce transmission latency and ensure real-time data transmission.
[0025] S102: Calculate trajectory change data based on the current field of view trajectory data and the field of view trajectory data at the previous moment; In this embodiment, the server receives the current field-of-view trajectory data sent by the client, and calculates trajectory change data based on the current field-of-view trajectory data and the field-of-view trajectory data at the previous moment. The method includes: Calculate the rotation angle of the fixation point based on the current visual field trajectory data and the visual field trajectory data at the previous moment; Calculate the angular velocity and the change in angular velocity based on the rotation angle of the gaze point.
[0026] The current visual field trajectory data includes the viewport quaternion at the current moment and the timestamp of that moment, while the previous visual field trajectory data includes the viewport quaternion at the previous moment and the timestamp of that moment. Based on the current and previous visual field trajectory data, the fixation point rotation angle is calculated using the following method: (1) (2) In the formula, for t The viewport quaternion of time, for t- The viewport quaternion at time 1. The angle of rotation of the gaze point. It is the dot product of quaternions.
[0027] Calculate based on the fixation point rotation angle and time interval. t The angular velocity at time t is obtained by: (3) In the formula, From time t -1 to time t The time interval.
[0028] according to t angular velocity at time and t- The angular velocity at time 1, and the change in angular velocity, are calculated as follows: (4) In the formula, for t Angular velocity at time -1.
[0029] S103: Update the field of view prediction frequency based on trajectory change data; In this embodiment, based on the current visual field trajectory data and the visual field trajectory data from the previous moment, trajectory change data is determined, that is, after determining the change in user viewing behavior, the visual field prediction frequency is updated. The method includes: The desired prediction frequency is calculated based on the change in angular velocity, the preset default prediction frequency, the maximum prediction frequency, the minimum prediction frequency, and the field-of-view prediction frequency of the previous moment. The visual field prediction frequency is calculated based on the expected prediction frequency and the visual field prediction frequency at the previous moment.
[0030] The method for calculating the expected prediction frequency is as follows: (5) In the formula, F 0 represents the default prediction frequency. F min For the minimum prediction frequency, F max For the maximum prediction frequency, To control the parameters of response sensitivity, Angular velocity threshold F(t-1) for t The frequency of visual prediction at time -1 F desired This represents the expected predicted frequency.
[0031] According to formula (5), when When the change in angular velocity exceeds the angle threshold, it is determined that the user's viewing behavior has changed drastically; when When the user's viewing behavior changes gradually, it is determined that the change is moderate; when At that time, it is determined that there is no change in viewing behavior. Therefore, the expected prediction frequency can be determined based on the degree of change in the user's viewing behavior. Then, the current visual field prediction frequency can be calculated based on the expected prediction frequency and the visual field prediction frequency of the previous moment. The method is as follows: (6) In the formula, F(t) for t Frequency of visual prediction at any given time This is the smoothing coefficient.
[0032] S104: Select the target vision trajectory prediction model based on the vision prediction frequency; In this embodiment, after determining the field of view prediction frequency, a suitable target field of view trajectory prediction model is dynamically selected based on the field of view prediction frequency. The method includes: When the field of view prediction frequency is greater than the preset frequency threshold, the first field of view trajectory prediction model is selected. When the field of view prediction frequency is less than or equal to the frequency threshold, the second field of view trajectory prediction model is selected.
[0033] In this embodiment, the prediction model can be adaptively adjusted based on the field of view prediction frequency. When the field of view prediction frequency is greater than a frequency threshold... F th ,Right now When the frequency of field of view prediction is less than or equal to the frequency threshold, the first-view trajectory prediction model is selected. This model is a lightweight prediction model with low computational complexity and fast response time, suitable for frequent predictions. F th ,Right now When choosing a second field of view trajectory prediction model, it is recommended to select the model with higher accuracy but higher computational cost. This model is suitable for low-frequency, high-precision prediction.
[0034] In some embodiments, a target visual trajectory prediction model is selected based on the visual field prediction frequency, including: The prediction time interval is determined based on the field of view prediction frequency and the length of the video data block; When transmitting video data blocks, a target view trajectory prediction model is selected according to the prediction time interval.
[0035] like Figure 4 As shown, in this embodiment, the visual field prediction frequency is determined by the changes in the user's visual field trajectory. That is, after dynamically adjusting the visual field prediction frequency according to the degree of change in the user's viewing behavior, the prediction time interval is calculated based on the size of the video data block and the adjusted visual field prediction frequency, as expressed as: (7) in, T ( t() represents the prediction time interval. L This refers to the size of the video data block.
[0036] At the start of transmission of each video data chunk, when the prediction time interval is reached, a target view trajectory prediction model is selected, and a view prediction is performed using the target view trajectory prediction model. That is, according to the calculated prediction time interval, when the prediction opportunity is reached... Predict i At that time, the selected target vision trajectory prediction model is used. i For field-of-view prediction, a video data block may include multiple video tiles corresponding to field-of-view predictions. This allows for dynamic adjustment of the field-of-view prediction frequency and timing based on changes in user viewing behavior, and selection of the optimal field-of-view prediction model to achieve adaptive video transmission in different scenarios.
[0037] S105: Predict the vision trajectory data at the next moment using the target vision trajectory prediction model; In this embodiment, the selected target visual field trajectory prediction model predicts the visual field trajectory data at the next moment based on the current visual field trajectory data. That is, the current visual field trajectory data is input into the target visual field trajectory prediction model, and the model outputs the predicted visual field trajectory data at the next moment.
[0038] S106: Based on the view trajectory data at the next moment, find the corresponding video tile from the pre-built video tile set; S107: Send the found video tiles to the client.
[0039] In this embodiment, the server preprocesses the acquired panoramic video to generate a complete background video and a set of video tiles consisting of multiple video tiles, each with its own identifier and coordinate information. The complete background video includes a low-quality full-frame image; compared to the original panoramic video, it has less data and only needs to be sent to the client once, minimizing network resource consumption. Each video tile belongs to the foreground layer and has higher quality, allowing for on-demand transmission based on user viewing behavior, improving resource utilization efficiency. In some methods, the server segments the panoramic video temporally, generating multiple video chunks. For each chunk, a multi-level quality hierarchy is constructed using encoding standards such as H.264 / HEVC. Frames within each chunk are cropped into multiple video tiles. The identifier of each video tile is determined using spatial indexing or geometric calculation methods, generating a low-quality background video containing the full-frame image and multiple high-quality video tiles for that video chunk. This embodiment does not provide a detailed description of the panoramic video preprocessing methods and procedures.
[0040] In some embodiments, after determining the view trajectory data for the next moment using a target view trajectory prediction model, the corresponding video tile is searched from the video tile set. The method includes: Calculate the coordinates of the viewport on the ERP projection plane based on the viewport quaternion at the next moment; Based on this coordinate and the coordinate information of each video tile, find the video tile that includes this coordinate.
[0041] In this embodiment, after predicting the viewport quaternion for the next moment, the coordinates of the viewport on the ERP projection plane are calculated. The video tile containing the coordinates is searched in the video tile set to determine the video tile within the predicted viewport range. The found video tile is then sent to the client. The client reconstructs the panoramic video based on the video tile and the low-quality background video containing the complete image that has been received locally. This effectively reduces bandwidth consumption while ensuring the video quality within the field of view.
[0042] In some methods, the client's video decoder decodes the received background video and video tiles. Based on the coordinate information of the video tiles, it precisely overlays or stitches the video tiles onto the corresponding areas of the low-quality background video, thereby generating a complete video frame with high definition in the center of the user's viewport and maintaining basic quality in the surrounding areas. The rendering module then renders the synthesized complete video frame and displays it on the client.
[0043] like Figure 2 As shown, the execution subject of the method in this embodiment is the view prediction agent on the server side. The view prediction agent uses the user's current viewing behavior based on the user's current view trajectory data, updates the view prediction frequency, selects a suitable target view trajectory prediction model, uses the model to predict the view that the user may view in the next moment, finds the video tile corresponding to the view and sends it to the client, so as to realize the on-demand transmission perceived by the user.
[0044] The panoramic video on-demand transmission method based on a view prediction agent provided in this application involves the client sending its current view trajectory data to the server. The server calculates the trajectory change data of the user's viewing behavior based on the current and previous view trajectory data. Based on this trajectory change data, the server dynamically adjusts the view prediction frequency. According to the adjusted frequency, a target view trajectory prediction model is selected, and the model is used to predict the view trajectory data for the next moment. Based on the next moment's view trajectory data, the required video tile is located from the video tile set and sent to the client. This allows the client to reconstruct the panoramic video from the background video and the video tile. This application can efficiently and accurately transmit the required video tiles to the client, reducing bandwidth consumption, system overhead, and ensuring a good user viewing experience, meeting the application needs of various scenarios such as panoramic video-on-demand transmission systems.
[0045] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.
[0046] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0047] like Figure 5 As shown, this application provides a panoramic video on-demand transmission device based on a view prediction agent, comprising: The receiving module is used to receive the current field-of-view trajectory data sent by the client; The calculation module is used to calculate trajectory change data based on the current view trajectory data and the view trajectory data of the previous moment; The update module is used to update the field of view prediction frequency based on trajectory change data; The selection module is used to select the target vision trajectory prediction model based on the vision prediction frequency. The prediction module is used to predict the vision trajectory data at the next moment using the target vision trajectory prediction model. The search module is used to find the corresponding video tile from a pre-built set of video tiles based on the view trajectory data at the next moment; The sending module is used to send the found video tiles to the client.
[0048] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.
[0049] The apparatus described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0050] Figure 6This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.
[0051] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0052] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.
[0053] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.
[0054] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0055] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.
[0056] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.
[0057] The electronic devices described above are used to implement the corresponding methods in the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0058] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0059] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.
[0060] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.
[0061] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.
[0062] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this disclosure.
Claims
1. A method for on-demand transmission of panoramic video based on a view-prediction agent, characterized in that, include: Receive current field-of-view trajectory data sent by the client; Based on the current field of view trajectory data and the field of view trajectory data at the previous moment, calculate the trajectory change data; Update the field of view prediction frequency based on the trajectory change data; Based on the vision prediction frequency, a target vision trajectory prediction model is selected; The target view trajectory prediction model is used to predict the view trajectory data at the next moment; Based on the view trajectory data at the next moment, the corresponding video tile is searched from the pre-constructed video tile set; The found video tiles are sent to the client.
2. The method according to claim 1, characterized in that, Based on the current field-of-view trajectory data and the previous field-of-view trajectory data, trajectory change data is calculated, including: Calculate the rotation angle of the gaze point based on the current visual field trajectory data and the visual field trajectory data at the previous moment; Calculate the angular velocity and the change in angular velocity based on the rotation angle of the gaze point.
3. The method according to claim 2, characterized in that, The visual field trajectory data includes viewport quaternions and corresponding timestamps; the fixation point rotation angle is calculated based on the current visual field trajectory data and the visual field trajectory data from the previous moment, using the following method: ； ； in, for t The viewport quaternion of time, for t- The viewport quaternion at time 1. The angle of rotation of the gaze point; calculate t The method for determining the angular velocity at a given time is as follows: ； in, From time t -1 to time t The time interval; The method for calculating the change in angular velocity is as follows: ； in, for t Angular velocity at time -1.
4. The method according to claim 2, characterized in that, Based on the trajectory change data, update the field of view prediction frequency, including: The desired prediction frequency is calculated based on the change in angular velocity, the preset default prediction frequency, the maximum prediction frequency, the minimum prediction frequency, and the field of view prediction frequency at the previous moment. The field of view prediction frequency is calculated based on the expected prediction frequency and the field of view prediction frequency at the previous moment.
5. The method according to claim 4, characterized in that, The desired prediction frequency is calculated based on the change in angular velocity, the preset default prediction frequency, the maximum prediction frequency, the minimum prediction frequency, and the field-of-view prediction frequency of the previous moment. The method is as follows: ； in, F 0 represents the default prediction frequency. F min For the minimum prediction frequency, F max For the maximum prediction frequency, To control the parameters of response sensitivity, The change in angular velocity Angular velocity threshold F(t-1) for t The frequency of visual prediction at time -1 F desired This represents the expected predicted frequency.
6. The method according to claim 5, characterized in that, The visual field prediction frequency is calculated based on the expected prediction frequency and the visual field prediction frequency at the previous moment, using the following method: ； in, F(t) for t Frequency of visual prediction at any given time This is the smoothing coefficient.
7. The method according to claim 1, characterized in that, Based on the stated field of view prediction frequency, a target field of view trajectory prediction model is selected, including: When the field of view prediction frequency is greater than a preset frequency threshold, the first field of view trajectory prediction model is selected. When the field of view prediction frequency is less than or equal to the frequency threshold, the second field of view trajectory prediction model is selected.
8. The method according to claim 1, characterized in that, The step of selecting a target visual trajectory prediction model based on the visual field prediction frequency includes: The prediction time interval is determined based on the field of view prediction frequency and the length of the video data block; When transmitting video data blocks, a target visual trajectory prediction model is selected according to the predicted time interval.
9. The method according to claim 1, characterized in that, Also includes: The panoramic video is preprocessed to generate a complete background video and a set of video tiles consisting of multiple video tiles, each with corresponding coordinate information. Based on the view trajectory data at the next moment, the corresponding video tile is searched from the pre-constructed video tile set, including: Calculate the coordinates of the viewport on the ERP projection plane based on the viewport quaternion at the next moment; Based on this coordinate and the coordinate information of each video tile, find the video tile that includes this coordinate.
10. A panoramic video on-demand transmission device based on a view prediction intelligent agent, characterized in that, include: The receiving module is used to receive the current field-of-view trajectory data sent by the client; The calculation module is used to calculate trajectory change data based on the current field of view trajectory data and the field of view trajectory data at the previous moment; The update module is used to update the field of view prediction frequency based on the trajectory change data; The selection module is used to select a target visual trajectory prediction model based on the visual field prediction frequency. The prediction module is used to predict the vision trajectory data at the next moment using the target vision trajectory prediction model; The search module is used to search for the corresponding video tile from a pre-built set of video tiles based on the view trajectory data at the next moment; The sending module is used to send the found video tiles to the client.