Image display method and display system

Abstract

An image display method is provided. The image display method comprises the following steps: tracking eye movement of a user through terminal equipment, and obtaining eye tracking data; transmitting the eye tracking data from the terminal device to a content server; determining, by the content server, a dynamic high resolution region of an original image based on the eye tracking data; rendering, by the content server, a first image having a first resolution; rendering, by the content server, a second image having a second resolution based on the dynamic high resolution region; encoding, by the content server, the second image having the second resolution, the first image having the first resolution, and parameter data in a data stream; and transmitting the data stream from the content server to the terminal device for image display. The second resolution is greater than the first resolution.

Description

Technical Field

[0001] The present invention relates to display technology, and in particular to an image display method and a display system. Background Art

[0002] Real-time communication transmission (RTC) refers to the use of real-time communication technology for data transmission. Compared with traditional video network transmission technology, real-time communication transmission provides lower latency and higher interactivity. RTC transmission optimizes network transmission speed, latency and bandwidth in real time, thereby improving real-time performance, stability and audio-visual quality. As a result, RTC live streaming transmission is more suitable for interactive live broadcasting, online education, telehealth, online conferencing and other scenarios, making it highly applicable in various fields.

[0003] Eye tracking technology is a biometric technology that uses the recognition of eye movement patterns to achieve various applications, such as identity authentication, emotion analysis, and gaze tracking. At the same time, network transmission technology is an essential component of the development of modern information technology. However, there are currently problems of low transmission efficiency and poor transmission stability. In order to solve these problems, the present application proposes a partition-based transmission system with dynamic expansion for load balancing. Summary of the invention

[0004] In one aspect, the present disclosure provides an image display method, comprising: tracking eye movements of a user through a terminal device and obtaining eye tracking data; transmitting the eye tracking data from the terminal device to a content server; determining a dynamic high-resolution area of ​​an original image through the content server based on the eye tracking data; rendering a first image with a first resolution through the content server, wherein the first image is a full low-resolution rendering of the original image; rendering a second image with a second resolution through the content server based on the dynamic high-resolution area, wherein the second image is a high-resolution rendering of an area of ​​the original image; encoding the second image with the second resolution, the first image with the first resolution, and parameter data in a data stream through the content server; and transmitting the data stream from the content server to the terminal device for image display; wherein the second resolution is greater than the first resolution.

[0005] Optionally, the method also includes: obtaining updated real-time eye tracking coordinates through the terminal device; cropping the second image having the second resolution based on the updated real-time eye tracking coordinates detected by the terminal device to generate an updated image through the terminal device; and combining the first image having the first resolution with the updated image through the terminal device to generate a composite image.

[0006] Optionally, the data stream includes a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding the parameter data.

[0007] Optionally, transmitting the data stream from the content server to the terminal device includes: transmitting the first data stream at a first bit rate; transmitting the second data stream at a second bit rate; and transmitting the third data stream at a third bit rate; wherein the third bit rate is higher than the first bit rate; and the first bit rate is higher than the second bit rate.

[0008] Optionally, the method also includes: predefining a preset high-resolution area through the content server; receiving, through the content server, the eye tracking data including the eye tracking coordinates detected by the terminal device at a first point in time; and calculating, through the content server, predicted eye tracking coordinates at a second point in time.

[0009] Optionally, the dynamic high-resolution area is determined based on: the width of a preset high-resolution area; the length of the preset high-resolution area; the instantaneous eye movement vector velocity of the eye movement at the first time point; the instantaneous eye movement acceleration of the eye movement at the first time point; and the difference between the second time point and the first time point.

[0010] Optionally, the difference between the second time point and the first time point is substantially the same as the round trip time between the content server and the terminal device.

[0011] Optionally, the predicted eye tracking coordinates are the center point of the dynamic high-resolution area at the second time point; and based on the changes of the instantaneous eye movement vector velocity and the instantaneous eye movement acceleration over time, the center point, width and length of the dynamic high-resolution area dynamically change over time.

[0012] Optionally, the method further includes frame synchronizing, by the terminal device, the first data stream encoding the second image having the second resolution, the second data stream encoding the first image having the first resolution, and the third data stream encoding the parameter data.

[0013] Optionally, the method also includes: extracting a first presentation timestamp from a decoded first data stream through the terminal device; extracting a second presentation timestamp from a decoded second data stream through the terminal device; extracting a third presentation timestamp from a decoded third data stream through the terminal device; and determining whether the first presentation timestamp, the second presentation timestamp and the third presentation timestamp are the same.

[0014] Optionally, the method also includes: when it is determined that the first presentation timestamp is earlier than the second presentation timestamp, discarding the current frame of the decoded first data stream and waiting for the next frame of the decoded first data stream; or when it is determined that the first presentation timestamp is later than the second presentation timestamp, discarding the current frame of the decoded second data stream and waiting for the next frame of the decoded second data stream.

[0015] Optionally, the method also includes: when it is determined that the first presentation timestamp is earlier than the third presentation timestamp, discarding the current frame of the decoded first data stream and the current frame of the decoded second data stream, and waiting for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the first presentation timestamp is later than the third presentation timestamp, discarding the current frame of the decoded third data stream and waiting for the next frame of the decoded third data stream.

[0016] Optionally, the method also includes: when it is determined that the second presentation timestamp is earlier than the third presentation timestamp, discarding the current frame of the decoded first data stream and the current frame of the decoded second data stream, and waiting for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the second presentation timestamp is later than the third presentation timestamp, discarding the current frame of the decoded third data stream and waiting for the next frame of the decoded third data stream.

[0017] Optionally, the method further comprises: generating, by the content server, the parameter data based on the dynamic high-resolution area.

[0018] Optionally, a third data stream encoding the parameter data is transmitted from the content server to the terminal device, including: transmitting the coordinates of the center point of the dynamic high-resolution area; the width of the preset high-resolution area; the length of the preset high-resolution area; the width of the dynamic high-resolution area; the length of the dynamic high-resolution area; and a presentation timestamp.

[0019] Optionally, the parameter data is transmitted from the content server to the terminal device using lossless compression.

[0020] Optionally, rendering the second image with the second resolution based on the dynamic high-resolution area includes: rendering the second image with the second resolution based at least in part on the coordinates of a center point of the dynamic high-resolution area, a width of the dynamic high-resolution area, and a length of the dynamic high-resolution area.

[0021] Optionally, rendering the first image having the first resolution includes: performing horizontal compression and performing vertical compression.

[0022] On the other hand, the present disclosure provides a display system, including a content server and a terminal device; wherein the terminal device is configured to: track eye movements of a user; obtain eye tracking data; and transmit the eye tracking data to the content server; wherein the content server is configured to: determine a dynamic high-resolution area of ​​an original image based on the eye tracking data; render a first image with a first resolution, wherein the first image is a full low-resolution rendering of the original image; render a second image with a second resolution based on the dynamic high-resolution area, wherein the second image is a high-resolution rendering of an area of ​​the original image; encode the second image with the second resolution, the first image with the first resolution, and parameter data in a data stream; and transmit the data stream from the content server to the terminal device for image display; wherein the second resolution is greater than the first resolution.

[0023] On the other hand, the present disclosure provides a computer program product, including a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by one or more first processors to cause the one or more first processors to perform the following operations: tracking eye movements of a user and obtaining eye tracking data; and transmitting the eye tracking data from a terminal device to a content server; the computer-readable instructions being executable by one or more second processors to cause the one or more second processors to perform the following operations: determining a dynamic high-resolution area of ​​an original image based on the eye tracking data; rendering a first image having a first resolution, wherein the first image is a full low-resolution rendering of the original image; rendering a second image having a second resolution based on the dynamic high-resolution area, wherein the second image is a high-resolution rendering of an area of ​​the original image; encoding the second image having the second resolution, the first image having the first resolution, and parameter data in a data stream; and transmitting the data stream from the content server to a terminal device for image display; wherein the second resolution is greater than the first resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] According to various disclosed embodiments, the following drawings are examples only for illustration purposes and are not intended to limit the scope of the present invention.

[0025] Figure 1 is a flowchart illustrating an image display method according to some embodiments of the present disclosure.

[0026] Figure 2 is a schematic diagram showing a display system for implementing an image display method according to some embodiments of the present disclosure.

[0027] Figure 3 is a flowchart illustrating a process of frame synchronization of a first data stream encoding a second image having a second resolution, a second data stream encoding the first image having a first resolution, and a third data stream encoding parameter data according to some embodiments of the present disclosure.

[0028] Figure 4 is a flowchart illustrating a process of frame synchronization of a first data stream encoding a second image having a second resolution, a second data stream encoding the first image having a first resolution, and a third data stream encoding parameter data according to some embodiments of the present disclosure.

[0029] Figure 5 The process of rendering a second image with a second resolution and a first image with a first resolution in some embodiments according to the present disclosure is shown.

[0030] Figure 6 A process of cropping a second image having a second resolution based on updated real-time eye tracking coordinates used to generate an updated image in accordance with some embodiments of the present disclosure is shown.

[0031] Figure 7 is a schematic diagram showing a terminal device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0032] The present disclosure will now be described in more detail with reference to the following examples. It should be noted that the following description of some of the embodiments presented herein is for illustration and description purposes only. It is not intended to be exhaustive or limited to the precise form disclosed.

[0033] Users can use head mounted devices (HMDs) to view virtual reality content, for example, content stored on cloud servers. Compared with storing a large amount of virtual reality content resources locally, storing content on the cloud can reduce the performance requirements on the client side. However, the transmission of VR content over the Internet faces challenges such as bandwidth limitations and latency.

[0034] Therefore, the present disclosure provides, in particular, an image display method and a display system, which substantially eliminate one or more problems caused by the limitations and disadvantages of the prior art. In one aspect, the present disclosure provides an image display method. In some embodiments, the image display method includes: tracking the user's eye movement through a terminal device and obtaining eye tracking data; transmitting the eye tracking data from the terminal device to a content server; determining the dynamic high-resolution area of ​​the original image through the content server based on the eye tracking data; rendering a first image with a first resolution through the content server, wherein the first image is a full low-resolution rendering of the original image; rendering a second image with a second resolution through the content server based on the dynamic high-resolution area, wherein the second image is a high-resolution rendering of an area of ​​the original image; encoding the second image with the second resolution, the first image with the first resolution and parameter data in a data stream through the content server; and transmitting the data stream from the content server to the terminal device for image display. The second resolution is greater than the first resolution. Optionally, the first image is a full low-resolution image; the second image is a high-resolution image; and the resolution of the high-resolution image is greater than the resolution of the full low-resolution image.

[0035] Figure 1 is a flowchart illustrating an image display method according to some embodiments of the present disclosure. Figure 1 In some embodiments, the image display method includes: tracking eye movement of a user through a terminal device and obtaining eye tracking data; transmitting the eye tracking data from the terminal device to a content server; determining a dynamic high-resolution area through the content server based on the eye tracking data; rendering a first image with a first resolution through the content server; rendering a second image with a second resolution based on the dynamic high-resolution area through the content server; encoding the second image with the second resolution, the first image with the first resolution, and parameter data in three separate data streams, respectively; transmitting the three separate data streams from the content server to the terminal device; and cropping the second image with the second resolution through the terminal device based on updated real-time eye tracking coordinates detected by the terminal device, thereby generating an updated image. Optionally, the second resolution is greater than the first resolution.

[0036] Figure 2 is a schematic diagram showing a display system for implementing an image display method according to some embodiments of the present disclosure. Figure 1 and Figure 2In some embodiments, a display system implementing the image display method includes a content server CS and a terminal device TD. The content server CS and the terminal device are configured to communicate via a network NT (eg, the Internet).

[0037] The content server CS is a key component in a distributed system, which is responsible for storing content and providing it to clients or end users. It acts as a central repository for data (including files, documents, media, or any other form of information that needs to be accessed or distributed within the system). The content server CS manages the storage, retrieval, and delivery of content efficiently and securely.

[0038] Terminal device TD refers to the hardware or interface through which the user interacts with the system or network. It serves as a gateway for users to access and utilize the functions and services provided by the system. Terminal devices can take various forms. Examples of terminal devices include personal computers; mobile devices such as smart phones and tablets; the Internet of Things, such as smart TVs, smart speakers, and home automation systems; and virtual reality / augmented reality devices, such as head-mounted displays (HMDs) and other immersive devices. Terminal devices are usually installed with software applications or web browsers that enable users to access system services, browse content, perform transactions, or communicate with other users. These devices communicate with servers or back-end systems through various protocols (such as HTTP, TCP / IP, or proprietary communication protocols).

[0039] refer to Figure 2 In some embodiments, the content server CS includes an image generation module IGM, an image processing module IPM, a tracking prediction module TPM, a first encoding and decoding module EDM1 and a first network transmission module NTM1. The image processing module IPM is configured to generate an original image (eg, a video stream) and send the original image to the image processing module IPM.

[0040] In some embodiments, the first network transmission module NTM1 is configured to receive eye tracking data from the terminal device TD, and is configured to send the eye tracking data to the first codec module EDM1. The first codec module EDM1 is configured to decode the eye tracking data and transmit the eye tracking data to the image processing module IPM and the tracking prediction module TPM. The first network transmission module NTM1 is configured to determine the round-trip time of the current time. As used herein, the term "round-trip time" refers to the total time taken for a data packet to travel from a source to a destination and back again. It is a measure of delay or latency in a network. In order to calculate the round-trip time, a device (such as a computer or server) sends a data packet to a destination, and the destination confirms the receipt of the packet by sending the packet back to the source. The round-trip time is then determined by measuring the time taken for the packet to travel from the source to the destination and back again.

[0041] In some embodiments, the image processing module IPM is configured to render a first image with a first resolution. The first image with a first resolution refers to an image that has been rendered in its entirety but has a low level of detail or resolution. In one example, the image processing module IPM generates a complete image with simplified details. The first image with a first resolution is used as a basis for rendering subsequent high-resolution areas thereon. The inventors of the present disclosure have found that by making the complete image at a lower resolution, the system can focus the computational workload on generating high-resolution portions only for specific areas that require more details. This approach helps optimize network transmission and resource allocation while still providing users with an acceptable level of visual representation.

[0042] In some embodiments, upon receiving eye tracking data, the tracking prediction module TPM is configured to determine a dynamic high-resolution area based on the eye tracking data. A dynamic high-resolution area refers to a specific area within an image that is presented at a higher level of detail or resolution than the rest of the image. In some embodiments, the area is calculated based on eye tracking data and dynamically adjusted as the user's gaze or focus changes. The purpose of the dynamic high-resolution area is to efficiently allocate computing resources and network bandwidth. Instead of rendering the entire image at high resolution (which would require significant computing power and transmission capacity), the system focuses on rendering only specific areas corresponding to where the user is watching. By utilizing eye tracking data, the system can determine the user's point of interest or gaze point and dynamically adjust the high-resolution rendering accordingly. This ensures that the area that the user is actively watching appears clear and detailed, while the remaining area can be rendered at a lower resolution to save resources. The inventors of the present disclosure have found that by prioritizing the rendering of important areas with high detail, this dynamic method allows for more efficient allocation of computing resources, reduced network bandwidth requirements, and provides an enhanced visual experience for the user.

[0043] In some embodiments, the tracking prediction module TPM is also configured to generate parameter data based on the dynamic high-resolution area. In some embodiments, the parameter data includes specific configuration settings that are applied only to the dynamic high-resolution area. These settings can control various aspects, such as image processing algorithms, rendering techniques, or quality parameters that are specifically customized for the dynamic high-resolution area. In some embodiments, the parameter data also includes data that provides additional descriptive information about the dynamic high-resolution area. The parameter data may include attributes such as the size or dimensions of the dynamic high-resolution area, its position in the image, or any other relevant characteristics. In some embodiments, the parameter data also includes timestamp information. The timestamp information provides timing or time data associated with the dynamic high-resolution area. It helps to synchronize and coordinate actions or processes related to high-resolution rendering and ensure proper alignment with other components or systems. By including timestamp information in the parameter data, the system can accurately track and manage the timing aspects of the dynamic high-resolution area. This information can be used for synchronization purposes, data coordination, or to ensure that the high-resolution rendering is properly aligned with other visual or interactive elements in the system.

[0044] In some embodiments, the image processing module IPM is configured to render a second image with a second resolution based on a dynamic high-resolution area. The image processing module IPM uses various image processing techniques, algorithms and computing resources to render a second image with a second resolution. It focuses its rendering work on the dynamic high-resolution area, thereby ensuring that the specific area appears with enhanced details, clarity and visual fidelity. The inventors of the present disclosure have found that by selectively rendering a second image with a second resolution for a dynamic high-resolution area, the image processing module IPM optimizes computing resources and reduces processing overhead. Instead of rendering the entire image at a high resolution (which may be computationally expensive), the image processing module IPM prioritizes areas that require more details based on the dynamic high-resolution area. The second image with a second resolution generated by the IPM can be further used for subsequent processing, such as encoding, compression or transmission. It forms part of the entire image representation that is ultimately transmitted to the display end for viewing by the user.

[0045] In some embodiments, the first codec module EDM1 is configured to encode the second image with the second resolution, the first image with the first resolution, and the parameter data in three separate data streams, respectively, the three separate data streams comprising a first data stream encoding the second image with the second resolution, a second data stream encoding the first image with the first resolution, and a third data stream encoding the parameter data. Optionally, the first data stream, the second data stream, and the third data stream have different bit rates.

[0046] In some embodiments, the first network transmission module NTM1 is configured to send three separate data streams to the terminal device TD, for example, to send three separate data streams to the second network transmission module NTM2 in the terminal device TD. Optionally, the first network transmission module NTM1 is configured to send the three separate data streams to the second network transmission module NTM2 via Web Real-Time Communication (WebRTC).

[0047] The inventors of the present disclosure have found that there is a delay during the transmission of eye tracking data from the terminal device TD to the content server CS, and there is a delay during the transmission of the second image with a second resolution from the content server CS to the terminal device TD. In public network transmission, network delays may vary due to user location, different telecommunications broadband providers, and varying network access times (such as during low or peak network usage periods). Therefore, network delays may be relatively high and sometimes exceed 100 milliseconds, during which significant changes in eye position may occur, which introduces significant errors.

[0048] In the present disclosure, eye tracking data is sent to a content server to predict eye coordinates. In addition, the present disclosure determines a dynamic high-resolution region to ensure that a dynamically expanded and enlarged high-resolution region will cover changes in the high-resolution gaze region caused by eye movement during transmission, thereby preventing any overflow of the high-resolution gaze region. In one example, the dynamic high-resolution region can be a dynamic high-resolution expansion region.

[0049] Reference Figure 2 In some embodiments, the terminal device TD includes a second network transmission module NTM2, a second codec module EDM2, an eye tracking module ETM and a display module DM. The second network transmission module NTM2 is configured to receive three separate data streams from the first network transmission module NTM1. The second codec module EDM2 is configured to decode the three separate data streams. Optionally, the second codec module EDM2 is configured to decode each of the three separate data streams respectively to extract the corresponding data. Once the second codec module EDM2 has successfully received and extracted data from the three streams, further processing can be performed. This may include combining high-resolution and low-resolution data, applying any necessary image processing based on the information stream, or any other operation required to generate a final image for display or further use.

[0050] In some embodiments, the display module DM is configured to crop the second image having the second resolution based on updated real-time eye tracking coordinates detected by the eye tracking module ETM, thereby generating an updated image.The updated real-time eye tracking coordinates help determine specific areas of interest.

[0051] In some embodiments, the display module DM is configured to combine the updated image with the first image having the first resolution. Optionally, the combining process involves combining the updated image with a corresponding portion of the first image having the first resolution to generate a composite image.

[0052] In some embodiments, the eye tracking module ETM is configured to track the user's eye movement and obtain eye tracking data. Optionally, the second encoding and decoding module EDM2 is configured to encode the eye tracking data and send the eye tracking data to the second network transmission module NTM2. Optionally, the second network transmission module NTM2 is configured to transmit the eye tracking data to the first network transmission module NTM1.

[0053] In some embodiments, the eye tracking module ETM is further configured to obtain updated real-time eye tracking coordinates, and is configured to send the updated real-time eye tracking coordinates to the second codec module EDM2. Optionally, the second codec module EDM2 is configured to transmit the updated real-time eye tracking coordinates to the display module DM. In an alternative embodiment, the eye tracking module ETM is configured to directly transmit the updated real-time eye tracking coordinates to the display module DM.

[0054] In some embodiments, the eye tracking data includes eye tracking coordinates when the eye tracking module ETM is configured to track the user's eye movement. Eye tracking coordinates refer to, for example, the gaze coordinates of the user's eyeball. The eye tracking coordinates are transmitted to the content server for determining the dynamic high-resolution area. When the terminal device is configured to crop a second image with a second resolution based on the updated real-time eye tracking coordinates, the eye tracking coordinates are compared with the updated real-time eye tracking coordinates. In some embodiments, the eye tracking data includes instantaneous eye movement vector velocity and instantaneous eye movement acceleration. The term "instantaneous eye movement vector velocity" refers to the speed or rate of eye movement at a specific moment. It indicates the rate at which the eye moves in a specific direction at that moment. The term "instantaneous eye movement acceleration" refers to the acceleration of eye movement at a specific moment. Acceleration refers to the rate of change of velocity, which indicates how fast the eye's movement velocity changes at a specific moment.

[0055] In some embodiments, determining the dynamic high-resolution area includes: determining a first preset high-resolution area based on eye tracking coordinates detected at a first time point; calculating predicted eye tracking coordinates at a second time point based on an instantaneous eye movement vector velocity of the eye movement at the first time point and an instantaneous eye movement acceleration of the eye movement at the first time point; and determining a second preset high-resolution area based on the predicted eye tracking coordinates at the second time point. Optionally, the first preset high-resolution area and the second preset high-resolution area have the same width and the same length, but different center points. The dynamic high-resolution area is an area that includes the first preset high-resolution area and the second preset high-resolution area.

[0056] Therefore, in some embodiments, the image display method includes tracking the user's eye movement through a terminal device and obtaining eye tracking data; encoding the eye tracking data through the terminal device; and sending the eye tracking data to a content server through the terminal device.

[0057] In some embodiments, the method further comprises receiving eye tracking data from the terminal device via a content server; and decoding the eye tracking data via the content server. Optionally, the method further comprises determining a round trip time of the current time via the content server.

[0058] In some embodiments, the method further includes rendering, by the content server, the first image having the first resolution.

[0059] In some embodiments, the method further includes determining a dynamic high-resolution region based on the eye tracking data; and rendering, by the content server, a second image having a second resolution based on the dynamic high-resolution region.

[0060] In some embodiments, the method further comprises generating, by the content server, parameter data based on the dynamic high-resolution region.

[0061] In some embodiments, the method further comprises encoding, by the content server, a second image having a second resolution, a first image having a first resolution, and parameter data in three separate data streams, respectively, the three separate data streams comprising a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding the parameter data. Optionally, the first data stream, the second data stream, and the third data stream have different bit rates.

[0062] In the related image display method and display system, there is only one transmission stream for a single video. The present disclosure divides the single video into three separate data streams: a first data stream encoding a second image with a second resolution, a second data stream encoding the first image with a first resolution, and a third data stream encoding parameter data, surprisingly and unexpectedly, realizing a low-latency real-time communication transmission protocol. The present disclosure allows different compression rates for the three separate data streams, thereby ensuring clear image quality while reducing the bit rate.

[0063] In some embodiments, a first data stream encoding a second image having a second resolution is transmitted at a higher bit rate to prioritize image quality, and a second data stream encoding a first image having a first resolution is transmitted at a low bit rate. The first image having the first resolution undergoes pixel compression during the rendering process and is further compressed by encoding at a low bit rate to reduce the amount of transmitted data. In one example, a third data stream encoding parameter data is not compressed. The inventors of the present disclosure have discovered that by adopting a three-stream real-time communication transmission protocol, the present disclosure achieves compression and transmission of the above three different bit rates.

[0064] In some embodiments, the method further comprises sending, by the content server, the three separate data streams to the terminal device.

[0065] In some embodiments, the method further comprises decoding, by the terminal device, the three separate data streams.

[0066] In some embodiments, the method further comprises obtaining, by the terminal device, the updated real-time eye tracking coordinates.

[0067] In some embodiments, the method further comprises cropping, by the terminal device, the second image having the second resolution based on the updated real-time eye tracking coordinates, thereby generating an updated image.

[0068] In some embodiments, the method further includes combining, by the terminal device, the updated image with the first image having the first resolution to generate a composite image.

[0069] In some embodiments, the method further includes displaying the composite image via a terminal device.

[0070] As described above, in some embodiments, the method includes determining the round trip time of the current time by the content server. Optionally, the round trip time is determined periodically, for example, by the content server. The content server is also configured to receive eye tracking data including eye tracking coordinates detected at a first time point t1. The eye tracking coordinates can be represented by (x, y). In some embodiments, the method also includes calculating the predicted eye tracking coordinates (x', y') at the second time point t2 by the content server, where t2-t1=Δt. In one example, the difference Δt between the second time point t2 and the first time point t1 is substantially the same as the round trip time. As used herein, the term "substantially the same" means that the difference between two values ​​does not exceed 10% of the base value (e.g., one of the two values), for example, not more than 8%, not more than 6%, not more than 4%, not more than 2%, not more than 1%, not more than 0.5%, not more than 0.1%, not more than 0.05% and not more than 0.01% of the base value. As described above, in some embodiments, the eye tracking data also includes an instantaneous eye movement vector velocity v and an instantaneous eye movement acceleration a. In some embodiments, the method predefines a width w and a length h of the preset high-resolution area.

[0071] In some embodiments, the method includes determining, by the content server, the dynamic high-resolution area based on (w, h, v, a, Δt), wherein w represents the width of the preset high-resolution area, h represents the length of the preset high-resolution area, v represents the instantaneous eye movement vector velocity of the eye movement at the first time point, a represents the instantaneous eye movement acceleration of the eye movement at the first time point, and Δt represents the difference between the second time point and the first time point. Optionally, the difference Δt between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device.

[0072] In some embodiments, the method includes determining the center point (x', y'), width w' and length h' of the dynamic high-resolution area based on (w, h, v, a, Δt). Optionally, based on the change of the instantaneous eye movement vector velocity v and the instantaneous eye movement acceleration a over time, the center point (x', y'), width w' and length h' of the dynamic high-resolution area change dynamically over time. The high-resolution area of ​​the present disclosure is dynamically expanded and enlarged to cover the changes in the high-resolution gaze area caused by eye movement during transmission, thereby preventing any overflow of the high-resolution gaze area and minimizing the amount of data in the high-resolution area as much as possible.

[0073] In some embodiments, the method further includes frame synchronization of a first data stream encoding a second image having a second resolution, a second data stream encoding a first image having a first resolution, and a third data stream encoding parameter data. The term "frame synchronization" refers to the process of aligning frames or video sequences from different sources so that they can be displayed or processed together in a synchronized manner. It involves matching the timing and sequence of frames between multiple video streams to ensure correct playback or further processing. The inventors of the present disclosure have found that frame synchronization is necessary to ensure that a first data stream encoding a second image having a second resolution, a second data stream encoding a first image having a first resolution, and a third data stream encoding parameter data are played or processed together in perfect synchronization. Timestamp information (such as presentation timestamp (PTS)) is used to accurately align and synchronize the frames of these streams at the display end, thereby ensuring that visual and control parameters are properly coordinated and displayed without any noticeable differences or timing issues.

[0074] In some embodiments, during a process of encoding a second image having a second resolution, a first image having a first resolution, and parameter data into a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding parameter data, respectively, a timestamp of the third data stream is extracted from a presentation timestamp (PTS) of the first data stream and / or the second data stream.

[0075] A presentation timestamp (PTS) is a timestamp associated with each frame or packet in a video or audio stream. It represents the expected presentation time of the frame or packet in the media playback timeline. The PTS provides a reference for a decoder or renderer to properly synchronize frames for smooth and accurate playback. The PTS is typically measured in units of time, such as milliseconds or microseconds, and is used to determine the order in which frames should be displayed or processed. By comparing the PTS of each frame, the playback system can ensure that the frames are rendered or played back in the correct order and at the expected timing, thereby maintaining the temporal coherence and smoothness of the media presentation.

[0076] In some embodiments, the parameter data includes (x', y', w, h, w', h', pts), where x' and y' represent the coordinates of the center point of the dynamic high-resolution area, w represents the width of the preset high-resolution area, h represents the length of the preset high-resolution area, w' represents the width of the dynamic high-resolution area, h' represents the length of the dynamic high-resolution area, and pts represents the presentation timestamp. In some embodiments, the encoding parameter data includes encoding (x', y', w, h, w', h', pts). In some embodiments, the method includes sending the encoded (x', y', w, h, w', h', pts) from the content server to the terminal device. In one example, lossless compression (h265-lossless) is used to send the encoded (x', y', w, h, w', h', pts). The inventors of the present disclosure have found that it is advantageous to use lossless compression because the parameter data includes control parameters that require precision. Because the third data stream has a relatively small amount of data, it requires negligible bandwidth resources.

[0077] In some embodiments, rendering a second image with a second resolution based on a dynamic high-resolution region includes rendering a second image with a second resolution based at least in part on (x', y', w', h'), wherein x' and y' represent the coordinates of the center point of the dynamic high-resolution region, w' represents the width of the dynamic high-resolution region, and h' represents the length of the dynamic high-resolution region. The second image with the second resolution is rendered to have a center point at (x', y'), a width of x', and a length of y'. Compared with rendering a full high-resolution image, when rendering a second image with a second resolution in an area corresponding to the dynamic high-resolution region, the rendering pressure and the amount of data are greatly reduced. In one example, once the rendering is completed, the second image with the second resolution will undergo high bit rate compression encoding (h265-cbr) and be transmitted to the terminal device through the network as a first data stream. The second image with the second resolution is compressed at a high bit rate, resulting in a larger amount of data compared to a low bit rate encoding method. However, it significantly improves the image quality. Since it is a local high-resolution region and the amount of data is relatively small compared to a full high-resolution image, both image quality is ensured and the bandwidth requirements for transmission are reduced.

[0078] In some embodiments, rendering a first image having a first resolution includes performing horizontal compression and performing vertical compression. Horizontal compression refers to the process of reducing the number of pixels or adjusting the width of an image in the horizontal direction. It involves compressing an image horizontally by combining adjacent pixels or reducing the resolution in a horizontal plane. This compression technique can reduce the file size or amount of data of the image, making it more efficient for storage or transmission. On the other hand, vertical compression refers to the process of reducing the number of pixels or adjusting the height of an image in the vertical direction. It involves compressing an image vertically by combining adjacent pixels or reducing the resolution in a vertical plane. This compression technique can also reduce the file size or amount of data of the image, making it more efficient for storage or transmission. In one example, the first image having a first resolution has a width w" and a length h".

[0079] In one example, a first image with a first resolution is encoded using low bit rate compression (h265-cbr), and the first image with the first resolution is sent to a terminal device as a second data stream through a network. Because the user does not have strict image quality requirements for low-resolution areas, this low bit rate compression specifically for low-resolution areas significantly reduces the network bandwidth required for transmission.

[0080] The present disclosure utilizes a low-delay real-time communication protocol to send data streams. However, the low-delay real-time communication protocol may encounter the problem of packet loss during transmission. If only a single data stream is used, the loss of data packets will at most affect the frames associated with the lost packets, while the impact on subsequent displays is minimal. However, with the three-way stream splitting method, data stream synchronization becomes a focus. In some embodiments, when receiving three separate data streams, a synchronization process is implemented at the terminal device.

[0081] Figure 3 1 is a flowchart illustrating a process of frame synchronizing a first data stream encoding a second image having a second resolution, a second data stream encoding the first image having a first resolution, and a third data stream encoding parameter data according to some embodiments of the present disclosure. Figure 3 , the method includes receiving three separate data streams through a terminal device, and decoding the three separate data streams through the terminal device. After decoding, a decoded first data stream, a decoded second data stream, and a decoded third data stream are obtained. In some embodiments, the method also includes extracting a first presentation timestamp pts-A from the decoded first data stream through the terminal device, extracting a second presentation timestamp pts-B from the decoded second data stream through the terminal device, and extracting a third presentation timestamp pts-C from the decoded third data stream through the terminal device.

[0082] In some embodiments, the method further includes determining whether the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is the same as the third presentation timestamp pts-C. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the first presentation timestamp pts-A is the same as the third presentation timestamp pts-C, the method includes combining the second image having the second resolution and the first image having the first resolution.

[0083] When it is determined that the first presentation timestamp pts-A is different from the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B. If the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, it indicates that a frame loss occurs in the second data stream. When it is determined that the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, the method further includes discarding a current frame FA(n) of the decoded first data stream and waiting for a next frame FA(n+1) of the decoded first data stream.

[0084] If the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, it indicates that frame loss occurs in the first data stream. When determining that the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, the method further includes discarding the current frame FB(n) of the decoded second data stream and waiting for the next frame FB(n+1) of the decoded second data stream.

[0085] These processes are repeated until it is determined that the first presentation timestamp pts-A is identical to the second presentation timestamp pts-B, which indicates that the decoded first data stream and the decoded second data stream have been frame synchronized.

[0086] When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is the same as the third presentation timestamp pts-C. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the first presentation timestamp pts-A is the same as the third presentation timestamp pts-C, the method includes combining the second image having the second resolution and the first image having the first resolution.

[0087] When it is determined that the first presentation timestamp pts-A is different from the third presentation timestamp pts-C, the method further includes determining whether the first presentation timestamp pts-A is earlier than the third presentation timestamp pts-C. If the first presentation timestamp pts-A is earlier than the third presentation timestamp pts-C, it indicates that frame loss has occurred in the third data stream. When it is determined that the first presentation timestamp pts-A is earlier than the third presentation timestamp pts-C, the method further includes discarding the current frame FA(n) of the decoded first data stream and the current frame FB(n) of the decoded second data stream, and waiting for the next frame FA(n+1) of the decoded first data stream and the next frame FB(n+1) of the decoded second data stream.

[0088] If the first presentation timestamp pts-A is later than the third presentation timestamp pts-C, it indicates that frame loss occurs in the first data stream and frame loss occurs in the second data stream. When determining that the first presentation timestamp pts-A is later than the third presentation timestamp pts-C, the method further includes discarding the current frame FC(n) of the decoded third data stream and waiting for the next frame FC(n+1) of the decoded third data stream.

[0089] These processes are repeated until it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the third presentation timestamp pts-C, which indicates that the decoded first data stream, the decoded second data stream and the decoded third data stream have been frame synchronized.

[0090] Figure 4 1 is a flowchart illustrating a process of performing frame synchronization on a first data stream encoding a second image having a second resolution, a second data stream encoding the first image having a first resolution, and a third data stream encoding parameter data according to some embodiments of the present disclosure. Figure 4 , the method includes receiving three separate data streams by a terminal device, and decoding the three separate data streams by the terminal device. After decoding, a decoded first data stream, a decoded second data stream, and a decoded third data stream are obtained. In some embodiments, the method also includes extracting a first presentation timestamp pts-A from the decoded first data stream, extracting a second presentation timestamp pts-B from the decoded second data stream, and extracting a third presentation timestamp pts-C from the decoded third data stream.

[0091] In some embodiments, the method further includes determining whether the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the second presentation timestamp pts-B is the same as the third presentation timestamp pts-C. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, and the second presentation timestamp pts-B is the same as the third presentation timestamp pts-C, the method includes combining the second image having the second resolution with the first image having the first resolution.

[0092] When it is determined that the first presentation timestamp pts-A is different from the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B. If the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, it indicates that a frame loss occurs in the second data stream. When it is determined that the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, the method further includes discarding a current frame FA(n) of the decoded first data stream and waiting for a next frame FA(n+1) of the decoded first data stream.

[0093] If the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, it indicates that frame loss occurs in the first data stream. When determining that the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, the method further includes discarding the current frame FB(n) of the decoded second data stream and waiting for the next frame FB(n+1) of the decoded second data stream.

[0094] These processes are repeated until it is determined that the first presentation timestamp pts-A is identical to the second presentation timestamp pts-B, which indicates that the decoded first data stream and the decoded second data stream have been frame synchronized.

[0095] When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the second presentation timestamp pts-B is the same as the third presentation timestamp pts-C. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the second presentation timestamp pts-B is the same as the third presentation timestamp pts-C, the method includes combining the second image having the second resolution and the first image having the first resolution.

[0096] When it is determined that the second presentation timestamp pts-B is different from the third presentation timestamp pts-C, the method also includes determining whether the second presentation timestamp pts-B is earlier than the third presentation timestamp pts-C. If the second presentation timestamp pts-B is earlier than the third presentation timestamp pts-C, it indicates that frame loss has occurred in the third data stream. When it is determined that the second presentation timestamp pts-B is earlier than the third presentation timestamp pts-C, the method also includes discarding the current frame FA(n) of the decoded first data stream and the current frame FB(n) of the decoded second data stream, and waiting for the next frame FA(n+1) of the decoded first data stream and the next frame FB(n+1) of the decoded second data stream.

[0097] If the second presentation timestamp pts-B is later than the third presentation timestamp pts-C, it indicates that frame loss occurs in the first data stream and frame loss occurs in the second data stream. When determining that the second presentation timestamp pts-B is later than the third presentation timestamp pts-C, the method further includes discarding the current frame FC(n) of the decoded third data stream and waiting for the next frame FC(n+1) of the decoded third data stream.

[0098] These processes are repeated until it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the third presentation timestamp pts-C, which indicates that the decoded first data stream, the decoded second data stream and the decoded third data stream have been frame synchronized.

[0099] Figure 5 The process of rendering a second image with a second resolution and a first image with a first resolution in some embodiments of the present disclosure is shown. Figure 5 In some embodiments, the method includes predefining a preset high-resolution area PDH having a width w and a length h. A center point of the preset high-resolution area PDH is defined by the eye tracking coordinates detected at the first time point t1. In some embodiments, the method further includes calculating, by the content server, predicted eye tracking coordinates (x', y') at the second time point t2, where t2-t1=Δt.

[0100] In some embodiments, the method includes determining a center point (x', y'), a width w' and a length h' of a dynamic high-resolution area DHE based on (w, h, v, a, Δt), wherein w represents the width of a preset high-resolution area PDH, h represents the length of a preset high-resolution area PDH, v represents the instantaneous eye movement vector velocity of the eye movement at a first time point, a represents the instantaneous eye movement acceleration of the eye movement at the first time point, and Δt represents the difference between the second time point and the first time point. Optionally, the difference Δt between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device. In some embodiments, the four vertex coordinates of the dynamically extended high-resolution area DHE may be expressed as (x'-w' / 2, y'-h' / 2), (x'-w' / 2, y'+h' / 2), (x'+w' / 2, y'+h' / 2) and (x'+w' / 2, y'-h' / 2).

[0101] In some embodiments, the method further comprises rendering a first image FLRI having a first resolution, and rendering a second image HRI having a second resolution based on the dynamic high resolution area DHE.

[0102] Figure 6 The process of cropping a second image having a second resolution based on updated real-time eye tracking coordinates for generating an updated image in accordance with some embodiments of the present disclosure is shown. Figure 6 , the updated real-time eye tracking coordinates are expressed as (x3, y3). In some embodiments, based on parameter data including (x', y', w, h, w', h'), the method also includes cropping the second image HRI with a second resolution to obtain an updated image UHRI, wherein x' and y' represent the coordinates of the center point of the dynamic high-resolution area, w represents the width of the preset high-resolution area, h represents the length of the preset high-resolution area, w' represents the width of the dynamic high-resolution area, and h' represents the length of the dynamic high-resolution area. Optionally, the four vertex coordinates of the updated image UHRI can be expressed as (x3-w / 2, y3-h / 2), (x3-w / 2, y3+h / 2), (x3+w / 2, y3+h / 2) and (x3+w / 2, y3-h / 2). Since the dynamic high-resolution area DHE has been dynamically expanded, the dynamic high-resolution area DHE covers and includes the area corresponding to the updated image UHRI. Therefore, the dynamic high-resolution area DHE can be cropped based on the four vertex coordinates of the updated image UHRI to obtain the updated image UHRI. As Figure 6 As shown, in some embodiments, the method further includes combining the updated image UHRI with the first image FLRI having the first resolution to generate a composite image.

[0103] In some embodiments, combining the updated image UHRI and the first image FLRI having the first resolution includes expanding pixels of the first image FLRI having the first resolution and rendering the first image FLRI having the first resolution at full resolution; and rendering the updated image UHRI at a position based on four vertex coordinates of the updated image UHRI.

[0104] In another aspect, the present disclosure provides a display system. In some embodiments, the display system includes a content server and a terminal device. Figure 7 is a schematic diagram showing a terminal device according to some embodiments of the present disclosure. Figure 7 In some embodiments, the terminal device includes a processor 1002, a storage medium 1004, a display 1006, a communication module 1008, a database 1010, a peripheral device 1012, and a camera 1014. Some devices may be omitted and other devices may be included to better describe the relevant embodiments. The terminal device may include any appropriate type of display panel, such as a plasma display panel, a liquid crystal display (LCD) panel, a touch screen display panel, a projection display panel, a non-intelligent display panel, an intelligent display panel, etc. The terminal device may also include other computing systems, such as a personal computer (PC), a tablet computer, a mobile computer, or a smart phone. In addition, the terminal device may be any appropriate content presentation device capable of presenting any appropriate content. The user may interact with the terminal device to perform other activities of interest.

[0105] The processor 1002 may include any appropriate one or more processors. The processor 1002 may include multiple cores for multithreading or parallel processing. The processor 1002 may execute a sequence of computer program instructions to perform various processes. The storage medium 1004 may include a memory module (e.g., ROM, RAM, flash memory module) and a large capacity memory (e.g., CD-ROM and hard disk), etc. The storage medium 1004 may store a computer program to implement various processes when the computer program is executed by the processor 1002. For example, the storage medium 1004 may store a computer program to implement various algorithms when the computer program is executed by the processor 1002.

[0106] Further, the communication module 1008 may include some network interface devices for establishing a connection through a communication network (e.g., a TV cable network, a wireless network, the Internet). The database 1010 may include one or more databases for storing some data and performing some operations on some stored data, such as database search.

[0107] Display 1006 can provide information to a user. Display 1006 can include any suitable type of computer display device or electronic device display, such as an LCD or OLED based device. Peripheral devices 1012 can include various sensors or other I / O devices, such as a keyboard or mouse.

[0108] Examples of suitable terminal devices include, but are not limited to, electronic paper, mobile phones, tablet computers, televisions, monitors, laptop computers, digital photo albums, GPS, etc. Optionally, the terminal device is an organic light emitting diode display device. Optionally, the terminal device is a micro light emitting diode display device. Optionally, the terminal device is a mini light emitting diode display device.

[0109] In some embodiments, the terminal device includes a display panel; a first memory; and one or more first processors. Optionally, the first memory and the one or more first processors are connected to each other; and the first memory stores computer executable instructions for controlling the one or more first processors to perform various tasks described herein.

[0110] In some embodiments, the content server includes a second memory; and one or more second processors. Optionally, the second memory and the one or more second processors are connected to each other; and the second memory stores computer executable instructions for controlling the one or more second processors to perform various tasks described herein.

[0111] In some embodiments, the terminal device is configured to track the user's eye movement; obtain eye tracking data; and transmit the eye tracking data to the content server. In some embodiments, the content server is configured to: determine a dynamic high-resolution area based on the eye tracking data; render a first image with a first resolution; render a second image with a second resolution based on the dynamic high-resolution area; encode the second image with the second resolution, the first image with the first resolution, and parameter data in three separate data streams, respectively; and transmit the three separate data streams from the content server to the terminal device for image display; optionally, the second resolution is greater than the first resolution.

[0112] In some embodiments, the terminal device is also configured to obtain updated real-time eye tracking coordinates; crop the second image having the second resolution based on the updated real-time eye tracking coordinates detected by the terminal device to generate an updated image; and combine the first image having the first resolution with the updated image to generate a composite image.

[0113] In some embodiments, the three separate data streams include a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding parameter data.

[0114] In some embodiments, the content server is configured to transmit the first data stream at a first bit rate; transmit the second data stream at a second bit rate; and transmit the third data stream at a third bit rate. Optionally, the third bit rate is higher than the first bit rate. Optionally, the first bit rate is higher than the second bit rate.

[0115] In some embodiments, the content server is further configured to predefine a preset high-resolution area; receive the eye tracking data including the eye tracking coordinates detected by the terminal device at a first point in time; and calculate predicted eye tracking coordinates at a second point in time.

[0116] In some embodiments, the content server is configured to determine the dynamic high-resolution area based on: a width of a preset high-resolution area; a length of the preset high-resolution area; an instantaneous eye movement vector velocity of the eye movement at the first time point; an instantaneous eye movement acceleration of the eye movement at the first time point; and a difference between the second time point and the first time point. Optionally, the difference between the second time point and the first time point is substantially the same as a round-trip time between the content server and the terminal device. Optionally, the predicted eye tracking coordinates are a center point of the dynamic high-resolution area at the second time point. Optionally, based on changes in the instantaneous eye movement vector velocity and the instantaneous eye movement acceleration over time, the center point, width and length of the dynamic high-resolution area change dynamically over time.

[0117] In some embodiments, the terminal device is further configured to frame synchronize the first data stream encoding the second image having the second resolution, the second data stream encoding the first image having the first resolution, and the third data stream encoding the parameter data.

[0118] In some embodiments, the terminal device is also configured to extract a first presentation timestamp from the decoded first data stream; extract a second presentation timestamp from the decoded second data stream; extract a third presentation timestamp from the decoded third data stream; and determine whether the first presentation timestamp, the second presentation timestamp and the third presentation timestamp are the same.

[0119] In some embodiments, the terminal device is further configured to: when it is determined that the first presentation timestamp is earlier than the second presentation timestamp, discard the current frame of the decoded first data stream and wait for the next frame of the decoded first data stream; or when it is determined that the first presentation timestamp is later than the second presentation timestamp, discard the current frame of the decoded second data stream and wait for the next frame of the decoded second data stream.

[0120] In some embodiments, the terminal device is further configured to: when it is determined that the first presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the first presentation timestamp is later than the third presentation timestamp, discard the current frame of the decoded third data stream and wait for the next frame of the decoded third data stream.

[0121] In some embodiments, the terminal device is further configured to, when determining that the second presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or to discard the current frame of the decoded third data stream and wait for the next frame of the decoded third data stream when determining that the second presentation timestamp is later than the third presentation timestamp.

[0122] In some embodiments, the content server is configured to generate the parameter data based on the dynamic high resolution region. Optionally, the parameter data is transmitted from the content server to the terminal device using lossless compression.

[0123] In some embodiments, the content server is configured to transmit to the terminal device the coordinates of the center point of the dynamic high-resolution area; the width of the preset high-resolution area; the length of the preset high-resolution area; the width of the dynamic high-resolution area; the length of the dynamic high-resolution area; and a presentation timestamp.

[0124] In some embodiments, the content server is configured to render the second image having the second resolution based at least in part on coordinates of a center point of the dynamic high-resolution area, a width of the dynamic high-resolution area, and a length of the dynamic high-resolution area.

[0125] In some embodiments, the content server is configured to perform horizontal compression and vertical compression on the first image having the first resolution.

[0126] In some embodiments, the terminal device is further configured to display the composite image.

[0127] In another aspect, the present disclosure provides a computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon. In some embodiments, the computer-readable instructions may be executed by one or more first processors to cause the one or more first processors to perform: tracking the user's eye movements and obtaining eye tracking data; and transmitting the eye tracking data from the terminal device to a content server. In some embodiments, the computer-readable instructions may be executed by one or more second processors to cause the one or more second processors to perform: determining a dynamic high-resolution area based on the eye tracking data; rendering a first image with a first resolution; rendering a second image with a second resolution based on the dynamic high-resolution area; encoding the second image with the second resolution, the first image with the first resolution, and parameter data in three separate data streams, respectively; and transmitting the three separate data streams from the content server to the terminal device for image display. Optionally, the second resolution is greater than the first resolution.

[0128] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to: obtain updated real-time eye tracking coordinates; crop the second image having the second resolution based on the updated real-time eye tracking coordinates detected by the terminal device to generate an updated image; and combine the first image having the first resolution with the updated image to generate a composite image.

[0129] In some embodiments, the three separate data streams include a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding parameter data.

[0130] In some embodiments, the computer readable instructions are executable by one or more second processors to further cause the one or more second processors to: transmit the first data stream at a first bit rate; transmit the second data stream at a second bit rate; and transmit the third data stream at a third bit rate. Optionally, the third bit rate is higher than the first bit rate. Optionally, the first bit rate is higher than the second bit rate.

[0131] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to: predefine a preset high-resolution area; receive the eye tracking data including the eye tracking coordinates detected by the terminal device at a first point in time; and calculate the predicted eye tracking coordinates at a second point in time.

[0132] In some embodiments, the computer readable instructions may be executed by one or more second processors to further cause the one or more second processors to: determine the dynamic high-resolution area based on: a preset width of the high-resolution area; a preset length of the high-resolution area; an instantaneous eye movement vector velocity of the eye movement at the first time point; an instantaneous eye movement acceleration of the eye movement at the first time point; and a difference between the second time point and the first time point. Optionally, the difference between the second time point and the first time point is substantially the same as a round trip time between the content server and the terminal device.

[0133] In some embodiments, the predicted eye tracking coordinates are the center point of the dynamic high-resolution area at the second time point. Optionally, the center point, width and length of the dynamic high-resolution area change dynamically over time based on the change of instantaneous eye movement vector velocity and instantaneous eye movement acceleration over time.

[0134] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform: frame synchronization of the first data stream encoding the second image having the second resolution, the second data stream encoding the first image having the first resolution, and the third data stream encoding the parameter data.

[0135] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to: extract a first presentation timestamp from a decoded first data stream; extract a second presentation timestamp from a decoded second data stream through the terminal device; extract a third presentation timestamp from a decoded third data stream through the terminal device; and determine whether the first presentation timestamp, the second presentation timestamp, and the third presentation timestamp are the same.

[0136] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform: when it is determined that the first presentation timestamp is earlier than the second presentation timestamp, discard the current frame of the decoded first data stream and wait for the next frame of the decoded first data stream; or when it is determined that the first presentation timestamp is later than the second presentation timestamp, discard the current frame of the decoded second data stream and wait for the next frame of the decoded second data stream.

[0137] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform: when it is determined that the first presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the first presentation timestamp is later than the third presentation timestamp, discard the current frame of the decoded third data stream and wait for the next frame of the decoded third data stream.

[0138] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform: when it is determined that the second presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the second presentation timestamp is later than the third presentation timestamp, discard the current frame of the decoded third data stream and wait for the next frame of the decoded third data stream.

[0139] In some embodiments, the computer readable instructions are executable by one or more second processors to further cause the one or more second processors to generate the parameter data based on the dynamic high resolution region. Optionally, the parameter data is transmitted from the content server to the terminal device using lossless compression.

[0140] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to execute: transmitting the coordinates of the center point of the dynamic high-resolution area to the terminal device; presetting the width of the high-resolution area; presetting the length of the high-resolution area; the width of the dynamic high-resolution area; the length of the dynamic high-resolution area; and presenting a timestamp.

[0141] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to perform: rendering the second image having the second resolution based at least in part on the coordinates of a center point of the dynamic high-resolution area, the width of the dynamic high-resolution area, and the length of the dynamic high-resolution area.

[0142] In some embodiments, the computer readable instructions are executable by one or more second processors to further cause the one or more second processors to perform: performing horizontal compression and performing vertical compression on the first image having the first resolution.

[0143] In some embodiments, the computer readable instructions are executable by the one or more first processors to further cause the one or more first processors to: cause the display panel to display the composite image.

[0144] For the purpose of illustration and description, the above description of the embodiments of the present invention has been given. It is not exhaustive, nor is it intended to limit the present invention to the precise form or exemplary embodiments disclosed. Therefore, the foregoing description should be considered illustrative rather than restrictive. Obviously, many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described to explain the principles of the present invention and its best mode practical application, so that those skilled in the art can understand the various embodiments of the present invention and the various modifications suitable for the specific use or implementation under consideration. The scope of the present invention is intended to be defined by the appended claims and their equivalents, wherein all terms are meant to have the broadest reasonable meaning unless otherwise stated. Therefore, the term "the present invention" and the like do not necessarily limit the scope of the claims to a specific embodiment, and the reference to the exemplary embodiments of the present invention does not mean a limitation of the present invention, and such limitation should not be inferred. The present invention is limited only by the spirit and scope of the appended claims. In addition, these claims may involve the use of "first", "second", etc., followed by a noun or element. These terms should be understood as nomenclature, and should not be interpreted as limiting the number of elements modified by these nomenclatures, unless a specific number has been given. Any advantages and benefits described may not apply to all embodiments of the present invention. It should be understood that those skilled in the art may make changes to the described embodiments without departing from the scope of the present invention as defined by the appended claims. In addition, the elements and assemblies in this disclosure are not intended to be contributed to the public, regardless of whether the element or assembly is clearly described in the appended claims.

Claims

1. An image display method, comprising: Track the user's eye movements through the terminal device and obtain eye tracking data; transmitting the eye tracking data from the terminal device to a content server; determining, by the content server, a dynamic high-resolution region of an original image based on the eye tracking data; Rendering, by the content server, a first image having a first resolution, wherein the first image is a full low-resolution rendering of the original image; rendering, by the content server, a second image having a second resolution based on the dynamic high-resolution region, wherein the second image is a high-resolution rendering of a region of the original image; encoding, by the content server, the second image having the second resolution, the first image having the first resolution, and parameter data in a data stream; and Transmitting the data stream from the content server to the terminal device for image display; The second resolution is greater than the first resolution.

2. The method according to claim 1, further comprising: Obtaining updated real-time eye tracking coordinates via the terminal device; cropping, by the terminal device, the second image having the second resolution based on the updated real-time eye tracking coordinates detected by the terminal device, thereby generating an updated image; as well as The first image having the first resolution is combined with the updated image by the terminal device to generate a composite image.

3. The method according to claim 1, wherein: The data streams include a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding the parameter data.

4. The method according to claim 3, wherein: Transmitting the data stream from the content server to the terminal device comprises: transmitting the first data stream at a first bit rate; transmitting the second data stream at a second bit rate; and transmitting the third data stream at a third bit rate; wherein the third bit rate is higher than the first bit rate; and The first bit rate is higher than the second bit rate.

5. The method according to any one of claims 1 to 4, further comprising: Predefine a preset high-resolution area through the content server; receiving, by the content server, the eye tracking data including the eye tracking coordinates detected by the terminal device at a first time point; as well as The predicted eye tracking coordinates at a second point in time are calculated by the content server.

6. The method according to claim 5, wherein: The dynamic high-resolution area is determined based on: Preset the width of the high-resolution area; The length of the preset high-resolution area; an instantaneous eye movement vector velocity of said eye movement at said first point in time; an instantaneous eye movement acceleration of the eye movement at the first time point; and The difference between the second point in time and the first point in time.

7. The method according to claim 5, wherein: The difference between the second time point and the first time point is substantially the same as the round trip time between the content server and the terminal device.

8. The method according to claim 5, wherein: The predicted eye tracking coordinates are the center point of the dynamic high-resolution area at the second time point; and Based on the changes of instantaneous eye movement vector velocity and instantaneous eye movement acceleration over time, the center point, width and length of the dynamic high-resolution area dynamically change over time.

9. The method according to claim 3 further includes frame synchronization, through the terminal device, of the first data stream encoding the second image with the second resolution, the second data stream encoding the first image with the first resolution, and the third data stream encoding the parameter data.

10. The method according to claim 3, further comprising: extracting, by the terminal device, a first presentation timestamp from the decoded first data stream; extracting, by the terminal device, a second presentation timestamp from the decoded second data stream; extracting, by the terminal device, a third presentation timestamp from the decoded third data stream; as well as It is determined whether the first presentation timestamp, the second presentation timestamp, and the third presentation timestamp are the same.

11. The method according to claim 10, further comprising: When it is determined that the first presentation timestamp is earlier than the second presentation timestamp, discarding a current frame of the decoded first data stream and waiting for a next frame of the decoded first data stream; or When it is determined that the first presentation timestamp is later than the second presentation timestamp, a current frame of the decoded second data stream is discarded, and a next frame of the decoded second data stream is waited for.

12. The method according to claim 11, further comprising: When it is determined that the first presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or When it is determined that the first presentation timestamp is later than the third presentation timestamp, a current frame of the decoded third data stream is discarded, and a next frame of the decoded third data stream is waited for.

13. The method according to claim 11, further comprising: When it is determined that the second presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or When it is determined that the second presentation timestamp is later than the third presentation timestamp, a current frame of the decoded third data stream is discarded, and a next frame of the decoded third data stream is waited for.

14. The method of claim 1, further comprising generating, by the content server, the parameter data based on the dynamic high-resolution region.

15. The method according to claim 14, wherein: A third data stream encoding the parameter data is transmitted from the content server to the terminal device, including: transmitting the coordinates of the center point of the dynamic high-resolution area; the width of the preset high-resolution area; the length of the preset high-resolution area; the width of the dynamic high-resolution area; the length of the dynamic high-resolution area; and a presentation timestamp.

16. The method according to claim 1, wherein: Rendering the second image with the second resolution based on the dynamic high-resolution area includes: rendering the second image with the second resolution based at least in part on the coordinates of a center point of the dynamic high-resolution area, a width of the dynamic high-resolution area, and a length of the dynamic high-resolution area.

17. The method according to claim 1, wherein: Rendering the first image having the first resolution includes performing horizontal compression and performing vertical compression.

18. The method according to claim 2, wherein: The updated real-time eye tracking coordinates detected by the terminal device are represented as (x3, y3); The four vertex coordinates of the updated image are represented as (x3-w / 2, y3-h / 2), (x3-w / 2, y3+h / 2), (x3+w / 2, y3+h / 2) and (x3+w / 2, y3-h / 2); w represents the width of the preset high-resolution area; as well as h represents the length of the preset high-resolution area.

19. A display system comprising a content server and a terminal device; in, The terminal device is configured as follows: Tracking user’s eye movements; Obtain eye tracking data; transmitting the eye tracking data to the content server; Wherein, the content server is configured as: determining a dynamic high-resolution region of the original image based on the eye tracking data; rendering a first image having a first resolution, wherein the first image is a full low-resolution rendering of the original image; rendering a second image having a second resolution based on the dynamic high-resolution region, wherein the second image is a high-resolution rendering of the region of the original image; encoding in a data stream the second image having the second resolution, the first image having the first resolution and parameter data; and Transmitting the data stream from the content server to the terminal device for image display; The second resolution is greater than the first resolution.

20. A computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, The computer readable instructions are executable by the one or more first processors to cause the one or more first processors to: Tracking the user's eye movements and obtaining eye tracking data; and transmitting the eye tracking data from the terminal device to a content server; The computer readable instructions are executable by the one or more second processors to cause the one or more second processors to: determining a dynamic high-resolution region of the original image based on the eye tracking data; Rendering a first image having a first resolution, wherein The first image is a full low-resolution rendering of the original image; rendering a second image having a second resolution based on the dynamic high-resolution region, wherein the second image is a high-resolution rendering of the region of the original image; encoding in a data stream the second image having the second resolution, the first image having the first resolution and parameter data; and Transmitting the data stream from the content server to the terminal device for image display; The second resolution is greater than the first resolution.

Images