Image display methods and display systems

Abstract

An image display method is provided. The image display method includes: tracking a user's eye movement and obtaining eye-tracking data via a terminal device; transmitting the eye-tracking data from the terminal device to a content server; determining a dynamic high-resolution region of an original image based on the eye-tracking data via the content server; rendering a first image with a first resolution via the content server; rendering a second image with a second resolution via the content server based on the dynamic high-resolution region; encoding the second image with the second resolution, the first image with the first resolution, and parameter data in a data stream via the content server; and transmitting the data stream from the content server to the terminal device for image display. The second resolution is greater than the first resolution.

Description

Technical Field

[0001] This invention relates to display technology, and more particularly to an image display method and display system. Background Technology

[0002] Real-time communication transmission (RTC) refers to data transmission using real-time communication technology. Compared to traditional video network transmission technologies, RTC offers lower latency and higher interactivity. RTC transmission optimizes network transmission speed, latency, and bandwidth in real time, thereby improving real-time performance, stability, and audiovisual quality. As a result, RTC live streaming is more suitable for interactive live broadcasting, online education, remote healthcare, online conferencing, and other scenarios, making it highly applicable across various fields.

[0003] Eye-tracking technology is a biometric technology that utilizes the recognition of eye movement patterns to achieve various applications, such as identity authentication, emotion analysis, and gaze tracking. Meanwhile, network transmission technology is an essential component of modern information technology development. However, current technologies suffer from low transmission efficiency and poor transmission stability. To address these issues, this application proposes a partition-based transmission system with dynamic scaling for load balancing. Summary of the Invention

[0004] In one aspect, this disclosure provides an image display method, comprising: tracking a user's eye movement via a terminal device and obtaining eye-tracking data; transmitting the eye-tracking data from the terminal device to a content server; determining a dynamic high-resolution region of an original image based on the eye-tracking data via the content server; rendering a first image having a first resolution via the content server, wherein the first image is a full low-resolution rendering of the original image; rendering a second image having a second resolution based on the dynamic high-resolution region via the content server, wherein the second image is a high-resolution rendering of a region of the original image; encoding the second image having the second resolution, the first image having the first resolution, and parameter data in a data stream via the content server; and transmitting the data stream from the content server to the terminal device for image display; wherein the second resolution is greater than the first resolution.

[0005] Optionally, the method further includes: obtaining updated real-time eye-tracking coordinates through the terminal device; cropping the second image with the second resolution based on the updated real-time eye-tracking coordinates detected by the terminal device to generate an updated image; and combining the first image with the first resolution and the updated image through the terminal device to generate a composite image.

[0006] Optionally, the data stream includes a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding the parameter data.

[0007] Optionally, transmitting the data stream from the content server to the terminal device includes: transmitting the first data stream at a first bit rate; transmitting the second data stream at a second bit rate; and transmitting the third data stream at a third bit rate; wherein the third bit rate is higher than the first bit rate; and the first bit rate is higher than the second bit rate.

[0008] Optionally, the method further includes: predefining a preset high-resolution region via the content server; receiving eye-tracking data including the eye-tracking coordinates detected by the terminal device at a first time point via the content server; and calculating predicted eye-tracking coordinates at a second time point via the content server.

[0009] Optionally, the dynamic high-resolution region is determined based on: the width of the preset high-resolution region; the length of the preset high-resolution region; the instantaneous eye movement vector velocity of the eye movement at the first time point; the instantaneous eye movement acceleration of the eye movement at the first time point; and the difference between the second time point and the first time point.

[0010] Optionally, the difference between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device.

[0011] Optionally, the predicted eye-tracking coordinates are the center point of the dynamic high-resolution region at the second time point; and the center point, width, and length of the dynamic high-resolution region dynamically change over time based on the changes in instantaneous eye movement vector velocity and instantaneous eye movement acceleration over time.

[0012] Optionally, the method further includes performing frame synchronization on the first data stream encoding the second image having the second resolution, the second data stream encoding the first image having the first resolution, and the third data stream encoding the parameter data via the terminal device.

[0013] Optionally, the method further includes: extracting a first presentation timestamp from a decoded first data stream using the terminal device; extracting a second presentation timestamp from a decoded second data stream using the terminal device; extracting a third presentation timestamp from a decoded third data stream using the terminal device; and determining whether the first presentation timestamp, the second presentation timestamp, and the third presentation timestamp are the same.

[0014] Optionally, the method further includes: when it is determined that the first presentation timestamp is earlier than the second presentation timestamp, discarding the current frame of the decoded first data stream and waiting for the next frame of the decoded first data stream; or when it is determined that the first presentation timestamp is later than the second presentation timestamp, discarding the current frame of the decoded second data stream and waiting for the next frame of the decoded second data stream.

[0015] Optionally, the method further includes: when it is determined that the first presentation timestamp is earlier than the third presentation timestamp, discarding the current frame of the decoded first data stream and the current frame of the decoded second data stream, and waiting for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the first presentation timestamp is later than the third presentation timestamp, discarding the current frame of the decoded third data stream, and waiting for the next frame of the decoded third data stream.

[0016] Optionally, the method further includes: when it is determined that the second presentation timestamp is earlier than the third presentation timestamp, discarding the current frame of the decoded first data stream and the current frame of the decoded second data stream, and waiting for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the second presentation timestamp is later than the third presentation timestamp, discarding the current frame of the decoded third data stream, and waiting for the next frame of the decoded third data stream.

[0017] Optionally, the method further includes: generating the parameter data based on the dynamic high-resolution region via the content server.

[0018] Optionally, transmitting a third data stream encoding the parameter data from the content server to the terminal device includes: transmitting the coordinates of the center point of the dynamic high-resolution region; the width of the preset high-resolution region; the length of the preset high-resolution region; the width of the dynamic high-resolution region; the length of the dynamic high-resolution region; and a presentation timestamp.

[0019] Optionally, lossless compression can be used to transfer parameter data from the content server to the terminal device.

[0020] Optionally, rendering the second image with the second resolution based on the dynamic high-resolution region includes: rendering the second image with the second resolution based at least in part on the coordinates of the center point of the dynamic high-resolution region, the width of the dynamic high-resolution region, and the length of the dynamic high-resolution region.

[0021] Optionally, rendering the first image having the first resolution includes performing horizontal compression and performing vertical compression.

[0022] In another aspect, this disclosure provides a display system including a content server and a terminal device; wherein the terminal device is configured to: track a user's eye movements; obtain eye-tracking data; and transmit the eye-tracking data to the content server; wherein the content server is configured to: determine a dynamic high-resolution region of an original image based on the eye-tracking data; render a first image having a first resolution, wherein the first image is a full low-resolution rendering of the original image; render a second image having a second resolution based on the dynamic high-resolution region, wherein the second image is a high-resolution rendering of a region of the original image; encode the second image having the second resolution, the first image having the first resolution, and parameter data in a data stream; and transmit the data stream from the content server to the terminal device for image display; wherein the second resolution is greater than the first resolution.

[0023] In another aspect, this disclosure provides a computer program product including a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by one or more first processors to cause the one or more first processors to perform the following operations: tracking a user's eye movements and obtaining eye-tracking data; and transmitting the eye-tracking data from a terminal device to a content server; the computer-readable instructions being executable by one or more second processors to cause the one or more second processors to perform the following operations: determining a dynamic high-resolution region of an original image based on the eye-tracking data; rendering a first image having a first resolution, wherein the first image is a full low-resolution rendering of the original image; rendering a second image having a second resolution based on the dynamic high-resolution region, wherein the second image is a high-resolution rendering of a region of the original image; encoding the second image having the second resolution, the first image having the first resolution, and parameter data in a data stream; and transmitting the data stream from the content server to a terminal device for image display; wherein the second resolution is greater than the first resolution. Attached Figure Description

[0024] The following figures are merely illustrative examples based on various disclosed embodiments and are not intended to limit the scope of the invention.

[0025] Figure 1 This is a flowchart illustrating an image display method according to some embodiments of the present disclosure.

[0026] Figure 2 This is a schematic diagram illustrating a display system that implements an image display method according to some embodiments of the present disclosure.

[0027] Figure 3 This is a flowchart illustrating the process of frame synchronization of a first data stream encoding a second image with a second resolution, a second data stream encoding a first image with a first resolution, and a third data stream encoding parameter data according to some embodiments of the present disclosure.

[0028] Figure 4 This is a flowchart illustrating the process of frame synchronization of a first data stream encoding a second image with a second resolution, a second data stream encoding a first image with a first resolution, and a third data stream encoding parameter data according to some embodiments of the present disclosure.

[0029] Figure 5 The process of rendering a second image having a second resolution and a first image having a first resolution is illustrated according to some embodiments of the present disclosure.

[0030] Figure 6 The process of cropping a second image with a second resolution based on updated real-time eye-tracking coordinates used to generate an updated image, according to some embodiments of the present disclosure, is illustrated.

[0031] Figure 7 This is a schematic diagram illustrating a terminal device according to some embodiments of the present disclosure. Detailed Implementation

[0032] This disclosure will now be described in more detail with reference to the following embodiments. It should be noted that the following description of some embodiments presented herein is for illustrative and descriptive purposes only. It is not exhaustive or limited to the precise forms disclosed.

[0033] Users can use head-mounted displays (HMDs) to watch virtual reality content, such as content stored on cloud servers. Storing content in the cloud reduces the performance requirements on the client side compared to storing large amounts of virtual reality content locally. However, transmitting VR content over the internet faces challenges such as bandwidth limitations and latency.

[0034] Therefore, this disclosure provides, in particular, an image display method and display system that substantially eliminates one or more problems caused by the limitations and disadvantages of the prior art. In one aspect, this disclosure provides an image display method. In some embodiments, the image display method includes: tracking a user's eye movement via a terminal device and obtaining eye-tracking data; transmitting the eye-tracking data from the terminal device to a content server; determining, via the content server, a dynamic high-resolution region of an original image based on the eye-tracking data; rendering, via the content server, a first image having a first resolution, wherein the first image is a full low-resolution rendering of the original image; rendering, via the content server, a second image having a second resolution based on the dynamic high-resolution region, wherein the second image is a high-resolution rendering of a region of the original image; encoding, via the content server, the second image having the second resolution, the first image having the first resolution, and parameter data in a data stream; and transmitting the data stream from the content server to the terminal device for image display. The second resolution is greater than the first resolution. Optionally, the first image is a full low-resolution image; the second image is a high-resolution image; and the resolution of the high-resolution image is greater than the resolution of the full low-resolution image.

[0035] Figure 1 This is a flowchart illustrating an image display method according to some embodiments of the present disclosure. (Reference) Figure 1 In some embodiments, the image display method includes: tracking a user's eye movements and obtaining eye-tracking data via a terminal device; transmitting the eye-tracking data from the terminal device to a content server; determining a dynamic high-resolution region via the content server based on the eye-tracking data; rendering a first image with a first resolution via the content server; rendering a second image with a second resolution via the content server based on the dynamic high-resolution region; encoding the second image with the second resolution, the first image with the first resolution, and parameter data in three separate data streams respectively; transmitting the three separate data streams from the content server to the terminal device; and cropping the second image with the second resolution via the terminal device based on updated real-time eye-tracking coordinates detected by the terminal device, thereby generating an updated image. Optionally, the second resolution is greater than the first resolution.

[0036] Figure 2 This is a schematic diagram illustrating a display system that implements an image display method according to some embodiments of the present disclosure. (Refer to...) Figure 1 and Figure 2In some embodiments, the display system implementing the image display method includes a content server (CS) and a terminal device (TD). The content server (CS) and the terminal device are configured to communicate via a network (NT) (e.g., the Internet).

[0037] The content server (CS) is a critical component in a distributed system, responsible for storing and delivering content to clients or end users. It acts as a central repository for data (including files, documents, media, or any other form of information that needs to be accessed or distributed within the system). The content server (CS) efficiently and securely manages the storage, retrieval, and delivery of content.

[0038] Terminal devices (TDs) refer to the hardware or interface through which users interact with a system or network. They act as gateways for users to access and utilize the functions and services provided by the system. Terminal devices can take various forms. Examples include personal computers; mobile devices such as smartphones and tablets; the Internet of Things (IoT), such as smart TVs, smart speakers, and home automation systems; and virtual reality / augmented reality devices, such as head-mounted displays (HMDs) and other immersive devices. Terminal devices typically have software applications or web browsers installed, enabling users to access system services, browse content, perform transactions, or communicate with other users. These devices communicate with servers or backend systems via various protocols, such as HTTP, TCP / IP, or proprietary communication protocols.

[0039] refer to Figure 2 In some embodiments, the content server CS includes an image generation module IGM, an image processing module IPM, a tracking and prediction module TPM, a first encoding / decoding module EDM1, and a first network transmission module NTM1. The image processing module IPM is configured to generate raw images (e.g., video streams) and send the raw images to the image processing module IPM.

[0040] In some embodiments, the first network transmission module NTM1 is configured to receive eye-tracking data from the terminal device TD and to send the eye-tracking data to the first codec module EDM1. The first codec module EDM1 is configured to decode the eye-tracking data and transmit the eye-tracking data to the image processing module IPM and the tracking prediction module TPM. The first network transmission module NTM1 is configured to determine the round-trip time for the current time. As used herein, the term "round-trip time" refers to the total time it takes for a data packet to travel from the source to the destination and back again. It is a measure of latency or delay in the network. To calculate the round-trip time, a device (such as a computer or server) sends a data packet to the destination, and the destination acknowledges receipt of the packet by sending it back to the source. The round-trip time is then determined by measuring the time it takes for the packet to travel from the source to the destination and back again.

[0041] In some embodiments, the image processing module (IPM) is configured to render a first image with a first resolution. The first image with the first resolution refers to an image that has been rendered as a whole but with low detail or a low resolution level. In one example, the IPM generates a full image with simplified details. The first image with the first resolution serves as the basis for rendering subsequent high-resolution regions on top of it. The inventors of this disclosure have discovered that by rendering the full image at a lower resolution, the system can focus computational effort on generating high-resolution portions only for specific areas requiring more detail. This approach helps optimize network transmission and resource allocation while still providing the user with an acceptable level of visual representation.

[0042] In some embodiments, upon receiving eye-tracking data, the Tracking Prediction Module (TPM) is configured to determine a dynamic high-resolution region based on the eye-tracking data. A dynamic high-resolution region refers to a specific area within an image that is rendered with a higher level of detail or resolution compared to the rest of the image. In some embodiments, this region is calculated based on eye-tracking data and dynamically adjusted as the user's gaze or focus changes. The purpose of dynamic high-resolution regions is to efficiently allocate computational resources and network bandwidth. Instead of rendering the entire image at high resolution (which would require significant computational power and transmission capacity), the system focuses on rendering only the specific region corresponding to where the user is looking. By utilizing eye-tracking data, the system can determine the user's point of interest or gaze and dynamically adjust the high-resolution rendering accordingly. This ensures that the area the user is actively looking at appears sharp and detailed, while the remaining areas can be rendered at a lower resolution to conserve resources. The inventors of this disclosure have discovered that by prioritizing the rendering of important regions with high detail, this dynamic method allows for more efficient allocation of computational resources, reduced network bandwidth requirements, and an enhanced visual experience for the user.

[0043] In some embodiments, the Tracking Prediction Module (TPM) is also configured to generate parameter data based on a dynamic high-resolution region. In some embodiments, the parameter data includes specific configuration settings applied only to the dynamic high-resolution region. These settings can control aspects such as image processing algorithms, rendering techniques, or quality parameters specifically tailored to the dynamic high-resolution region. In some embodiments, the parameter data also includes data providing additional descriptive information about the dynamic high-resolution region. This parameter data may include attributes such as the size or dimensions of the dynamic high-resolution region, its location in the image, or any other relevant characteristics. In some embodiments, the parameter data also includes timestamp information. Timestamp information provides timing or time data associated with the dynamic high-resolution region. It helps to synchronize and coordinate actions or processes related to high-resolution rendering and ensure proper alignment with other components or systems. By including timestamp information in the parameter data, the system can accurately track and manage the timing aspects of the dynamic high-resolution region. This information can be used for synchronization purposes, data coordination, or to ensure that high-resolution rendering is correctly aligned with other visual or interactive elements in the system.

[0044] In some embodiments, the Image Processing Module (IPM) is configured to render a second image with a second resolution based on dynamic high-resolution regions. The IPM utilizes various image processing techniques, algorithms, and computational resources to render the second image with the second resolution. It focuses its rendering efforts on dynamic high-resolution regions, thereby ensuring that these specific regions appear with enhanced detail, sharpness, and visual fidelity. The inventors of this disclosure have discovered that by selectively rendering a second image with a second resolution for dynamic high-resolution regions, the IPM optimizes computational resources and reduces processing overhead. Instead of rendering the entire image at high resolution (which can be computationally intensive), the IPM prioritizes regions requiring more detail based on dynamic high-resolution regions. The second image with the second resolution generated by the IPM can be further used for subsequent processing, such as encoding, compression, or transmission. It forms part of the overall image representation that is ultimately transmitted to a display for user viewing.

[0045] In some embodiments, the first encoding / decoding module EDM1 is configured to encode a second image with a second resolution, a first image with a first resolution, and parameter data in three separate data streams, respectively. The three separate data streams include a first data stream encoding the second image with the second resolution, a second data stream encoding the first image with the first resolution, and a third data stream encoding the parameter data. Optionally, the first, second, and third data streams have different bit rates.

[0046] In some embodiments, the first network transmission module NTM1 is configured to send three separate data streams to the terminal device TD, for example, to send three separate data streams to the second network transmission module NTM2 in the terminal device TD. Optionally, the first network transmission module NTM1 is configured to send three separate data streams to the second network transmission module NTM2 via Web Real-Time Communication (WebRTC).

[0047] The inventors of this disclosure have discovered a delay during the transmission of eye-tracking data from the terminal device TD to the content server CS, and a delay during the transmission of a second image with a second resolution from the content server CS to the terminal device TD. In public network transmissions, network latency can vary due to user location, different telecommunications broadband providers, and varying network access times (such as during periods of low or peak network usage). Therefore, network latency can be relatively high, sometimes exceeding 100 milliseconds, during which significant changes in eye position can occur, introducing significant errors.

[0048] In this disclosure, eye-tracking data is sent to a content server to predict eye coordinates. Furthermore, this disclosure defines a dynamic high-resolution region to ensure that dynamically expanded and magnified high-resolution regions cover changes in the high-resolution gaze region caused by eye movement during transmission, thereby preventing any overflow of the high-resolution gaze region. In one example, the dynamic high-resolution region may be a dynamic high-resolution expansion region.

[0049] Reference Figure 2 In some embodiments, the terminal device TD includes a second network transmission module NTM2, a second codec module EDM2, an eye-tracking module ETM, and a display module DM. The second network transmission module NTM2 is configured to receive three separate data streams from a first network transmission module NTM1. The second codec module EDM2 is configured to decode the three separate data streams. Optionally, the second codec module EDM2 is configured to decode each of the three separate data streams separately to extract the corresponding data. Once the second codec module EDM2 has successfully received and extracted data from the three streams, further processing can be performed. This may include combining high-resolution and low-resolution data, applying any necessary image processing based on the information streams, or any other operations required to generate a final image for display or further use.

[0050] In some embodiments, the display module DM is configured to crop a second image with a second resolution based on updated real-time eye-tracking coordinates detected by the eye-tracking module ETM, thereby generating an updated image. The updated real-time eye-tracking coordinates help determine a specific region of interest.

[0051] In some embodiments, the display module DM is configured to combine an updated image with a first image having a first resolution. Optionally, the combination process involves combining a corresponding portion of the updated image with a first image having a first resolution to generate a composite image.

[0052] In some embodiments, the eye-tracking module ETM is configured to track the user's eye movements and acquire eye-tracking data. Optionally, the second codec module EDM2 is configured to encode the eye-tracking data and send it to the second network transmission module NTM2. Optionally, the second network transmission module NTM2 is configured to transmit the eye-tracking data to the first network transmission module NTM1.

[0053] In some embodiments, the eye-tracking module ETM is further configured to obtain updated real-time eye-tracking coordinates and to send the updated real-time eye-tracking coordinates to the second codec module EDM2. Optionally, the second codec module EDM2 is configured to transmit the updated real-time eye-tracking coordinates to the display module DM. In an alternative embodiment, the eye-tracking module ETM is configured to directly transmit the updated real-time eye-tracking coordinates to the display module DM.

[0054] In some embodiments, eye-tracking data includes eye-tracking coordinates as the eye-tracking module (ETM) is configured to track the user's eye movements. Eye-tracking coordinates refer to, for example, the gaze coordinates of the user's eyeballs. The eye-tracking coordinates are transmitted to a content server for determining dynamic high-resolution regions. When the terminal device is configured to crop a second image with a second resolution based on updated real-time eye-tracking coordinates, the eye-tracking coordinates are compared with the updated real-time eye-tracking coordinates. In some embodiments, eye-tracking data includes instantaneous eye movement vector velocity and instantaneous eye movement acceleration. The term "instantaneous eye movement vector velocity" refers to the speed or rate of eye movement at a specific moment. It represents the rate at which the eye moves in a specific direction at that moment. The term "instantaneous eye movement acceleration" refers to the acceleration of eye movement at a specific moment. Acceleration refers to the rate of change of velocity; it represents how quickly the speed of eye movement changes at a specific moment.

[0055] In some embodiments, determining the dynamic high-resolution region includes: determining a first preset high-resolution region based on eye-tracking coordinates detected at a first time point; calculating predicted eye-tracking coordinates at a second time point based on the instantaneous eye movement vector velocity and instantaneous eye movement acceleration at the first time point; and determining a second preset high-resolution region based on the predicted eye-tracking coordinates at the second time point. Optionally, the first and second preset high-resolution regions have the same width and the same length, but different center points. The dynamic high-resolution region is the region that includes both the first and second preset high-resolution regions.

[0056] Therefore, in some embodiments, the image display method includes tracking a user's eye movements and obtaining eye-tracking data via a terminal device; encoding the eye-tracking data via the terminal device; and sending the eye-tracking data to a content server via the terminal device.

[0057] In some embodiments, the method further includes receiving eye-tracking data from a terminal device via a content server; and decoding the eye-tracking data via the content server. Optionally, the method further includes determining the round-trip time of the current time via the content server.

[0058] In some embodiments, the method further includes rendering a first image having a first resolution via a content server.

[0059] In some embodiments, the method further includes determining a dynamic high-resolution region based on eye-tracking data; and rendering a second image with a second resolution based on the dynamic high-resolution region via a content server.

[0060] In some embodiments, the method further includes generating parameter data based on a dynamic high-resolution region via a content server.

[0061] In some embodiments, the method further includes encoding a second image with a second resolution, a first image with a first resolution, and parameter data in three separate data streams via a content server, the three separate data streams including a first data stream encoding the second image with the second resolution, a second data stream encoding the first image with the first resolution, and a third data stream encoding the parameter data. Optionally, the first data stream, the second data stream, and the third data stream have different bit rates.

[0062] In related image display methods and systems, there is only one transport stream for a single video. This disclosure divides a single video into three separate data streams: a first data stream encoding a second image with a second resolution, a second data stream encoding a first image with a first resolution, and a third data stream encoding parameter data. Surprisingly and unexpectedly, this achieves a low-latency real-time communication transport protocol. This disclosure allows for different compression rates for the three separate data streams, thereby ensuring clear image quality while reducing the bit rate.

[0063] In some embodiments, a first data stream encoding a second image with a second resolution is transmitted at a higher bit rate to prioritize image quality, while a second data stream encoding a first image with a first resolution is transmitted at a lower bit rate. The first image with the first resolution undergoes pixel compression during the rendering process and is further compressed by encoding at a low bit rate to reduce the amount of data transmitted. In one example, a third data stream encoding parameter data is not compressed. The inventors of this disclosure have discovered that by employing a three-stream real-time communication transmission protocol, this disclosure achieves the compression and transmission of the three different bit rates described above.

[0064] In some embodiments, the method further includes sending the three separate data streams to the terminal device via the content server.

[0065] In some embodiments, the method further includes decoding the three separate data streams via the terminal device.

[0066] In some embodiments, the method further includes obtaining the updated real-time eye-tracking coordinates via the terminal device.

[0067] In some embodiments, the method further includes cropping the second image with the second resolution based on the updated real-time eye-tracking coordinates using the terminal device, thereby generating an updated image.

[0068] In some embodiments, the method further includes combining the updated image with a first image having a first resolution via a terminal device to generate a composite image.

[0069] In some embodiments, the method further includes displaying the synthesized image via a terminal device.

[0070] As described above, in some embodiments, the method includes determining the round-trip time of the current time via a content server. Optionally, for example, the round-trip time is determined periodically via the content server. The content server is also configured to receive eye-tracking data including eye-tracking coordinates detected at a first time point t1. The eye-tracking coordinates may be represented by (x, y). In some embodiments, the method further includes calculating predicted eye-tracking coordinates (x', y') at a second time point t2 via the content server, where t2 - t1 = Δt. In one example, the difference Δt between the second time point t2 and the first time point t1 is substantially the same as the round-trip time. As used herein, the term "substantially the same" means that the difference between the two values ​​does not exceed 10% of a baseline value (e.g., one of the two values), such as not exceeding 8%, 6%, 4%, 2%, 1%, 0.5%, 0.1%, 0.05%, and 0.01% of the baseline value. As described above, in some embodiments, the eye-tracking data also includes instantaneous eye movement vector velocity v and instantaneous eye movement acceleration a. In some embodiments, the method predefines the width w and length h of a preset high-resolution region.

[0071] In some embodiments, the method includes determining the dynamic high-resolution region via the content server based on (w, h, v, a, Δt), where w represents the width of a preset high-resolution region, h represents the length of the preset high-resolution region, v represents the instantaneous eye movement vector velocity at a first time point, a represents the instantaneous eye movement acceleration at the first time point, and Δt represents the difference between a second time point and the first time point. Optionally, the difference Δt between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device.

[0072] In some embodiments, the method includes determining the center point (x', y'), width w', and length h' of the dynamic high-resolution region based on (w, h, v, a, Δt). Optionally, the center point (x', y'), width w', and length h' of the dynamic high-resolution region dynamically change over time based on the changes in instantaneous eye movement vector velocity v and instantaneous eye movement acceleration a over time. The high-resolution region of this disclosure is dynamically expanded and enlarged to cover changes in the high-resolution gaze region caused by eye movement during transmission, thereby preventing any overflow of the high-resolution gaze region and minimizing the amount of data in the high-resolution region as much as possible.

[0073] In some embodiments, the method further includes frame synchronization of a first data stream encoding a second image with a second resolution, a second data stream encoding a first image with a first resolution, and a third data stream encoding parameter data. The term "frame synchronization" refers to the process of aligning frames or video sequences from different sources so that they can be displayed or processed together in a synchronized manner. It involves matching the timing and sequence of frames between multiple video streams to ensure proper playback or further processing. The inventors of this disclosure have found that frame synchronization is necessary to ensure that the first data stream encoding a second image with a second resolution, the second data stream encoding a first image with a first resolution, and the third data stream encoding parameter data are played or processed together in perfect synchronization. Timestamp information (such as a presentation timestamp (PTS)) is used to accurately align and synchronize the frames of these streams at the display end, thereby ensuring that visual and control parameters are properly coordinated and displayed without any noticeable discrepancies or timing issues.

[0074] In some embodiments, during the process of encoding a second image with a second resolution, a first image with a first resolution, and parameter data into a first data stream encoding the second image with a second resolution, a second data stream encoding the first image with a first resolution, and a third data stream encoding parameter data, the timestamp of the third data stream is extracted from the presentation timestamp (PTS) of the first data stream and / or the second data stream.

[0075] A Presentation Timestamp (PTS) is a timestamp associated with each frame or packet in a video or audio stream. It represents the expected presentation time of a frame or packet within the media playback timeline. The PTS provides a reference for the decoder or renderer to correctly synchronize frames for smooth and accurate playback. PTS is typically measured in units of time (such as milliseconds or microseconds) and is used to determine the order in which frames should be displayed or processed. By comparing the PTS of each frame, the playback system ensures that frames are rendered or played back in the correct order and at the expected timing, thus maintaining the temporal coherence and smoothness of media presentation.

[0076] In some embodiments, the parameter data includes (x',y',w,h,w',h',pts), where x' and y' represent the coordinates of the center point of the dynamic high-resolution region, w represents the width of the preset high-resolution region, h represents the length of the preset high-resolution region, w' represents the width of the dynamic high-resolution region, h' represents the length of the dynamic high-resolution region, and pts represents the presentation timestamp. In some embodiments, the encoded parameter data includes encoded (x',y',w,h,w',h',pts). In some embodiments, the method includes sending the encoded (x',y',w,h,w',h',pts) from a content server to a terminal device. In one example, lossless compression (h265-lossless) is used to send the encoded (x',y',w,h,w',h',pts). The inventors of this disclosure have found that using lossless compression is advantageous because the parameter data includes control parameters that require precision. Because the third data stream has a relatively small data volume, it requires negligible bandwidth resources.

[0077] In some embodiments, rendering a second image with a second resolution based on a dynamic high-resolution region includes rendering the second image with a second resolution at least partially based on (x', y', w', h'), where x' and y' represent the coordinates of the center point of the dynamic high-resolution region, w' represents the width of the dynamic high-resolution region, and h' represents the length of the dynamic high-resolution region. The second image with the second resolution is rendered with a center point at (x', y'), a width of x', and a length of y'. Compared to rendering a full-resolution image, rendering a second image with a second resolution in a region corresponding to the dynamic high-resolution region significantly reduces rendering pressure and data volume. In one example, once rendering is complete, the second image with the second resolution undergoes high-bit-rate compression encoding (H.265-CBR) and is transmitted to the terminal device via the network as a first data stream. The second image with the second resolution is compressed at a high bit rate, resulting in a larger data volume compared to low-bit-rate encoding methods. However, it significantly improves image quality. Because it is a local high-resolution region and the data volume is relatively small compared to a full-resolution image, both image quality and transmission bandwidth requirements are ensured.

[0078] In some embodiments, rendering a first image having a first resolution includes performing horizontal compression and performing vertical compression. Horizontal compression refers to the process of reducing the number of pixels or adjusting the width of an image in the horizontal direction. It involves compressing the image horizontally by combining adjacent pixels or reducing the resolution within a horizontal plane. This compression technique can reduce the file size or data volume of the image, making it more efficient for storage or transmission. Vertical compression, on the other hand, refers to the process of reducing the number of pixels or adjusting the height of an image in the vertical direction. It involves compressing the image vertically by combining adjacent pixels or reducing the resolution within a vertical plane. This compression technique can also reduce the file size or data volume of the image, making it more efficient for storage or transmission. In one example, the first image having a first resolution has a width w” and a length h”.

[0079] In one example, a first image with a first resolution is encoded using low bit-rate compression (H.265-CBR), and this first image with the first resolution is sent to the terminal device via the network as a second data stream. Because users do not have strict image quality requirements for low-resolution areas, this low bit-rate compression, specifically designed for low-resolution areas, significantly reduces the network bandwidth required for transmission.

[0080] This disclosure utilizes a low-latency real-time communication protocol to transmit data streams. However, low-latency real-time communication protocols may encounter packet loss issues during transmission. If only a single data stream is used, the loss of data packets at most affects the frames associated with the lost packets, with minimal impact on subsequent display. However, with a three-stream splitting approach, data stream synchronization becomes a concern. In some embodiments, the synchronization process is implemented at the terminal device when three separate data streams are received.

[0081] Figure 3 This is a flowchart illustrating the process of frame synchronization of a first data stream encoding a second image with a second resolution, a second data stream encoding a first image with a first resolution, and a third data stream encoding parameter data, according to some embodiments of this disclosure. (See reference...) Figure 3 The method includes receiving three separate data streams via a terminal device and decoding the three separate data streams via the terminal device. After decoding, a decoded first data stream, a decoded second data stream, and a decoded third data stream are obtained. In some embodiments, the method further includes extracting a first presentation timestamp pts-A from the decoded first data stream, extracting a second presentation timestamp pts-B from the decoded second data stream, and extracting a third presentation timestamp pts-C from the decoded third data stream via the terminal device.

[0082] In some embodiments, the method further includes determining whether a first presentation timestamp pts-A is the same as a second presentation timestamp pts-B. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is the same as a third presentation timestamp pts-C. When it is determined that both the first presentation timestamp pts-A and the second presentation timestamp pts-B are the same and the first presentation timestamp pts-A is the same as the third presentation timestamp pts-C, the method includes combining a second image having a second resolution and a first image having a first resolution.

[0083] When determining that the first presentation timestamp pts-A is different from the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B. If the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, it indicates that a frame loss has occurred in the second data stream. When determining that the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, the method further includes discarding the current frame FA(n) of the decoded first data stream and waiting for the next frame FA(n+1) of the decoded first data stream.

[0084] If the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, it indicates that a frame loss has occurred in the first data stream. Upon determining that the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, the method also includes discarding the current frame FB(n) of the decoded second data stream and waiting for the next frame FB(n+1) of the decoded second data stream.

[0085] Repeat these processes until it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, which means that the decoded first data stream and the decoded second data stream have been frame-synchronized.

[0086] When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is the same as the third presentation timestamp pts-C. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the first presentation timestamp pts-C is the same, the method includes combining a second image having a second resolution and a first image having a first resolution.

[0087] When determining that the first presentation timestamp pts-A is different from the third presentation timestamp pts-C, the method further includes determining whether the first presentation timestamp pts-A is earlier than the third presentation timestamp pts-C. If the first presentation timestamp pts-A is earlier than the third presentation timestamp pts-C, it indicates that a frame loss has occurred in the third data stream. When determining that the first presentation timestamp pts-A is earlier than the third presentation timestamp pts-C, the method further includes discarding the current frame FA(n) of the decoded first data stream and the current frame FB(n) of the decoded second data stream, and waiting for the next frame FA(n+1) of the decoded first data stream and the next frame FB(n+1) of the decoded second data stream.

[0088] If the first presentation timestamp pts-A is later than the third presentation timestamp pts-C, it indicates that a frame loss occurred in the first data stream and also in the second data stream. When determining that the first presentation timestamp pts-A is later than the third presentation timestamp pts-C, the method further includes discarding the current frame FC(n) of the decoded third data stream and waiting for the next frame FC(n+1) of the decoded third data stream.

[0089] Repeat these processes until it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the third presentation timestamp pts-C. This indicates that the first data stream, the second data stream, and the third data stream have been frame-synchronized.

[0090] Figure 4 This is a flowchart illustrating the process of frame synchronization of a first data stream encoding a second image with a second resolution, a second data stream encoding a first image with a first resolution, and a third data stream encoding parameter data, according to some embodiments of this disclosure. (See reference...) Figure 4 The method includes receiving three separate data streams via a terminal device and decoding the three separate data streams via the terminal device. After decoding, a decoded first data stream, a decoded second data stream, and a decoded third data stream are obtained. In some embodiments, the method further includes extracting a first presentation timestamp pts-A from the decoded first data stream, a second presentation timestamp pts-B from the decoded second data stream, and a third presentation timestamp pts-C from the decoded third data stream.

[0091] In some embodiments, the method further includes determining whether a first presentation timestamp pts-A is the same as a second presentation timestamp pts-B. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the second presentation timestamp pts-B is the same as a third presentation timestamp pts-C. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, and the second presentation timestamp pts-B is the same as the third presentation timestamp pts-C, the method includes combining a second image having a second resolution with a first image having a first resolution.

[0092] When determining that the first presentation timestamp pts-A is different from the second presentation timestamp pts-B, the method further includes determining whether the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B. If the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, it indicates that a frame loss has occurred in the second data stream. When determining that the first presentation timestamp pts-A is earlier than the second presentation timestamp pts-B, the method further includes discarding the current frame FA(n) of the decoded first data stream and waiting for the next frame FA(n+1) of the decoded first data stream.

[0093] If the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, it indicates that a frame loss has occurred in the first data stream. Upon determining that the first presentation timestamp pts-A is later than the second presentation timestamp pts-B, the method also includes discarding the current frame FB(n) of the decoded second data stream and waiting for the next frame FB(n+1) of the decoded second data stream.

[0094] Repeat these processes until it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, which means that the decoded first data stream and the decoded second data stream have been frame-synchronized.

[0095] When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B, the method further includes determining whether the second presentation timestamp pts-B is the same as the third presentation timestamp pts-C. When it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the second presentation timestamp pts-B is the same as the third presentation timestamp pts-C, the method includes combining a second image having a second resolution and a first image having a first resolution.

[0096] When determining that the second presentation timestamp pts-B is different from the third presentation timestamp pts-C, the method further includes determining whether the second presentation timestamp pts-B is earlier than the third presentation timestamp pts-C. If the second presentation timestamp pts-B is earlier than the third presentation timestamp pts-C, it indicates that a frame loss has occurred in the third data stream. When determining that the second presentation timestamp pts-B is earlier than the third presentation timestamp pts-C, the method further includes discarding the current frame FA(n) of the decoded first data stream and the current frame FB(n) of the decoded second data stream, and waiting for the next frame FA(n+1) of the decoded first data stream and the next frame FB(n+1) of the decoded second data stream.

[0097] If the second presentation timestamp pts-B is later than the third presentation timestamp pts-C, it indicates that a frame loss occurred in both the first and second data streams. When determining that the second presentation timestamp pts-B is later than the third presentation timestamp pts-C, the method also includes discarding the current frame FC(n) of the decoded third data stream and waiting for the next frame FC(n+1) of the decoded third data stream.

[0098] Repeat these processes until it is determined that the first presentation timestamp pts-A is the same as the second presentation timestamp pts-B and the third presentation timestamp pts-C. This indicates that the first data stream, the second data stream, and the third data stream have been frame-synchronized.

[0099] Figure 5 This illustrates the process of rendering a second image having a second resolution and a first image having a first resolution according to some embodiments of the present disclosure. See also Figure 5 In some embodiments, the method includes predefining a preset high-resolution region PDH having a width w and a length h. The center point of the preset high-resolution region PDH is defined by eye-tracking coordinates detected at a first time point t1. In some embodiments, the method further includes calculating predicted eye-tracking coordinates (x', y') at a second time point t2 via a content server, where t2 - t1 = Δt.

[0100] In some embodiments, the method includes determining the center point (x', y'), width w', and length h' of a dynamically expanded high-resolution region DHE based on (w, h, v, a, Δt), where w represents the width of a preset high-resolution region PDH, h represents the length of the preset high-resolution region PDH, v represents the instantaneous eye movement vector velocity at a first time point, a represents the instantaneous eye movement acceleration at the first time point, and Δt represents the difference between a second time point and the first time point. Optionally, the difference Δt between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device. In some embodiments, the coordinates of the four vertices of the dynamically expanded high-resolution region DHE can be represented as (x'-w' / 2, y'-h' / 2), (x'-w' / 2, y'+h' / 2), (x'+w' / 2, y'+h' / 2), and (x'+w' / 2, y'-h' / 2).

[0101] In some embodiments, the method further includes rendering a first image FLRI with a first resolution, and rendering a second image HRI with a second resolution based on the dynamic high-resolution region DHE.

[0102] Figure 6 This illustrates a process for cropping a second image with a second resolution based on updated real-time eye-tracking coordinates used to generate an updated image, according to some embodiments of this disclosure. (Reference) Figure 6 The updated real-time eye-tracking coordinates are represented as (x3, y3). In some embodiments, based on parameter data including (x', y', w, h, w', h'), the method further includes cropping a second image HRI with a second resolution to obtain an updated image UHRI, where x' and y' represent the coordinates of the center point of the dynamic high-resolution region, w represents the width of the preset high-resolution region, h represents the length of the preset high-resolution region, w' represents the width of the dynamic high-resolution region, and h' represents the length of the dynamic high-resolution region. Optionally, the four vertex coordinates of the updated image UHRI can be represented as (x3-w / 2, y3-h / 2), (x3-w / 2, y3+h / 2), (x3+w / 2, y3+h / 2), and (x3+w / 2, y3-h / 2). Since the dynamic high-resolution region DHE has been dynamically expanded, it covers and includes the region corresponding to the updated image UHRI. Therefore, the updated image UHRI can be obtained by cropping the dynamic high-resolution region DHE based on the four vertex coordinates of the updated image UHRI. Figure 6 As shown, in some embodiments, the method further includes combining the updated image UHRI and the first image FLRI with a first resolution to generate a composite image.

[0103] In some embodiments, combining the updated image UHRI and the first image FLRI with a first resolution includes expanding the pixels of the first image FLRI with the first resolution and rendering the first image FLRI with the first resolution at full resolution; and rendering the updated image UHRI at the positions based on the four vertex coordinates of the updated image UHRI.

[0104] In another aspect, this disclosure provides a display system. In some embodiments, the display system includes a content server and a terminal device. Figure 7 This is a schematic diagram illustrating a terminal device according to some embodiments of the present disclosure. Reference Figure 7 In some embodiments, the terminal device includes a processor 1002, a storage medium 1004, a display 1006, a communication module 1008, a database 1010, peripheral devices 1012, and a camera 1014. Certain devices may be omitted, and other devices may be included to better describe the relevant embodiments. The terminal device may include any suitable type of display panel, such as a plasma display panel, a liquid crystal display (LCD) panel, a touchscreen display panel, a projection display panel, a non-smart display panel, a smart display panel, etc. The terminal device may also include other computing systems, such as a personal computer (PC), a tablet computer, a mobile computer, or a smartphone. Furthermore, the terminal device may be any suitable content presentation device capable of presenting any suitable content. Users can interact with the terminal device to perform other activities of interest.

[0105] Processor 1002 may include any suitable one or more processors. Processor 1002 may include multiple cores for multithreaded or parallel processing. Processor 1002 may execute a sequence of computer program instructions to perform various processes. Storage medium 1004 may include memory modules (e.g., ROM, RAM, flash memory modules) and mass storage devices (e.g., CD-ROM and hard disk). Storage medium 1004 may store computer programs to implement various processes when the computer program is executed by processor 1002. For example, storage medium 1004 may store computer programs to implement various algorithms when the computer program is executed by processor 1002.

[0106] Furthermore, the communication module 1008 may include certain network interface devices for establishing connections via communication networks (e.g., TV cable networks, wireless networks, the Internet). The database 1010 may include one or more databases for storing data and performing operations on the stored data, such as database searches.

[0107] Display 1006 can provide information to the user. Display 1006 can include any suitable type of computer display device or electronic device display, such as an LCD or OLED-based device. Peripheral device 1012 can include various sensors or other I / O devices, such as a keyboard or mouse.

[0108] Examples of suitable terminal devices include, but are not limited to, electronic paper, mobile phones, tablet computers, televisions, monitors, laptops, digital photo albums, GPS devices, etc. Optionally, the terminal device is an organic light-emitting diode (OLED) display device. Optionally, the terminal device is a miniature OLED display device. Optionally, the terminal device is a miniature OLED display device.

[0109] In some embodiments, the terminal device includes a display panel; a first memory; and one or more first processors. Optionally, the first memory is connected to the one or more first processors; and the first memory stores computer-executable instructions for controlling the one or more first processors to perform the various tasks described herein.

[0110] In some embodiments, the content server includes a second memory; and one or more second processors. Optionally, the second memory is connected to the one or more second processors; and the second memory stores computer-executable instructions for controlling the one or more second processors to perform the various tasks described herein.

[0111] In some embodiments, the terminal device is configured to track a user's eye movements; obtain eye-tracking data; and transmit the eye-tracking data to the content server. In some embodiments, the content server is configured to: determine a dynamic high-resolution region based on the eye-tracking data; render a first image with a first resolution; render a second image with a second resolution based on the dynamic high-resolution region; encode the second image with the second resolution, the first image with the first resolution, and parameter data in three separate data streams; and transmit the three separate data streams from the content server to the terminal device for image display; optionally, the second resolution is greater than the first resolution.

[0112] In some embodiments, the terminal device is further configured to obtain updated real-time eye-tracking coordinates; crop the second image having the second resolution based on the updated real-time eye-tracking coordinates detected by the terminal device to generate an updated image; and combine the first image having the first resolution with the updated image to generate a composite image.

[0113] In some embodiments, the three separate data streams include a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding parameter data.

[0114] In some embodiments, the content server is configured to transmit the first data stream at a first bit rate; transmit the second data stream at a second bit rate; and transmit the third data stream at a third bit rate. Optionally, the third bit rate is higher than the first bit rate. Optionally, the first bit rate is higher than the second bit rate.

[0115] In some embodiments, the content server is further configured to predefine a preset high-resolution region; receive eye-tracking data including eye-tracking coordinates detected by the terminal device at a first time point; and calculate predicted eye-tracking coordinates at a second time point.

[0116] In some embodiments, the content server is configured to determine the dynamic high-resolution region based on: the width of a preset high-resolution region; the length of the preset high-resolution region; the instantaneous eye movement vector velocity of the eye movement at a first time point; the instantaneous eye movement acceleration of the eye movement at the first time point; and the difference between the second time point and the first time point. Optionally, the difference between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device. Optionally, the predicted eye-tracking coordinates are the center point of the dynamic high-resolution region at the second time point. Optionally, the center point, width, and length of the dynamic high-resolution region dynamically change over time based on the changes in the instantaneous eye movement vector velocity and instantaneous eye movement acceleration.

[0117] In some embodiments, the terminal device is further configured to perform frame synchronization on the first data stream encoding the second image having the second resolution, the second data stream encoding the first image having the first resolution, and the third data stream encoding the parameter data.

[0118] In some embodiments, the terminal device is further configured to extract a first presentation timestamp from a decoded first data stream; extract a second presentation timestamp from a decoded second data stream; extract a third presentation timestamp from a decoded third data stream; and determine whether the first presentation timestamp, the second presentation timestamp, and the third presentation timestamp are the same.

[0119] In some embodiments, the terminal device is further configured to: discard the current frame of the decoded first data stream and wait for the next frame of the decoded first data stream when it is determined that the first presentation timestamp is earlier than the second presentation timestamp; or discard the current frame of the decoded second data stream and wait for the next frame of the decoded second data stream when it is determined that the first presentation timestamp is later than the second presentation timestamp.

[0120] In some embodiments, the terminal device is further configured to: when it is determined that the first presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when it is determined that the first presentation timestamp is later than the third presentation timestamp, discard the current frame of the decoded third data stream, and wait for the next frame of the decoded third data stream.

[0121] In some embodiments, the terminal device is further configured to, when determining that the second presentation timestamp is earlier than the third presentation timestamp, discard the current frame of the decoded first data stream and the current frame of the decoded second data stream, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or when determining that the second presentation timestamp is later than the third presentation timestamp, discard the current frame of the decoded third data stream, and wait for the next frame of the decoded third data stream.

[0122] In some embodiments, the content server is configured to generate the parameter data based on the dynamic high-resolution region. Optionally, lossless compression is used to transmit the parameter data from the content server to the terminal device.

[0123] In some embodiments, the content server is configured to transmit the coordinates of the center point of the dynamic high-resolution region to the terminal device; the width of the preset high-resolution region; the length of the preset high-resolution region; the width of the dynamic high-resolution region; the length of the dynamic high-resolution region; and a presentation timestamp.

[0124] In some embodiments, the content server is configured to render the second image having the second resolution based at least in part on the coordinates of the center point of the dynamic high-resolution region, the width of the dynamic high-resolution region, and the length of the dynamic high-resolution region.

[0125] In some embodiments, the content server is configured to perform horizontal and vertical compression on a first image having a first resolution.

[0126] In some embodiments, the terminal device is also configured to display a composite image.

[0127] In another aspect, this disclosure provides a computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon. In some embodiments, the computer-readable instructions may be executed by one or more first processors to cause the one or more first processors to perform: tracking a user's eye movements and obtaining eye-tracking data; and transmitting the eye-tracking data from a terminal device to a content server. In some embodiments, the computer-readable instructions may be executed by one or more second processors to cause the one or more second processors to perform: determining a dynamic high-resolution region based on the eye-tracking data; rendering a first image having a first resolution; rendering a second image having a second resolution based on the dynamic high-resolution region; encoding the second image having the second resolution, the first image having the first resolution, and parameter data in three separate data streams; and transmitting the three separate data streams from the content server to the terminal device for image display. Optionally, the second resolution is greater than the first resolution.

[0128] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform: obtaining updated real-time eye-tracking coordinates; cropping the second image having the second resolution based on the updated real-time eye-tracking coordinates detected by the terminal device to generate an updated image; and combining the first image having the first resolution with the updated image to generate a composite image.

[0129] In some embodiments, the three separate data streams include a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding parameter data.

[0130] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to: transmit the first data stream at a first bit rate; transmit the second data stream at a second bit rate; and transmit the third data stream at a third bit rate. Optionally, the third bit rate is higher than the first bit rate. Optionally, the first bit rate is higher than the second bit rate.

[0131] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to perform: predefine a preset high-resolution region; receive eye-tracking data including eye-tracking coordinates detected by the terminal device at a first time point; and calculate predicted eye-tracking coordinates at a second time point.

[0132] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to: determine the dynamic high-resolution region based on: the width of a preset high-resolution region; the length of the preset high-resolution region; the instantaneous eye movement vector velocity of the eye movement at the first time point; the instantaneous eye movement acceleration of the eye movement at the first time point; and the difference between the second time point and the first time point. Optionally, the difference between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device.

[0133] In some embodiments, the predicted eye-tracking coordinates are the center point of the dynamic high-resolution region at the second time point. Optionally, the center point, width, and length of the dynamic high-resolution region dynamically change over time based on the changes in instantaneous eye movement vector velocity and instantaneous eye movement acceleration over time.

[0134] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform frame synchronization on the first data stream encoding the second image having the second resolution, the second data stream encoding the first image having the first resolution, and the third data stream encoding the parameter data.

[0135] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform: extracting a first presentation timestamp from a decoded first data stream; extracting a second presentation timestamp from a decoded second data stream via the terminal device; extracting a third presentation timestamp from a decoded third data stream via the terminal device; and determining whether the first presentation timestamp, the second presentation timestamp, and the third presentation timestamp are the same.

[0136] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to: discard the current frame of the decoded first data stream and wait for the next frame of the decoded first data stream when it is determined that the first presentation timestamp is earlier than the second presentation timestamp; or discard the current frame of the decoded second data stream and wait for the next frame of the decoded second data stream when it is determined that the first presentation timestamp is later than the second presentation timestamp.

[0137] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to: discard the current frame of the decoded first data stream and the current frame of the decoded second data stream when it is determined that the first presentation timestamp is earlier than the third presentation timestamp, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or discard the current frame of the decoded third data stream and wait for the next frame of the decoded third data stream when it is determined that the first presentation timestamp is later than the third presentation timestamp.

[0138] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to: discard the current frame of the decoded first data stream and the current frame of the decoded second data stream when it is determined that the second presentation timestamp is earlier than the third presentation timestamp, and wait for the next frame of the decoded first data stream and the next frame of the decoded second data stream; or discard the current frame of the decoded third data stream and wait for the next frame of the decoded third data stream when it is determined that the second presentation timestamp is later than the third presentation timestamp.

[0139] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to generate the parameter data based on the dynamic high-resolution region. Optionally, lossless compression may be used to transfer the parameter data from the content server to the terminal device.

[0140] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to perform: transmitting to a terminal device the coordinates of the center point of the dynamic high-resolution region; the width of the preset high-resolution region; the length of the preset high-resolution region; the width of the dynamic high-resolution region; the length of the dynamic high-resolution region; and presenting a timestamp.

[0141] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to: render the second image having the second resolution based at least in part on the coordinates of the center point of the dynamic high-resolution region, the width of the dynamic high-resolution region, and the length of the dynamic high-resolution region.

[0142] In some embodiments, the computer-readable instructions may be executed by one or more second processors to further cause the one or more second processors to perform: horizontal compression and vertical compression on the first image having the first resolution.

[0143] In some embodiments, the computer-readable instructions may be executed by one or more first processors to further cause the one or more first processors to perform: causing the display panel to display the composite image.

[0144] For illustrative and descriptive purposes, the foregoing description of embodiments of the invention has been provided. It is not exhaustive, nor is it intended to limit the invention to the precise forms or exemplary embodiments disclosed. Therefore, the foregoing description should be considered illustrative rather than restrictive. Clearly, many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described to explain the principles of the invention and its best mode of practical application, thereby enabling those skilled in the art to understand the various embodiments of the invention and the various modifications suitable for the particular use or implementation contemplated. The scope of the invention is intended to be defined by the appended claims and their equivalents, wherein, unless otherwise stated, all terms are to be interpreted in their broadest reasonable sense. Therefore, the terms “the invention,” “the present invention,” etc., do not necessarily limit the scope of the claims to the specific embodiments, and references to exemplary embodiments of the invention do not imply limitation of the invention, nor should such limitation be inferred. The invention is defined only by the spirit and scope of the appended claims. Furthermore, these claims may involve the use of “first,” “second,” etc., followed by nouns or elements. These terms should be understood as nomenclature and should not be construed as limiting the number of elements modified by these nomenclatures unless a specific number has been given. Any advantages and benefits described may not apply to all embodiments of the invention. It should be understood that changes to the described embodiments can be made by those skilled in the art without departing from the scope of the invention as defined by the appended claims. Furthermore, the elements and components in this disclosure are not intended for public distribution, whether or not they are expressly recited in the appended claims.

Claims

1. An image display method, comprising: The system tracks the user's eye movements and obtains eye-tracking data through a terminal device. The eye-tracking data is transmitted from the terminal device to the content server; The content server uses the eye-tracking data to determine the dynamic high-resolution region of the original image. The content server renders a first image with a first resolution, wherein the first image is a full low-resolution rendering of the original image. The content server renders a second image with a second resolution, based at least in part on the coordinates of the center point of the dynamic high-resolution region, the width of the dynamic high-resolution region, and the length of the dynamic high-resolution region, wherein the second image is a high-resolution rendering of a region of the original image. The content server encodes the second image with the second resolution, the first image with the first resolution, and parameter data in the data stream; and The data stream is transmitted from the content server to the terminal device for image display; Wherein, the second resolution is greater than the first resolution; The method further includes: The content server receives eye-tracking data, including eye-tracking coordinates detected by the terminal device at a first time point, and determines a preset high-resolution region based on the eye-tracking coordinates detected at the first time point; and The content server calculates the predicted eye-tracking coordinates at a second time point, and determines the dynamic high-resolution region based on the predicted eye-tracking coordinates at the second time point; the difference between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device. Specifically, the center point, length, and width of the dynamic high-resolution region are determined based on the width of the preset high-resolution region, the length of the preset high-resolution region, the instantaneous eye movement vector velocity of the eye movement at the first time point, the instantaneous eye movement acceleration of the eye movement at the first time point, and the difference between the second time point and the first time point; the predicted eye tracking coordinates are the center point of the dynamic high-resolution region at the second time point. The method further includes: The terminal device is used to obtain updated real-time eye-tracking coordinates. Using the terminal device, based on the updated real-time eye-tracking coordinates detected by the terminal device, and the length and width of a preset high-resolution region, the coordinates of the four vertices of the updated image are determined. Based on the coordinates of the four vertices of the updated image, the second image with the second resolution is cropped, thereby generating the updated image; and The terminal device combines the first image with the updated image, which has the first resolution, to generate a composite image.

2. The method of claim 1, wherein, The data stream includes a first data stream encoding the second image having the second resolution, a second data stream encoding the first image having the first resolution, and a third data stream encoding the parameter data.

3. The method of claim 2, wherein, Transmitting the data stream from the content server to the terminal device includes: The first data stream is transmitted at the first bit rate; The second data stream is transmitted at a second bit rate; and The third data stream is transmitted at a third bit rate; Wherein, the third bit rate is higher than the first bit rate; and The first bit rate is higher than the second bit rate.

4. The method of claim 1, wherein, Based on the changes in instantaneous eye movement vector velocity and instantaneous eye movement acceleration over time, the center point, width, and length of the dynamic high-resolution region change dynamically over time.

5. The method according to claim 2, further comprising, through the terminal device, performing frame synchronization on the first data stream encoding the second image having the second resolution, the second data stream encoding the first image having the first resolution, and the third data stream encoding the parameter data.

6. The method according to claim 2, further comprising: The terminal device extracts the first presentation timestamp from the decoded first data stream. The terminal device extracts a second presentation timestamp from the decoded second data stream. The terminal device extracts a third presentation timestamp from the decoded third data stream; as well as Determine whether the first presentation timestamp, the second presentation timestamp, and the third presentation timestamp are the same.

7. The method according to claim 6, further comprising: When it is determined that the first presentation timestamp is earlier than the second presentation timestamp, the current frame of the decoded first data stream is discarded, and the system waits for the next frame of the decoded first data stream. or When it is determined that the first presentation timestamp is later than the second presentation timestamp, the current frame of the decoded second data stream is discarded, and the system waits for the next frame of the decoded second data stream.

8. The method according to claim 7, further comprising: When it is determined that the first presentation timestamp is earlier than the third presentation timestamp, the current frame of the decoded first data stream and the current frame of the decoded second data stream are discarded, and the next frame of the decoded first data stream and the next frame of the decoded second data stream are waited for. or When it is determined that the first presentation timestamp is later than the third presentation timestamp, the current frame of the decoded third data stream is discarded, and the system waits for the next frame of the decoded third data stream.

9. The method according to claim 7, further comprising: When it is determined that the second presentation timestamp is earlier than the third presentation timestamp, the current frame of the decoded first data stream and the current frame of the decoded second data stream are discarded, and the next frame of the decoded first data stream and the next frame of the decoded second data stream are waited for. or When it is determined that the second presentation timestamp is later than the third presentation timestamp, the current frame of the decoded third data stream is discarded, and the system waits for the next frame of the decoded third data stream.

10. The method of claim 1, further comprising generating the parameter data based on the dynamic high-resolution region via the content server.

11. The method of claim 10, wherein, Transmitting a third data stream encoding the parameter data from the content server to the terminal device includes: transmitting the coordinates of the center point of the dynamic high-resolution region; the width of the preset high-resolution region; the length of the preset high-resolution region; the width of the dynamic high-resolution region; the length of the dynamic high-resolution region; and a presentation timestamp.

12. The method of claim 1, wherein, Rendering the first image having the first resolution includes performing horizontal compression and performing vertical compression.

13. The method of claim 1, wherein, The updated real-time eye-tracking coordinates detected by the terminal device are represented as (x3, y3); The coordinates of the four vertices of the updated image are represented as (x3 - w / 2, y3 - h / 2), (x3 - w / 2, y3 + h / 2), (x3 + w / 2, y3 + h / 2), and (x3 + w / 2, y3 - h / 2). w represents the width of the preset high-resolution region; as well as h represents the length of the preset high-resolution region.

14. A display system, comprising a content server and terminal devices; wherein, The terminal device is configured as follows: Track the user's eye movement; Obtain eye-tracking data; The eye-tracking data is transmitted to the content server; The content server is configured as follows: Based on the eye-tracking data, the dynamic high-resolution region of the original image is determined; Render a first image having a first resolution, wherein the first image is a full low-resolution rendering of the original image; A second image with a second resolution is rendered based at least in part on the coordinates of the center point of the dynamic high-resolution region, the width of the dynamic high-resolution region, and the length of the dynamic high-resolution region, wherein the second image is a high-resolution rendering of a region of the original image; Encode in the data stream the second image having the second resolution, the first image having the first resolution, and parameter data; and The data stream is transmitted from the content server to the terminal device for image display; Wherein, the second resolution is greater than the first resolution; The content server is also configured as follows: Receive eye-tracking data including eye-tracking coordinates detected by the terminal device at a first time point, and determine a preset high-resolution region based on the eye-tracking coordinates detected at the first time point; and The predicted eye-tracking coordinates at the second time point are calculated, and the dynamic high-resolution region is determined based on the predicted eye-tracking coordinates at the second time point; the difference between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device; Specifically, the center point, length, and width of the dynamic high-resolution region are determined based on the width of the preset high-resolution region, the length of the preset high-resolution region, the instantaneous eye movement vector velocity of the eye movement at the first time point, the instantaneous eye movement acceleration of the eye movement at the first time point, and the difference between the second time point and the first time point; the predicted eye tracking coordinates are the center point of the dynamic high-resolution region at the second time point. The terminal device is also configured to: Obtain updated real-time eye-tracking coordinates; Based on the updated real-time eye-tracking coordinates detected by the terminal device, and the length and width of a preset high-resolution region, the coordinates of the four vertices of the updated image are determined. Based on the coordinates of the four vertices of the updated image, the second image with the second resolution is cropped, thereby generating the updated image; and The first image with the first resolution is combined with the updated image to generate a composite image.

15. A computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon. The computer-readable instructions may be executed by one or more first processors to cause one or more first processors to perform the following operations: Tracking user eye movements and obtaining eye-tracking data; and The eye-tracking data is transmitted from the terminal device to the content server; The computer-readable instructions may be executed by one or more second processors to cause one or more second processors to perform the following operations: Based on the eye-tracking data, the dynamic high-resolution region of the original image is determined; Render a first image with a first resolution, wherein... The first image is a full low-resolution rendering of the original image; A second image with a second resolution is rendered based at least in part on the coordinates of the center point of the dynamic high-resolution region, the width of the dynamic high-resolution region, and the length of the dynamic high-resolution region, wherein the second image is a high-resolution rendering of a region of the original image; Encode in the data stream the second image having the second resolution, the first image having the first resolution, and parameter data; and The data stream is transmitted from the content server to the terminal device for image display; Wherein, the second resolution is greater than the first resolution; The computer-readable instructions may be executed by one or more second processors to cause one or more second processors to perform the following operations: Receive eye-tracking data including eye-tracking coordinates detected by the terminal device at a first time point, and determine a preset high-resolution region based on the eye-tracking coordinates detected at the first time point; and The predicted eye-tracking coordinates at the second time point are calculated, and the dynamic high-resolution region is determined based on the predicted eye-tracking coordinates at the second time point; the difference between the second time point and the first time point is substantially the same as the round-trip time between the content server and the terminal device; Specifically, the center point, length, and width of the dynamic high-resolution region are determined based on the width of the preset high-resolution region, the length of the preset high-resolution region, the instantaneous eye movement vector velocity of the eye movement at the first time point, the instantaneous eye movement acceleration of the eye movement at the first time point, and the difference between the second time point and the first time point; the predicted eye tracking coordinates are the center point of the dynamic high-resolution region at the second time point. The computer-readable instructions may be executed by one or more first processors to cause one or more first processors to perform the following operations: Obtain updated real-time eye-tracking coordinates; Based on the updated real-time eye-tracking coordinates detected by the terminal device, and the length and width of a preset high-resolution region, the coordinates of the four vertices of the updated image are determined. Based on the coordinates of the four vertices of the updated image, the second image with the second resolution is cropped, thereby generating the updated image; and The first image with the first resolution is combined with the updated image to generate a composite image.

Images