Display device, display method, and computer program product

Abstract

A display device is provided. The display device includes: a memory; and one or more processors. The memory and the one or more processors are connected to each other. The memory stores computer executable instructions to control the one or more processors to: identify image display scene information; scene complexity detection is executed based on the image display scene information; grouping the scene complexity of the image display scene into a plurality of levels based on the type of the image display scene; when it is determined that the scene complexity is at the first level, rendering the image in a first resolution range; and when it is determined that the scene complexity is at the second level, rendering the image under the second resolution range. The second resolution range is lower than the first resolution range. The scene complexity of the second level is higher than the scene complexity of the first level.

Description

Technical Field

[0001] The present invention relates to display technology, and more particularly, to a display device, a display method and a computer program product. Background Art

[0002] Virtual reality and augmented reality are two computer technologies used to view synthetic or partially synthetic environments. Virtual reality generally involves computer-generated representations of various real-world or virtual environments. Augmented reality generally involves various types of computer-assisted representations of the real world. Summary of the invention

[0003] In one aspect, the present disclosure provides a display device, comprising: a memory; and one or more processors; wherein the memory and the one or more processors are connected to each other; and the memory stores computer-executable instructions to control the one or more processors to: identify image display scene information; perform scene complexity detection based on the image display scene information; group the scene complexity of the image display scene into multiple levels based on the type of the image display scene; when it is determined that the scene complexity is at a first level, render the image at a first resolution range; and when it is determined that the scene complexity is at a second level, render the image at a second resolution range; wherein the second resolution range is lower than the first resolution range; and the scene complexity of the second level is higher than the scene complexity of the first level.

[0004] Optionally, the memory also stores computer executable instructions to control the one or more processors to: render a first sub-image and a second sub-image; wherein the first sub-image is in an area that intersects with the user's gaze direction; the second sub-image is in an area that does not intersect with the gaze direction; the first sub-image has a higher resolution than a resolution of the second sub-image; and when it is determined that the scene complexity is at the first level, render the first sub-image within the first resolution range, and when it is determined that the scene complexity is at the second level, render the first sub-image within the second resolution range.

[0005] Optionally, when rendering the image at the first resolution range, the memory also stores computer executable instructions to control the one or more processors to: monitor the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements; and render the image at the second resolution range when it is determined that the current system performance fails to meet the rendering requirements.

[0006] Optionally, the memory also stores computer executable instructions to control the one or more processors to: render an image at a third resolution range when it is determined that the scene complexity is at a third level; wherein the third resolution range is lower than the second resolution range; and the scene complexity at the third level is higher than the scene complexity at the second level.

[0007] Optionally, the memory further stores computer executable instructions to control the one or more processors to: control the frame rate to be higher than a threshold regardless of rendering the image in the first resolution range, the second resolution range or the third resolution range.

[0008] Optionally, when rendering the image at the second resolution range, the memory also stores computer executable instructions to control the one or more processors to: monitor the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements; and render the image at a third resolution range when it is determined that the current system performance fails to meet the rendering requirements; wherein the third resolution range is lower than the second resolution range.

[0009] Optionally, the memory further stores computer executable instructions to control the one or more processors to: start one or more applications; initialize image display scene information; and identify the image display scene information.

[0010] Optionally, the display device also includes a gaze point sensor, which is configured to detect the user's gaze; wherein the memory also stores computer executable instructions to control the one or more processors to: turn off the gaze point sensor when it is determined that the image display scene is the first scene.

[0011] Optionally, the memory further stores computer executable instructions to control the one or more processors to: monitor the power level of a power supply of the display device.

[0012] Optionally, the memory also stores computer executable instructions to control the one or more processors to: render an image at a fourth resolution range when it is determined that the power level of the power supply is greater than a threshold power level; and render an image at a fifth resolution range when it is determined that the power level of the power supply is less than or equal to the threshold power level; wherein the fourth resolution range is greater than the fifth resolution range.

[0013] Optionally, the display device also includes an acceleration sensor, which is configured to detect the user's head movement; wherein the memory also stores computer-executable instructions to control the one or more processors to: when it is determined that the image display scene is the second scene, turn on the acceleration sensor to monitor the user's head movement.

[0014] Optionally, the memory also stores computer executable instructions to control the one or more processors to: turn off the gaze point sensor and render the image at a sixth resolution range when it is determined that the speed of the head movement is greater than a first threshold speed; and turn on the gaze point sensor when it is determined that the speed of the head movement is less than or equal to the first threshold speed; and monitor the speed of eye movement.

[0015] Optionally, the memory also stores computer executable instructions to control the one or more processors to: render the image at a seventh resolution range and control the gaze point sensor to monitor the speed of the eye movement in a first speed exposure mode when it is determined that the speed of the eye movement is greater than a second threshold speed; and control the gaze point sensor to monitor the speed of the eye movement in a second speed exposure mode when it is determined that the speed of the eye movement is less than or equal to a second threshold speed; wherein, in the first speed exposure mode, the gaze point sensor is configured to monitor the speed of the eye movement at a first frequency; in the second speed exposure mode, the gaze point sensor is configured to monitor the speed of the eye movement at a second frequency; and the second frequency is greater than the first frequency.

[0016] Optionally, the memory further stores computer executable instructions to control the one or more processors to: perform the scene complexity detection when it is determined that the speed of the eye movement is less than or equal to a second threshold speed.

[0017] Optionally, the memory also stores computer executable instructions to control the one or more processors to: generate a synchronization signal to synchronize the operation of applications, underlying components, input hardware, and output hardware of the display device, wherein the underlying components include an underlying interface module; a display driver module configured to drive a display panel; a sensor driver module configured to drive a camera sensor; and a light-emitting element driver module configured to drive a fill light driver circuit; send the synchronization signal to a synchronization signal distribution thread; and distribute the synchronization signal to one or more applications by the synchronization signal distribution thread.

[0018] Optionally, the memory also stores computer executable instructions to control the one or more processors to: send the synchronization signal to the underlying interface module by the one or more applications; and send the synchronization signal to the underlying component driver module of the underlying component by the underlying interface module.

[0019] Optionally, the display device also includes the camera sensor, the fill light driving circuit and the display panel; wherein the memory also stores computer executable instructions to control the one or more processors to: distribute the synchronization signal to the sensor synchronization control thread, the light-emitting element synchronization control thread and the display panel synchronization control thread by the synchronization signal distribution thread; send the synchronization signal to the underlying interface module through the sensor synchronization control thread, the light-emitting element synchronization control thread and the display panel synchronization control thread; and send the synchronization signal to the display driving module configured to drive the display panel, the sensor driving module configured to drive the camera sensor, and the light-emitting element driving module configured to drive the fill light driving circuit by the underlying interface module.

[0020] Optionally, the display device includes a clock management module, an image rendering module configured to render an image, and a synchronization signal distribution thread; wherein, when the image is rendered, the image rendering module is configured to send a signal to the clock management module; and when receiving the signal, the clock management module is configured to generate a synchronization signal, and is configured to send the synchronization signal to the synchronization signal distribution thread.

[0021] Optionally, the display device also includes a fill light driving circuit and a variable frequency driving module, and the variable frequency driving module is configured to modulate the pulse width modulation timing of the fill light driving circuit; wherein the variable frequency driving module is configured to modulate the pulses used for the fill light driving circuit to have a first pulse frequency in a first driving mode and a second pulse frequency in a second driving mode; the second pulse frequency is greater than the first pulse frequency; the first driving mode corresponds to a first speed exposure mode of a gaze point sensor configured to monitor the speed of eye movement; the second driving mode corresponds to a second speed exposure mode of the gaze point sensor; and in the first speed exposure mode, the gaze point sensor is configured to monitor the speed of eye movement at a first frequency; in the second speed exposure mode, the gaze point sensor is configured to monitor the speed of eye movement at a second frequency; and the second frequency is greater than the first frequency.

[0022] On the other hand, the present disclosure provides a display method, including: identifying image display scene information; performing scene complexity detection based on the image display scene information; grouping the scene complexity of the image display scene into multiple levels based on the type of the image display scene; when it is determined that the scene complexity is at a first level, rendering the image at a first resolution range; and when it is determined that the scene complexity is at a second level, rendering the image at a second resolution range; wherein the second resolution range is lower than the first resolution range; and the scene complexity of the second level is higher than the scene complexity of the first level.

[0023] On the other hand, the present disclosure provides a computer program product, comprising a non-temporary tangible computer-readable medium having computer-readable instructions thereon, wherein the computer-readable instructions can be executed by a processor to cause the processor to perform: identifying image display scene information; performing scene complexity detection based on the image display scene information; grouping the scene complexity of the image display scene into multiple levels based on the type of the image display scene; rendering the image at a first resolution range when it is determined that the scene complexity is at a first level; and rendering the image at a second resolution range when it is determined that the scene complexity is at a second level; wherein the second resolution range is lower than the first resolution range; and the scene complexity of the second level is higher than the scene complexity of the first level. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] According to various disclosed embodiments, the following drawings are examples only for illustration purposes and are not intended to limit the scope of the present invention.

[0025] Figure 1 is a schematic diagram showing the structure of a display device in some embodiments of the present disclosure.

[0026] Figure 2 A schematic diagram showing the structure of a control module of a display device according to some embodiments of the present disclosure.

[0027] Figure 3 is a schematic diagram illustrating the functions of a control module of a display device in some embodiments of the present disclosure.

[0028] Figure 4 is a schematic diagram illustrating the functions of a control module of a display device in some embodiments of the present disclosure.

[0029] Figure 5 The interaction among applications, underlying components, input hardware, and output hardware in a display device according to some embodiments of the present disclosure is shown.

[0030] Figure 6The synchronous operation performed by the synchronous signal in the display device according to some embodiments of the present disclosure is shown.

[0031] Figure 7 The invention shows a process of information output in a display device according to some embodiments of the present disclosure.

[0032] Figure 8 The process of determining the first speed exposure mode or the second speed exposure mode in some embodiments according to the present disclosure is shown.

[0033] Fig. 9 is a schematic diagram showing a display device according to some embodiments of the present disclosure.

[0034] Fig.10 is a flowchart illustrating a display method in some embodiments according to the present disclosure.

[0035] Fig.11 is a flowchart illustrating a display method in some embodiments according to the present disclosure. DETAILED DESCRIPTION

[0036] The present disclosure will now be described in more detail with reference to the following examples. It should be noted that the following description of some of the embodiments presented herein is for illustration and description purposes only. It is not intended to be exhaustive or limited to the precise form disclosed.

[0037] Related virtual reality head-mounted display (HMD) devices typically integrate all of their components into a helmet. The increased image display resolution in related virtual reality head-mounted display devices enhances the virtual reality experience, but it also increases the overall system load and power consumption. Power consumption has always been an important issue in virtual reality displays, and the enhancement of display quality with 4K or 8K resolution exacerbates this problem, further affecting the already limited battery life of these products. Approaches such as increasing battery capacity or adding plug-in power modules typically result in increased device size and weight, which is counterproductive to the goal of making VR products lighter and more compact.

[0038] Therefore, the present disclosure provides, among other things, a display device, a display method, and a computer program product, which substantially eliminate one or more problems caused by the limitations and disadvantages of the prior art. In one aspect, the present disclosure provides a display device. In some embodiments, the display device includes a memory; and one or more processors. The memory and the one or more processors are connected to each other. The memory stores computer executable instructions to control one or more processors to: identify image display scene information; perform scene complexity detection based on the image display scene information; group the scene complexity of the image display scene into multiple levels based on the type of the image display scene; when it is determined that the scene complexity is at a first level, render the image at a first resolution range; and when it is determined that the scene complexity is at a second level, render the image at a second resolution range. Optionally, the second resolution range is lower than the first resolution range. Optionally, the scene complexity of the second level is higher than the scene complexity of the first level.

[0039] Figure 1 Schematic diagram showing the structure of a display device in some embodiments of the present disclosure. Figure 1 In some embodiments, the display device includes a control module, a power module, and a display module. In the present display device, the components of the display device are configured to be worn by different parts of the human body including the head and the neck. The display module is configured to be worn by the user's head for viewing images. The power module and the control module are configured to be worn by the user's neck. The structure of the display device results in a reduction in head weight. In some embodiments, the display module includes a display panel and a power circuit, an acceleration sensor and its peripheral circuits, a camera and its peripheral circuits. In some embodiments, the power module includes a battery, a charger and its peripheral circuits.

[0040] Figure 2 A schematic diagram showing the structure of a control module of a display device according to some embodiments of the present disclosure. Figure 2In some embodiments, the control module includes one or more processors and peripheral circuits thereof, interfaces and peripheral circuits for connecting the display module, control circuits and interfaces for peripheral devices (such as WiFi / Bluetooth, cameras, buttons, touch, fans, USB, audio, etc.). For example, the control module includes a USB2.0 interface, a USB3.0 interface, a Type-C interface, a power interface, one or more interfaces for touch (e.g., Gpio and I2C), an I2C interface for a P sensor, and an I2S interface for an audio IC connected to an audio system (e.g., a microphone MIC and a speaker SPK). The control module may also include a PCIE2.0 SDIO interface for a wireless network integrated circuit WIFIIC and a Bluetooth integrated circuit BT IC, an eMMC interface for an embedded multimedia card, a DDR interface for a random access memory, a MIPI CSI interface for a camera, a MIPIDS, a MIPIDSI interface for a display panel, and a serial peripheral interface SPI.

[0041] In some embodiments, the present display device is configured to provide an intelligent modulation mechanism that takes into account both user needs and usage scenario requirements, covering aspects such as human vision, hand movements, head movements, and different activities such as movie watching, gaming, and teaching.

[0042] In some embodiments, the control module (e.g., one or more processors in the control module) is configured to adjust (e.g., automatically adjust) the display resolution based on the requirement of balancing image quality and performance (display resolution modulation). In some embodiments, the control module is configured to adjust the frame rate of the display device as needed to ensure a balance between smoothness and performance (frame rate modulation). In some embodiments, the control module is configured to adjust the display resolution and / or frame rate according to the real-time performance of the display device, including CPU and GPU performance (system performance modulation). When the performance of the system can fully meet the current resolution and frame rate requirements, the resolution and / or frame rate can be adjusted (e.g., increased) to achieve a better user experience. Conversely, when the performance of the system does not meet the requirements, the display resolution can be reduced to maintain the frame rate. Display resolution modulation, frame rate modulation, and system performance modulation are performed in collaboration with each other.

[0043] In some embodiments, display resolution modulation, frame rate modulation, and system performance modulation are performed interactively with peripheral devices, including sensors such as camera sensors, accelerometers, and infrared sensors. The main goal is to save power, which is achieved by reducing the workload of the system, including CPU and GPU loads. By having this mechanism, the control module is configured to establish a multi-dimensional intelligent detection and determination mechanism, which is comprehensively determined based on real-time system performance, data captured by sensors, application scenarios, etc. It dynamically adjusts the operating strategy of the system. The sensor data includes accelerometer data for head movement and camera sensor data for eye movement.

[0044] For example, during a game application that requires high frame rate and resolution, the display device is configured to display images based on the user's gaze direction to reduce GPU and CPU power consumption. For example, the display device is configured to render and process images based on the user's gaze direction (e.g., gaze point display). In some embodiments, the display device (e.g., one or more processors in a control module) is configured to track and monitor the user's eye movements and gaze direction to determine where they are looking on the screen or in their field of vision. This information is then used to control and interact with digital interfaces, applications, or devices. Gaze point displays are commonly used for various purposes, including user interface navigation, games, accessibility, and research. When a user is looking at a specific area of ​​the screen, the gaze point display can give priority to rendering and processing resources for that specific area. This means that the area of ​​the screen outside the user's gaze can be rendered at a lower resolution or with less detail, referred to as a peripheral area or non-focused area.

[0045] In some embodiments, when the head movement speed exceeds a certain threshold, the display device is configured to display images in a low-resolution mode to further reduce the system load. Therefore, the present display device provides a multi-dimensional intelligent detection and determination mechanism to make a comprehensive determination based on the real-time operating status of the performance of the display device, the data captured by the sensor, and the application scenario information. These determinations are used to dynamically adjust the operating strategy of the system. The sensor data includes acceleration sensor data for capturing head movement and camera sensor data for capturing eye movement.

[0046] In some embodiments, the intelligent detection and determination mechanism monitors the type of application started by the user, such as a game application or a video viewing application. For game applications, the frame rate and resolution requirements are relatively high, thereby requiring higher performance. However, higher performance also increases power consumption. Therefore, the intelligent detection and determination mechanism continues to adjust based on other mechanisms (e.g., head movement) to reduce power consumption. For example, when the speed of head movement is less than 30 degrees per second, the display device is configured to render and process images based on the user's gaze direction (e.g., gaze point display mode) to reduce GPU and CPU power consumption. In another example, when the speed of head movement exceeds 30 degrees per second, the display device is configured to display images in a lower resolution mode to further reduce system load.

[0047] In some embodiments, when the speed of the head movement is less than 30 degrees per second, the intelligent detection and determination mechanism further checks to achieve a better match between system performance and application operation. In one example, the intelligent detection and determination mechanism first checks the complexity of the scene, classifying it into multiple levels (e.g., simple level: 1; medium complexity level: 2; and heavy load level: 3), each level having a corresponding threshold. These thresholds correspond to different resolutions. When the complexity is low, a slightly higher resolution mode can be used to enhance the experience. When the complexity is high, in order to save power and avoid a significant increase in power consumption due to system overload, a lower resolution mode can be switched to achieve an overall performance balance.

[0048] In some embodiments, scene complexity is detected based on the detection of the complexity of the scene model. In some embodiments, the complexity of the scene model can be understood as the number of vertices and polygons used when creating the scene. The more vertices and polygons a scene contains, the higher its model complexity and scene complexity are, which can lead to an increase in the rendering workload of the GPU. For example, in a game application or a 360-degree panoramic application, the scene can vary in terms of model complexity and scene complexity. The entire application can include complex scenes (such as buildings and dynamic characters) and low-complexity scenes (such as backgrounds and natural landscapes). Based on different processor platforms, it is necessary to adjust the scene complexity.

[0049] In the context of computer graphics and image rendering, "scene complexity" refers to the level of complexity, detail level, and computational requirements of a specific scene or environment that needs to be rendered on a display or in a virtual world. It includes various factors, such as the number of objects, their shapes, textures, lighting conditions, and the interactions between them. The term "model complexity" refers to the complexity, detail level, and perfection of a model (e.g., a 3D model) that represents a scene in a digital environment. Model complexity includes various aspects that define the richness and realism of the model. Some key factors that affect model complexity include geometric complexity, texture complexity, lighting complexity, physical and dynamic complexity, detail level, transparency and overlapping objects, shadow complexity, and rendering technology. Some key factors that affect model complexity include polygon count, vertex count, texture mapping, normal mapping, rigging and animation, detail level, material complexity, physical simulation, facial expressions, and interactivity.

[0050] In some embodiments, the intelligent detection and determination mechanism according to the present disclosure utilizes multiple threshold levels (e.g., threshold levels 1, 2, and 3) to determine the complexity of the scene. Each threshold level corresponds to a specific range of vertex and polygon counts. When the scene complexity is below threshold 1, it is classified as a simple scene. When the scene complexity is between 1 and 2, it is considered to be a moderately complex scene. When the scene complexity exceeds 3, it is classified as a heavy-loaded scene. Applications can then adjust their own resolutions based on these threshold levels. Reducing the rendering resolution directly reduces the number of vertices and polygons in the scene, thereby reducing the scene complexity.

[0051] In some embodiments, scene complexity is detected based on a detection of the model complexity of the scene. In some embodiments, the intelligent detection and determination mechanism utilizes a ray collision detection mechanism to determine model complexity, especially in production applications where the developer may not have prior knowledge of the model complexity of the application. During GPU rendering, the ray collision detection mechanism simulates the interaction of light with the model, evaluating the number of vertices and polygons by examining the reflections of light beams from the model.

[0052] In some embodiments, the intelligent detection and determination mechanism is configured to perform auxiliary performance checks in conjunction with scene complexity detection. The auxiliary performance check involves real-time monitoring of the GPU rendering frame rate to determine whether the current system performance meets the rendering requirements. If the performance is insufficient, even if the scene complexity detection result is 1, the system will further reduce the resolution to reduce the scene complexity until the required system performance is met.

[0053] Figure 3 is a schematic diagram showing the functions of a control module of a display device according to some embodiments of the present disclosure. Figure 3, the control module (e.g., one or more processors and peripheral circuits connected thereto) is configured to perform image display scene rendering. Image display scene rendering can be performed for various appropriate applications (e.g., game applications and video viewing applications). In a specific example, the image display scene rendering is a game scene rendering in a game application.

[0054] In some embodiments, the control module is configured to perform scene complexity detection. In some embodiments, the control module is configured to group the scene complexity of the image display scene into multiple levels, including a first level ("level 1"), a second level ("level 2"), and a third level ("level 3"). In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0055] In some embodiments, scene complexity is detected based on detection of model complexity of the scene. In some embodiments, in order to perform scene complexity detection, the control module is configured to perform ray collision detection. Optionally, performing ray collision detection includes simulating the interaction of light with the model. As used herein, the term "model" refers to a three-dimensional representation of an object, scene, or environment created within a computer graphics or simulation system. These models are used in various applications, such as video games, computer-aided design (CAD), architectural visualization, virtual reality (VR), etc. Complex models can have a large number of vertices, polygons, or texture mappings, which increases the computing resources required to render it realistically. Evaluating model complexity helps determine how resource-intensive it is to render a scene or object, and this information is crucial for optimizing performance in real-time applications, especially in computer graphics and games. Optionally, performing ray collision detection also includes evaluating multiple vertices and polygons by checking reflections of simulated light beams from the model.

[0056] In some embodiments, the control module is further configured to send complexity level information to one or more applications. As used herein, the term "application" refers to a software program or software fragment designed for a specific task or function. These applications are executed on a computer system, mobile device or other computing platform to perform various functions or provide specific services to users. Examples of applications include game applications, video viewing applications, mobile applications and web applications.

[0057] In some embodiments, the control module is further configured to render the image at a first resolution range when determining that the scene complexity is at a first level ("first resolution rendering"). Optionally, the first resolution range is a high resolution range. In some embodiments, the control module is configured to render the image at the first resolution range and a high frame rate range. Optionally, the first frame rate range is a high frame rate range.

[0058] In some embodiments, when the control module is configured to render an image at a first resolution range, the control module is further configured to control the display panel to display the image based on the user's gaze direction (e.g., in a gaze point display mode). Optionally, the control module is configured to track the user's eye movement and gaze direction. Optionally, the control module is further configured to render a first sub-image and a second sub-image, wherein the first sub-image is in an area that intersects with the gaze direction and the second sub-image is in an area that does not intersect with the gaze direction. Optionally, the first sub-image has a higher resolution than the second sub-image.

[0059] In some embodiments, the control module is further configured to render the image at a second resolution range when determining that the scene complexity is at a second level ("second resolution rendering"). Optionally, the second resolution range is lower than the first resolution range. Optionally, the second level of scene complexity is higher than the first level of scene complexity.

[0060] In some embodiments, the control module is further configured to render the image at a third resolution range when determining that the scene complexity is at a third level ("third resolution rendering"). Optionally, the third resolution range is lower than the second resolution range, and the second resolution range is lower than the first resolution range. Optionally, the third level of scene complexity is higher than the second level of scene complexity, and the second level of scene complexity is higher than the first level of scene complexity.

[0061] In some embodiments, when rendering an image at the first resolution range, the second resolution range, or the third resolution range, the display device is configured to display the image at a frame rate above a threshold. In a specific example, the threshold is 90 frames per second.

[0062] In some embodiments, the control module is further configured to send a resolution switching command to the underlying component and drive the underlying component to switch to the corresponding resolution output. As used herein, the term "underlying component" refers to the foundation or basic element of a system that provides basic functions and interacts closely with the underlying hardware. These components typically operate under abstract underlying operations and are responsible for tasks such as hardware control, resource management, and interfaces with hardware devices. Examples of underlying components include device drivers, kernels, firmware, hardware abstraction layers, etc.

[0063] In some embodiments, the control module is further configured to monitor the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements ("auxiliary performance check"). In some embodiments, when it is determined that the current system performance fails to meet the rendering requirements, the control module is further configured to render the image at a lower resolution range until the current system performance can meet the rendering requirements.

[0064] Figure 4 is a schematic diagram showing the functions of a control module of a display device according to some embodiments of the present disclosure. Figure 4 , a control module (e.g., one or more processors and peripheral circuits connected thereto) is configured to start one or more applications and initialize image display scene information. Subsequently, the control module is further configured to identify the image display scene information. In some embodiments, the image display scene includes a plurality of different display scenes, including a first scene and a second scene. In one example, the first scene is a video viewing scene for a user to watch a video or an image using a video viewing application. In another example, the second scene is a game scene for a user to play a video game using a game application.

[0065] In some embodiments, the control module is configured to prompt the user to select the type of application during installation of the application. For example, when the user installs the application, the control module is configured to prompt the user to select from a list of image display scenes such as a game scene or a video viewing scene. When installing the application, the control module adds a symbolic name of the application to indicate the type of image display scene. When the user starts the application, the control module is configured to initialize the image display scene information, during which the control module is configured to identify the symbolic name.

[0066] In some embodiments, when it is determined that the image display scene is the first scene, the control module is configured to turn off the gaze point sensor. In some embodiments, the control module is also configured to monitor the power of the power supply (e.g., battery) of the display device. As used herein, the term "power" refers to how much capacity of the power supply is currently available. For example, 50% power means half of the power.

[0067] In some embodiments, when it is determined that the power of the power supply is greater than a threshold power, the control module is configured to render the image at a fourth resolution range. In some embodiments, when it is determined that the power of the power supply is less than or equal to the threshold power, the control module is configured to render the image at a fifth resolution range. In one example, the threshold power is 50% of the full capacity of the power supply. In other examples, the fourth resolution range is a 4K×4K resolution. In other examples, the fourth resolution range is a 4K×2K resolution. In other examples, the fifth resolution range is a 2K×2K resolution.

[0068] In some embodiments, when it is determined that the image display scene is the second scene, the control module is configured to turn on the acceleration sensor to monitor the user's head movement. In some embodiments, when it is determined that the speed of the head movement is greater than the first threshold speed, the control module is configured to turn off the gaze point sensor; and render the image at a sixth resolution range. In a specific example, the first threshold speed is 30 degrees per second. In one example, the sixth resolution range is a resolution of 2K×2K.

[0069] In some embodiments, when it is determined that the speed of the head movement is less than or equal to a first threshold speed, the control module is configured to turn on the gaze point sensor; and monitor the speed of the eye movement. In some embodiments, when it is determined that the speed of the eye movement is greater than a second threshold speed, the control module is configured to render the image at a second resolution range. Optionally, when it is determined that the speed of the eye movement is greater than the second threshold speed, the control module is further configured to control the gaze point sensor to monitor the speed of the eye movement in the first speed exposure mode. In one example, the second threshold speed is 30 degrees per second.

[0070] In some embodiments, when it is determined that the speed of the eye movement is less than or equal to the second threshold speed, the control module is configured to control the gaze point sensor to monitor the speed of the eye movement in the second speed exposure mode. In the first speed exposure mode, the sensor monitors the speed of the eye movement at a first frequency; in the second speed exposure mode, the sensor monitors the speed of the eye movement at a second frequency; the second frequency is greater than the first frequency.

[0071] In some embodiments, when it is determined that the speed of the eye movement is less than or equal to the second threshold speed, the control module is configured to perform scene complexity detection. In some embodiments, the control module is configured to group the scene complexity of the image display scene into multiple levels, including a first level ("level 1"), a second level ("level 2"), and a third level ("level 3"). In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0072] In some embodiments, scene complexity is detected based on detection of model complexity of the scene. In some embodiments, to perform model complexity detection, the control module is configured to perform ray collision detection. Optionally, performing ray collision detection includes determining the number of vertices and polygons of the model. In some embodiments, the control module is configured to group the scene complexity of the image display scene into multiple levels, including a first level ("level 1"), a second level ("level 2"), and a third level ("level 3"). In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0073] In some embodiments, the control module is further configured to render the image at a first resolution range when determining that the scene complexity is at a first level. Optionally, the first resolution range is a high resolution range. In some embodiments, the control module is configured to render the image at the first resolution range and a high frame rate range. Optionally, the first frame rate range is a high frame rate range. In one example, the first resolution range is a 2K×2K resolution.

[0074] In some embodiments, the control module is further configured to render a first sub-image and a second sub-image, wherein the first sub-image is in an area intersecting with the gaze direction, and the second sub-image is in an area not intersecting with the gaze direction. Optionally, the first sub-image has a higher resolution than the second sub-image.

[0075] In some embodiments, the control module is configured to render the first sub-image at a first resolution range when determining that the scene complexity is at a first level.

[0076] In some embodiments, the control module is further configured to monitor the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements ("auxiliary performance check"). In some embodiments, upon determining that the current system performance fails to meet the rendering requirements, the control module is further configured to render the image at a second resolution range. In one example, the second resolution range is a resolution of 1.5K×1.5K. In some embodiments, the control module is configured to render the first sub-image at the second resolution range upon determining that the current system performance fails to meet the rendering requirements.

[0077] In some embodiments, the control module is further configured to render the image at a second resolution range when determining that the scene complexity is at a second level. Optionally, the second resolution range is lower than the first resolution range. Optionally, the second level of scene complexity is higher than the first level of scene complexity. In one example, the second resolution range is a resolution of 1.5K×1.5K.

[0078] In some embodiments, the control module is configured to render the first sub-image at a second resolution range when it is determined that the scene complexity is at a second level.

[0079] In some embodiments, the control module is further configured to monitor the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements ("auxiliary performance check"). In some embodiments, when it is determined that the current system performance fails to meet the rendering requirements, the control module is further configured to render the image at a third resolution range. In one example, the third resolution range is a resolution of 1.2K×1.2K. In some embodiments, the control module is configured to render the first sub-image at the third resolution range when it is determined that the current system performance fails to meet the rendering requirements.

[0080] In some embodiments, the control module is further configured to render the image at a third resolution range when determining that the scene complexity is at a third level. Optionally, the third resolution range is lower than the second resolution range, and the second resolution range is lower than the first resolution range. Optionally, the third level of scene complexity is higher than the second level of scene complexity, and the second level of scene complexity is higher than the first level of scene complexity. In one example, the third resolution range is a resolution of 1.2K×1.2K.

[0081] In some embodiments, the control module is configured to render the first sub-image at a third resolution range when it is determined that the scene complexity is at a third level.

[0082] Figure 5 The interaction between applications, underlying components, input hardware, and output hardware in a display device according to some embodiments of the present disclosure is shown. Figure 5 When an application in the application layer is triggered, a signal is sent to the clock management module. The clock management module is configured to send a synchronization signal to a synchronization signal distribution thread, and send a bottom-level clock synchronization signal to a bottom-level interface module. The synchronization signal distribution thread is configured to distribute synchronization signals to various applications, including a sensor synchronization control thread, a light-emitting element synchronization control thread, and a display panel synchronization control thread.

[0083] In some embodiments, the sensor synchronization control thread, the light emitting element synchronization control thread or the display panel synchronization control thread is configured to send a synchronization signal to the underlying interface module. The underlying interface module is configured to send a synchronization signal to various underlying component driver modules of the underlying components, including a display driver module, a sensor driver module and a light emitting element driver module. The sensor driver module is configured to drive a camera sensor. The light emitting element driver module is configured to drive a fill light driver circuit. The display driver module is configured to drive a display panel.

[0084] The synchronization signal synchronizes the operation of the application, the underlying components, the input hardware and the output hardware. When the application in the application layer is triggered, the synchronization signal distribution thread is configured to distribute the synchronization signal to various applications and various underlying components, and finally synchronize the input hardware and the output hardware.

[0085] Figure 6 FIG. 2 shows a synchronization operation performed by a synchronization signal in a display device according to some embodiments of the present disclosure. Figure 6, in some embodiments, the control module includes an image rendering module configured to render an image. For example, the image rendering module can be configured to render an image in a first resolution range, render an image in a second resolution range, render an image in a third resolution range, render an image in a fourth resolution range, render an image in a fifth resolution range, or render an image in a sixth resolution range. Optionally, the image rendering module can be configured to render a first sub-image and a second sub-image, wherein the first sub-image is in an area that intersects with the gaze direction and the second sub-image is in an area that does not intersect with the gaze direction. When rendering the image, the image rendering module is configured to send a signal to a clock management module. Upon receiving the signal, the clock management module is configured to generate a synchronization signal. In some embodiments, the clock management module is configured to send the synchronization signal to a synchronization signal distribution thread. Refer to Figure 5 The synchronization signal distribution thread is configured to distribute synchronization signals to various applications, including a sensor synchronization control thread, a light emitting element synchronization control thread, and a display panel synchronization control thread.

[0086] refer to Figure 6 In some embodiments, the control module includes a determination module configured to determine the speed of eye movement, determine the speed of head movement, determine the image display scene, and determine system performance, as previously described in conjunction with Figure 3 and Figure 4 The determination module is configured to send the determination result to the image rendering module.

[0087] In some embodiments, the determination module is configured to obtain the coordinates of the gaze point at two different moments and the angular velocity of the line of sight. Then, the eye movement speed is calculated as the angular velocity divided by the time difference between the two moments.

[0088] In some embodiments, the determination module is configured to determine the direction of the head movement, such as up and down, left and right, or diagonal. When determining the direction of the head movement, the determination module is also configured to extract the angular velocity data in the horizontal direction of the head movement and convert the angular velocity data into the speed of the head movement.

[0089] In some embodiments, the control module further comprises a real-time information acquisition module, which is configured to acquire information required by the determination module in real time. Examples of real-time information include position information, gaze point information, symbolic name of the application, rendering frame rate, and model complexity level or scene complexity level.

[0090] In some embodiments, the control module further includes an eye movement speed and head movement speed calculation module, which is configured to calculate the eye movement speed based on the gaze point image information transmitted by the camera sensor, and calculate the head movement speed based on the gesture information transmitted by the acceleration sensor. The eye movement speed and head movement speed calculation module is also configured to send the eye movement speed and the head movement speed to the real-time information acquisition module.

[0091] In some embodiments, the control module further includes an initialization module configured to initialize image display scene information based on scene content. Examples of scene content include moving scenes, static scenes, game scenes, and video viewing scenes. The initialization module is further configured to send the image display scene information to the real-time information acquisition module.

[0092] In some embodiments, the control module also includes additional modules, such as a software development rendering module and a graphics memory rendering module.

[0093] Figure 7 The process of information output in a display device according to some embodiments of the present disclosure is shown. Figure 7 In some embodiments, the display device is configured to pre-configure a list of supported display resolutions in the system. When the application triggers the resolution switching mechanism, it will call the resolution switching API interface and send the required resolution change information to the kernel driver layer. The kernel driver layer will traverse the parsing list, match it, and then reset the timing, recalculate the information buffer size, the required speed, and the bandwidth for information transmission. The kernel driver layer will also trigger the screen controller to refresh a new initialization sequence to prepare to receive information from the new resolution image. The system layer will also reset the frame buffer according to the new resolution to match the new display resolution.

[0094] As before, in some embodiments, when it is determined that the speed of the eye movement is greater than the second threshold speed, the control module is further configured to control the gaze point sensor to monitor the speed of the eye movement in the first speed exposure mode. When the speed of the eye movement is greater than the second threshold speed, the sensor that recognizes the eye movement data does not need to be precisely positioned; it only needs to detect the eye movement speed. Therefore, the sensor will be adjusted to the first speed exposure, and the fill light will also be synchronized to a low-frequency flash, illuminating only when the sensor is exposed.

[0095] In some embodiments, when it is determined that the speed of eye movement is less than or equal to the second threshold speed, the control module is configured to control the gaze point sensor to monitor the speed of eye movement in the second speed exposure mode. When the speed of eye movement is less than or equal to the second threshold speed, the gaze point needs to be accurately located, the sensor needs to be exposed at the second speed, and the fill light can also use the full lighting mode. By adopting this working mode, the energy consumption required for operation can be greatly reduced, and the power consumption of the system can be effectively reduced. Figure 8 The process of determining the first speed exposure mode or the second speed exposure mode according to some embodiments of the present disclosure is shown. Figure 8 As shown, the operation mechanism is based on a synchronization signal to achieve synchronization between the sensor, fill light and the system's operation clock.

[0096] In some embodiments, the display device includes a variable frequency drive module configured to modulate the pulse width modulation timing of the fill light drive circuit. The variable frequency drive module is configured to modulate the pulses for the fill light drive circuit to have a first pulse frequency in a first drive mode and a second pulse frequency in a second drive mode, wherein the first pulse frequency is less than the second pulse frequency. The first drive mode corresponds to a first speed exposure mode of a gaze point sensor configured to monitor the speed of eye movement, and the second drive mode corresponds to a second speed exposure mode of the gaze point sensor. When the speed of eye movement is greater than a second threshold speed, the sensor that identifies eye movement data does not need to be precisely positioned, and the pulses for the fill light drive circuit have a first pulse frequency in the first drive mode. When the speed of eye movement is less than or equal to the second threshold speed, the gaze point needs to be precisely positioned, and the pulses of the fill light drive circuit have a second pulse frequency in the second drive mode.

[0097] Fig. 9 is a schematic diagram showing a display device according to some embodiments of the present disclosure. Fig. 9 In some embodiments, the display device includes a processor 1002, a storage medium 1004, a display 1006, a communication module 1008, a database 1010, a peripheral device 1012, and a camera 1014. Certain devices may be omitted, and other devices may be included to better describe the relevant embodiments. The display device may include any appropriate type of display panel, such as a plasma display panel, a liquid crystal display (LCD) panel, a touch screen display panel, a projection display panel, a non-intelligent display panel, an intelligent display panel, etc. The display device may also include other computing systems, such as a personal computer (PC), a tablet or a portable computer, or a smart phone, etc. The display device may be any appropriate content presentation device capable of presenting any appropriate content. The user may interact with the display device to perform other activities of interest.

[0098] The processor 1002 may include any appropriate one or more processors. In addition, the processor 1002 may include multiple cores for multi-threading or parallel processing. The processor 1002 may execute a sequence of computer program instructions to perform various processes. The storage medium 1004 may include a memory module, such as a ROM, a RAM, a flash memory module, and a large-capacity memory, such as a CD-ROM and a hard disk. The storage medium 1004 may store a computer program for implementing various processes when the computer program is executed by the processor 1002. For example, the storage medium 1004 may store a computer program for implementing various algorithms when the processor 1002 executes the computer program.

[0099] In addition, the communication module 1008 may include certain network interface devices for establishing a connection through a communication network (e.g., a TV cable network, a wireless network, the Internet, etc.) The database 1010 may include one or more databases for storing certain data and for performing certain operations on the stored data, such as database searches.

[0100] Display 1006 can provide information to a user. Display 1006 can include any suitable type of computer display device or electronic device display, such as an LCD or OLED based device. Peripheral devices 1012 can include various sensors and other I / O devices, such as a keyboard and mouse.

[0101] Examples of suitable display devices include, but are not limited to, electronic paper, mobile phones, tablet computers, televisions, monitors, notebook computers, digital photo albums, GPS, etc. Optionally, the display device is an organic light emitting diode display device. Optionally, the display device is a micro light emitting diode display device. Optionally, the display device is a mini light emitting diode display device.

[0102] In some embodiments, the device includes a memory; and one or more processors. Optionally, the memory is interconnected with the one or more processors. Optionally, the memory stores computer executable instructions for controlling the one or more processors to perform various functions described herein.

[0103] In another aspect, the present disclosure provides a display method. Fig.10 is a flow chart showing a display method in some embodiments of the present disclosure. Fig.10In some embodiments, the method includes identifying image display scene information; and performing scene complexity detection based on the image display scene information. In some embodiments, performing scene complexity detection includes grouping scene complexity of the image display scene into multiple levels based on the type of the image display scene, including a first level, a second level, and a third level. In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0104] In some embodiments, scene complexity is detected based on a detection of model complexity of the scene. In some embodiments, performing model complexity detection includes performing ray collision detection. Optionally, performing ray collision detection includes simulating the interaction of light with the model. Optionally, performing ray collision detection also includes evaluating multiple vertices and polygons by examining reflections of simulated light beams from the model.

[0105] In some embodiments, the method further includes sending complexity level information to one or more applications.

[0106] In some embodiments, the method further comprises rendering the image at a first resolution range when determining that the scene complexity is at a first level. Optionally, the first resolution range is a high resolution range. In some embodiments, the control module is configured to render the image at the first resolution range and a high frame rate range. Optionally, the first frame rate range is a high frame rate range.

[0107] In some embodiments, the method further comprises controlling the display panel to display an image based on a gaze direction of the user. Optionally, the method further comprises tracking the user's eye movement and gaze direction. Optionally, the method further comprises rendering a first sub-image and a second sub-image, wherein the first sub-image is in an area intersecting with the gaze direction and the second sub-image is in an area not intersecting with the gaze direction. Optionally, the first sub-image has a higher resolution than a resolution of the second sub-image.

[0108] In some embodiments, the method further comprises rendering the image at a second resolution range when determining that the scene complexity is at a second level. Optionally, the second resolution range is lower than the first resolution range. Optionally, the second level of scene complexity is higher than the first level of scene complexity.

[0109] In some embodiments, the method further includes rendering the image at a third resolution range when determining that the scene complexity is at a third level. Optionally, the third resolution range is lower than the second resolution range, and the second resolution range is lower than the first resolution range. Optionally, the third level of scene complexity is higher than the second level of scene complexity, and the second level of scene complexity is higher than the first level of scene complexity.

[0110] In some embodiments, the method further comprises maintaining the frame rate of the display image above a threshold. In a specific example, the threshold is 90 frames per second.

[0111] In some embodiments, the method further includes sending a resolution switching command to the underlying component, and driving the underlying component to switch to a corresponding resolution output.

[0112] In some embodiments, the method further comprises monitoring the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements. In some embodiments, when it is determined that the current system performance fails to meet the rendering requirements, the method further comprises rendering the image at a lower resolution range until the current system performance can meet the rendering requirements.

[0113] Fig.11 is a flow chart showing a display method in some embodiments of the present disclosure. Fig.11 In some embodiments, the method includes starting one or more applications and initializing image display scene information; identifying image display scene information. In some embodiments, the image display scene includes a plurality of different display scenes, including a first scene and a second scene. In one example, the first scene is a video viewing scene for a user to use a video viewing application to watch a video or an image. In another example, the second scene is a game scene for a user to use a game application to play a video game.

[0114] In some embodiments, the method further includes turning off the gaze point sensor when determining that the image display scene is the first scene. In some embodiments, the method further includes monitoring the power of a power supply of the display device; and determining whether the power of the power supply is greater than a threshold power. In some embodiments, the method further includes rendering the image at a fourth resolution range when determining that the power of the power supply is greater than a threshold power. In some embodiments, the method further includes rendering the image at a fifth resolution range when determining that the power of the power supply is less than or equal to a threshold power. In one example, the threshold power is 50% of the full capacity of the power supply. In other examples, the fourth resolution range is a 4K×4K resolution. In other examples, the fourth resolution range is a 4K×2K resolution. In other examples, the fifth resolution range is a 2K×2K resolution.

[0115] In some embodiments, the method further includes turning on an acceleration sensor to monitor the user's head movement when it is determined that the image display scene is the second scene. In some embodiments, the method further includes turning off the gaze point sensor when it is determined that the speed of the head movement is greater than a first threshold speed; and rendering the image at a sixth resolution range. In a specific example, the first threshold speed is 30 degrees per second. In one example, the sixth resolution range is a 2K×2K resolution.

[0116] In some embodiments, the method further includes, when it is determined that the speed of the head movement is less than or equal to a first threshold speed, turning on the gaze point sensor; and monitoring the speed of the eye movement. In some embodiments, the method further includes rendering the image at a seventh resolution range when it is determined that the speed of the eye movement is greater than a second threshold speed. Optionally, the method further includes, when it is determined that the speed of the eye movement is greater than the second threshold speed, controlling the gaze point sensor to monitor the speed of the eye movement in a first speed exposure mode. In one example, the second threshold speed is 30 degrees per second. In one example, the seventh resolution range is a resolution of 2K×2K.

[0117] In some embodiments, the method further includes controlling the gaze point sensor to monitor the speed of the eye movement in a second speed exposure mode when it is determined that the speed of the eye movement is less than or equal to a second threshold speed.

[0118] In some embodiments, the method further includes performing scene complexity detection after determining that the speed of the eye movement is less than or equal to the second threshold speed. In some embodiments, the method further includes grouping the scene complexity of the image display scene into multiple levels, including a first level, a second level, and a third level. In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0119] In some embodiments, scene complexity is detected based on detection of model complexity of the scene. In some embodiments, performing model complexity detection includes performing ray collision detection. Optionally, performing ray collision detection includes determining the number of vertices and polygons of the model. In some embodiments, the method also includes grouping the scene complexity of the image display scene into multiple levels, including a first level, a second level, and a third level. In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0120] In some embodiments, the method further comprises rendering the image at a first resolution range when determining that the scene complexity is at a first level. Optionally, the first resolution range is a high resolution range. In some embodiments, the method further comprises rendering the image at the first resolution range and a high frame rate range. Optionally, the first frame rate range is a high frame rate range. In one example, the first resolution range is a 2K×2K resolution.

[0121] In some embodiments, the method further comprises rendering a first sub-image and a second sub-image, wherein the first sub-image is in an area intersecting the gaze direction and the second sub-image is in an area not intersecting the gaze direction. Optionally, the first sub-image has a higher resolution than the second sub-image.

[0122] In some embodiments, the method further comprises rendering the first sub-image at a first resolution range when the scene complexity is determined to be at a first level.

[0123] In some embodiments, the method further includes monitoring the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements. In some embodiments, the method further includes rendering the image at a second resolution range when it is determined that the current system performance fails to meet the rendering requirements. In one example, the second resolution range is a resolution of 1.5K×1.5K. In some embodiments, the method further includes rendering the first sub-image at the second resolution range when it is determined that the current system performance fails to meet the rendering requirements.

[0124] In some embodiments, the method further includes rendering the image at a second resolution range when determining that the scene complexity is at a second level. Optionally, the second resolution range is lower than the first resolution range. Optionally, the second level of scene complexity is higher than the first level of scene complexity. In one example, the second resolution range is a resolution of 1.5K×1.5K.

[0125] In some embodiments, the method further includes rendering the first sub-image at a second resolution range when the scene complexity is determined to be at a second level.

[0126] In some embodiments, the method further includes monitoring the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements. In some embodiments, the method further includes rendering the image at a third resolution range when it is determined that the current system performance fails to meet the rendering requirements. In one example, the third resolution range is a resolution of 1.2K×1.2K. In some embodiments, the method further includes rendering the first sub-image at the third resolution range when it is determined that the current system performance fails to meet the rendering requirements.

[0127] In some embodiments, the method further includes rendering the image at a third resolution range when determining that the scene complexity is at a third level. Optionally, the third resolution range is lower than the second resolution range, and the second resolution range is lower than the first resolution range. Optionally, the third level of scene complexity is higher than the second level of scene complexity, and the second level of scene complexity is higher than the first level of scene complexity. In one example, the third resolution range is a resolution of 1.2K×1.2K.

[0128] In some embodiments, the method further comprises rendering the first sub-image at a third resolution range when the scene complexity is determined to be at a third level.

[0129] In some embodiments, the method further includes generating (e.g., via a clock management module) a synchronization signal; sending the synchronization signal to a synchronization signal distribution thread; and distributing the synchronization signal to one or more applications (e.g., via the synchronization signal distribution thread). Examples of the one or more applications include a sensor synchronization control thread, a light emitting element synchronization control thread, and a display panel synchronization control thread.

[0130] In some embodiments, the method further includes (e.g., by one or more applications) sending a synchronization signal to an underlying interface module; and (e.g., through the underlying interface module) sending the synchronization signal to an underlying component driver module of the underlying component. Examples of underlying component driver modules include a display driver module, a sensor driver module, and a light emitting element driver module. The sensor driver module is configured to drive a camera sensor. The light emitting element driver module is configured to drive a fill light driver circuit. The display driver module is configured to drive a display panel.

[0131] In some embodiments, the method further includes sending a bottom-layer clock synchronization signal to the bottom-layer interface module.

[0132] The synchronization signal synchronizes the operation of the application, the underlying components, the input hardware and the output hardware. When the application in the application layer is triggered, the synchronization signal distribution thread is configured to distribute the synchronization signal to various applications and various underlying components, and finally synchronize the input hardware and the output hardware.

[0133] In another aspect, the present disclosure provides a computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon. In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to perform: identifying image display scene information; performing scene complexity detection based on the image display scene information; grouping the scene complexity of the image display scene into multiple levels based on the type of the image display scene; rendering the image at a first resolution range when it is determined that the scene complexity is at a first level; and rendering the image at a second resolution range when it is determined that the scene complexity is at a second level. Optionally, the second resolution range is lower than the first resolution range. Optionally, the second level of scene complexity is higher than the first level of scene complexity.

[0134] In some embodiments, the computer readable instructions are executable by one or more processors to further cause the one or more processors to perform: performing ray collision detection. Optionally, performing ray collision detection includes simulating the interaction of light with the model. Optionally, performing ray collision detection also includes evaluating multiple vertices and polygons by examining reflections of the simulated light beams from the model.

[0135] In some embodiments, the computer readable instructions are executable by one or more processors to further cause the one or more processors to: send complexity level information to one or more applications.

[0136] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering an image at a first resolution range when determining that the scene complexity is at a first level. Optionally, the first resolution range is a high resolution range. In some embodiments, the control module is configured to render the image at the first resolution range and a high frame rate range. Optionally, the first frame rate range is a high frame rate range.

[0137] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to: control the display panel to display an image based on the user's gaze direction. Optionally, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to: track the user's eye movement and gaze direction. Optionally, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to: render a first sub-image and a second sub-image, wherein the first sub-image is in an area that intersects the gaze direction and the second sub-image is in an area that does not intersect the gaze direction. Optionally, the first sub-image has a higher resolution than the second sub-image.

[0138] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering the image at a second resolution range when determining that the scene complexity is at a second level. Optionally, the second resolution range is lower than the first resolution range. Optionally, the second level of scene complexity is higher than the first level of scene complexity.

[0139] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: when determining that the scene complexity is at a third level, rendering the image at a third resolution range. Optionally, the third resolution range is lower than the second resolution range, and the second resolution range is lower than the first resolution range. Optionally, the third level of scene complexity is higher than the second level of scene complexity, and the second level of scene complexity is higher than the first level of scene complexity.

[0140] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: maintaining a frame rate of the displayed image above a threshold. In a specific example, the threshold is 90 frames per second.

[0141] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to: send a resolution switching command to an underlying component, and drive the underlying component to switch to a corresponding resolution output.

[0142] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: monitoring the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements. In some embodiments, when it is determined that the current system performance fails to meet the rendering requirements, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering the image at a lower resolution range until the current system performance can meet the rendering requirements.

[0143] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to: start one or more applications and initialize image display scene information; identify image display scene information. In some embodiments, the image display scene includes a plurality of different display scenes, including a first scene and a second scene. In one example, the first scene is a video viewing scene for a user to use a video viewing application to watch a video or image. In another example, the second scene is a game scene for a user to use a game application to play a video game.

[0144] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause one or more processors to execute: when determining that the image display scene is the first scene, turn off the gaze point sensor. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause one or more processors to execute: monitor the power of the power supply of the display device; and determine whether the power of the power supply is greater than the threshold power. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause one or more processors to execute: when determining that the power of the power supply is greater than the threshold power, render the image under the fourth resolution range. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause one or more processors to execute: when determining that the power of the power supply is less than or equal to the threshold power, render the image under the fifth resolution range. In one example, the threshold power is 50% of the full capacity of the power supply. In other examples, the fourth resolution range is a resolution of 4K×4K. In other examples, the fourth resolution range is a resolution of 4K×2K. In other examples, the fifth resolution range is a resolution of 2K×2K.

[0145] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to execute: when it is determined that the image display scene is the second scene, turn on the acceleration sensor to monitor the user's head movement. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to execute: when it is determined that the speed of the head movement is greater than the first threshold speed, turn off the gaze point sensor; and render the image at a sixth resolution range. In a specific example, the first threshold speed is 30 degrees per second. In one example, the sixth resolution range is a resolution of 2K×2K.

[0146] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to execute: when it is determined that the speed of the head movement is less than or equal to the first threshold speed, turn on the gaze point sensor; and monitor the speed of the eye movement. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to execute: when it is determined that the speed of the eye movement is greater than the second threshold speed, render the image under the seventh resolution range. Optionally, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to execute: when it is determined that the speed of the eye movement is greater than the second threshold speed, control the gaze point sensor to monitor the speed of the eye movement in the first speed exposure mode. In one example, the second threshold speed is 30 degrees per second. In one example, the seventh resolution range is a resolution of 2K×2K.

[0147] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: when it is determined that the speed of the eye movement is less than or equal to a second threshold speed, controlling the gaze point sensor to monitor the speed of the eye movement in a second speed exposure mode.

[0148] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: after determining that the speed of eye movement is less than or equal to the second threshold speed, perform scene complexity detection. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: grouping the scene complexity of the image display scene into multiple levels, including a first level, a second level, and a third level. In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0149] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: ray collision detection. Optionally, performing ray collision detection includes determining the number of vertices and polygons of the model. In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: grouping the scene complexity of the image display scene into multiple levels, including a first level, a second level, and a third level. In some embodiments, the third level indicates a complexity level higher than the second level, and the second level indicates a complexity level higher than the first level.

[0150] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering an image at a first resolution range when determining that the scene complexity is at a first level. Optionally, the first resolution range is a high resolution range. In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering an image at a first resolution range and a high frame rate range. Optionally, the first frame rate range is a high frame rate range. In one example, the first resolution range is a 2K×2K resolution.

[0151] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering a first sub-image and a second sub-image, wherein the first sub-image is in an area intersecting with the gaze direction, and the second sub-image is in an area not intersecting with the gaze direction. Optionally, the first sub-image has a higher resolution than a resolution of the second sub-image.

[0152] In some embodiments, the computer readable instructions are executable by the one or more processors to further cause the one or more processors to perform: rendering a first sub-image at a first resolution range when the scene complexity is determined to be at a first level.

[0153] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: monitoring the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering the image at a second resolution range when it is determined that the current system performance fails to meet the rendering requirements. In one example, the second resolution range is a resolution of 1.5K×1.5K. In some embodiments, the method further includes rendering the first sub-image at the second resolution range when it is determined that the current system performance fails to meet the rendering requirements.

[0154] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering an image at a second resolution range when determining that the scene complexity is at a second level. Optionally, the second resolution range is lower than the first resolution range. Optionally, the second level of scene complexity is higher than the first level of scene complexity. In one example, the second resolution range is a resolution of 1.5K×1.5K.

[0155] In some embodiments, the computer readable instructions are executable by the one or more processors to further cause the one or more processors to perform: rendering the first sub-image at a second resolution range when it is determined that the scene complexity is at a second level.

[0156] In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: monitoring the rendering frame rate in real time to determine whether the current system performance meets the rendering requirements. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering the image at a third resolution range when it is determined that the current system performance fails to meet the rendering requirements. In one example, the third resolution range is a resolution of 1.2K×1.2K. In some embodiments, the computer-readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering the first sub-image at the third resolution range when it is determined that the current system performance fails to meet the rendering requirements.

[0157] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to perform: rendering an image at a third resolution range when it is determined that the scene complexity is at a third level. Optionally, the third resolution range is lower than the second resolution range, and the second resolution range is lower than the first resolution range. Optionally, the third level of scene complexity is higher than the second level of scene complexity, and the second level of scene complexity is higher than the first level of scene complexity. In one example, the third resolution range is a resolution of 1.2K×1.2K.

[0158] In some embodiments, the computer readable instructions are executable by the one or more processors to further cause the one or more processors to perform: rendering the first sub-image at a third resolution range when it is determined that the scene complexity is at a third level.

[0159] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to: generate (e.g., via a clock management module) a synchronization signal; send the synchronization signal to a synchronization signal distribution thread; and distribute the synchronization signal to one or more applications (e.g., via the synchronization signal distribution thread). Examples of the one or more applications include a sensor synchronization control thread, a light emitting element synchronization control thread, and a display panel synchronization control thread.

[0160] In some embodiments, the computer readable instructions may be executed by one or more processors to further cause the one or more processors to execute: (e.g., by one or more applications) sending a synchronization signal to an underlying interface module; and (e.g., through the underlying interface module) sending a synchronization signal to an underlying component driver module of an underlying component. Examples of underlying component driver modules include a display driver module, a sensor driver module, and a light emitting element driver module. The sensor driver module is configured to drive a camera sensor. The light emitting element driver module is configured to drive a fill light driver circuit. The display driver module is configured to drive a display panel.

[0161] In some embodiments, the computer readable instructions are executable by the one or more processors to further cause the one or more processors to: send an underlying clock synchronization signal to the underlying interface module.

[0162] The synchronization signal synchronizes the operation of the application, the underlying components, the input hardware and the output hardware. When the application in the application layer is triggered, the synchronization signal distribution thread is configured to distribute the synchronization signal to various applications and various underlying components, and finally synchronize the input hardware and the output hardware.

[0163] The various illustrative operations described in conjunction with the configuration disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. These operations may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an ASIC or ASSP, an FPGA or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination thereof designed to produce the configuration disclosed herein. For example, such a configuration may be implemented at least in part as a hardwired circuit, as a circuit configuration manufactured into a dedicated integrated circuit, or as a firmware program loaded into a non-volatile storage, or as a software program loaded from or into a data storage medium as a machine-readable code, such code being an instruction executable by an array of logic elements such as a general purpose processor or other digital signal processing unit. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. The software module may reside in a non-transitory storage medium, such as RAM (random access memory), ROM (read only memory), non-volatile RAM (NVRAM), such as flash RAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, removable disk or CD-ROM; or in any other form of storage medium known in the art. The illustrative storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0164] For the purpose of illustration and description, the above description of the embodiments of the present invention has been given. It is not exhaustive, nor is it intended to limit the present invention to the precise form or exemplary embodiments disclosed. Therefore, the foregoing description should be considered illustrative rather than restrictive. Obviously, many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described to explain the principles of the present invention and its best mode practical application, so that those skilled in the art can understand the various embodiments of the present invention and the various modifications suitable for the specific use or implementation under consideration. The scope of the present invention is intended to be defined by the appended claims and their equivalents, wherein all terms are meant to have the broadest reasonable meaning unless otherwise stated. Therefore, the term "the present invention" and the like do not necessarily limit the scope of the claims to a specific embodiment, and the reference to the exemplary embodiments of the present invention does not mean a limitation of the present invention, and such limitation should not be inferred. The present invention is limited only by the spirit and scope of the appended claims. In addition, these claims may involve the use of "first", "second", etc., followed by a noun or element. These terms should be understood as nomenclature, and should not be interpreted as limiting the number of elements modified by these nomenclatures, unless a specific number has been given. Any advantages and benefits described may not apply to all embodiments of the present invention. It should be understood that those skilled in the art may make changes to the described embodiments without departing from the scope of the present invention as defined by the appended claims. In addition, the elements and assemblies in this disclosure are not intended to be contributed to the public, regardless of whether the element or assembly is clearly described in the appended claims.

Claims

1. A display device, include: Memory; as well as one or more processors; wherein the memory and the one or more processors are connected to each other; and The memory stores computer executable instructions to control the one or more processors to: Recognize the image to display scene information; Performing scene complexity detection based on the image display scene information; grouping scene complexity of the image display scenes into multiple levels based on the types of the image display scenes; When determining that the scene complexity is at a first level, rendering an image at a first resolution range; and When determining that the scene complexity is at a second level, rendering the image at a second resolution range; wherein the second resolution range is lower than the first resolution range; and The second level of scene complexity is higher than the first level of scene complexity.

2. The display device according to claim 1, in, The memory also stores computer executable instructions to control the one or more processors to: render a first sub-image and a second sub-image; wherein the first sub-image is in a region intersecting with the gaze direction of the user; The second sub-image is in a region that does not intersect the gaze direction; The first sub-image has a higher resolution than a resolution of the second sub-image; and When it is determined that the scene complexity is at the first level, the first sub-image is rendered at the first resolution range, and when it is determined that the scene complexity is at the second level, the first sub-image is rendered at the second resolution range.

3. The display device according to claim 1, in, When rendering the image at the first resolution range, the memory further stores computer executable instructions to control the one or more processors to: Monitor rendering frame rate in real time to determine whether the current system performance meets rendering requirements; as well as When it is determined that the current system performance fails to meet the rendering requirement, the image is rendered in the second resolution range.

4. The display device according to any one of claims 1 to 3, in, The memory also stores computer executable instructions to control the one or more processors to: render an image at a third resolution range when it is determined that the scene complexity is at a third level; wherein the third resolution range is lower than the second resolution range; and The scene complexity of the third level is higher than the scene complexity of the second level.

5. The display device according to claim 4, in, The memory also stores computer executable instructions to control the one or more processors to: control a frame rate above a threshold regardless of rendering the image at the first resolution range, the second resolution range, or the third resolution range.

6. The display device according to claim 1, in, When rendering the image at the second resolution range, the memory further stores computer executable instructions to control the one or more processors to: Monitor rendering frame rate in real time to determine whether the current system performance meets rendering requirements; as well as When it is determined that the current system performance fails to meet the rendering requirement, rendering the image in a third resolution range; Wherein, the third resolution range is lower than the second resolution range.

7. The display device according to any one of claims 1 to 6, in, The memory also stores computer executable instructions to control the one or more processors to: Launch one or more applications; Initialize the image to display scene information; as well as The image displays scene information.

8. The display device according to any one of claims 1 to 7, further comprising a gaze point sensor configured to detect a gaze of a user; in, The memory further stores computer executable instructions for controlling the one or more processors to: when it is determined that the image display scene is the first scene, turn off the gaze point sensor.

9. The display device according to claim 8, in, The memory also stores computer executable instructions to control the one or more processors to: monitor the power level of a power supply of the display device.

10. The display device according to claim 9, in, The memory also stores computer executable instructions to control the one or more processors to: When it is determined that the power level of the power source is greater than a threshold power level, rendering an image in a fourth resolution range; as well as When it is determined that the power level of the power source is less than or equal to the threshold power level, rendering an image in a fifth resolution range; Among them, the fourth resolution range is larger than the fifth resolution range.

11. The display device according to any one of claims 1 to 7, further comprising an acceleration sensor configured to detect a head movement of a user; in, The memory further stores computer executable instructions for controlling the one or more processors to: when it is determined that the image display scene is the second scene, turn on the acceleration sensor to monitor the head movement of the user.

12. The display device according to claim 11, in, The memory also stores computer executable instructions to control the one or more processors to: When it is determined that the speed of the head movement is greater than a first threshold speed, turning off the gaze point sensor and rendering the image at a sixth resolution range; as well as When it is determined that the speed of the head movement is less than or equal to the first threshold speed, the gaze point sensor is turned on; and the speed of the eye movement is monitored.

13. The display device according to claim 12, in, The memory also stores computer executable instructions to control the one or more processors to: When it is determined that the speed of the eye movement is greater than a second threshold speed, rendering the image at a seventh resolution range and controlling the gaze point sensor to monitor the speed of the eye movement in a first speed exposure mode; as well as When it is determined that the speed of the eye movement is less than or equal to a second threshold speed, controlling the gaze point sensor to monitor the speed of the eye movement in a second speed exposure mode; Wherein, in the first speed exposure mode, the gaze point sensor is configured to monitor the speed of the eye movement at a first frequency; in the second speed exposure mode, the gaze point sensor is configured to monitor the speed of the eye movement at a second frequency; and the second frequency is greater than the first frequency.

14. The display device according to claim 12, in, The memory also stores computer executable instructions to control the one or more processors to: perform the scene complexity detection when it is determined that the speed of the eye movement is less than or equal to a second threshold speed.

15. The display device according to any one of claims 1 to 14, in, The memory also stores computer executable instructions to control the one or more processors to: Generate a synchronization signal to synchronize the operation of the application, underlying components, input hardware and output hardware of the display device, wherein the underlying components include an underlying interface module; a display driver module configured to drive a display panel; a sensor driver module configured to drive a camera sensor; and a light-emitting element driver module configured to drive a fill light driver circuit; sending the synchronization signal to a synchronization signal distribution thread; and The synchronization signal is distributed to one or more applications by the synchronization signal distribution thread.

16. The display device according to claim 15, in, The memory also stores computer executable instructions to control the one or more processors to: The one or more applications send the synchronization signal to a bottom layer interface module; as well as The underlying interface module sends the synchronization signal to the underlying component driving module of the underlying component.

17. The display device according to claim 15, further comprising the camera sensor, the fill light driving circuit and the display panel; in, The memory also stores computer executable instructions to control the one or more processors to: The synchronization signal distribution thread distributes the synchronization signal to the sensor synchronization control thread, the light emitting element synchronization control thread and the display panel synchronization control thread; Sending the synchronization signal to the bottom interface module through the sensor synchronization control thread, the light emitting element synchronization control thread and the display panel synchronization control thread; as well as The synchronization signal is sent by the bottom layer interface module to a display driving module configured to drive the display panel, a sensor driving module configured to drive the camera sensor, and a light emitting element driving module configured to drive the fill light driving circuit.

18. The display device according to claim 1, comprising a clock management module, an image rendering module configured to render an image, and a synchronization signal distribution thread; in, When the image is rendered, the image rendering module is configured to send a signal to a clock management module; as well as Upon receiving the signal, the clock management module is configured to generate a synchronization signal, and is configured to send the synchronization signal to the synchronization signal distribution thread.

19. The display device according to claim 1, further comprising a fill light driving circuit and a variable frequency driving module, wherein the variable frequency driving module is configured to modulate the pulse width modulation timing of the fill light driving circuit; in, The variable frequency driving module is configured to modulate the pulses for the fill light driving circuit to have a first pulse frequency in the first driving mode and a second pulse frequency in the second driving mode; The second pulse frequency is greater than the first pulse frequency; The first drive mode corresponds to a first speed exposure mode of a gaze point sensor configured to monitor a speed of the eye movement; The second driving mode corresponds to a second speed exposure mode of the gaze point sensor; as well as In the first speed exposure mode, the gaze point sensor is configured to monitor the speed of the eye movement at a first frequency; In the second speed exposure mode, the gaze point sensor is configured to monitor the speed of the eye movement at a second frequency; and the second frequency is greater than the first frequency.

20. A display method, include: Recognize the image to display scene information; Performing scene complexity detection based on the image display scene information; grouping scene complexity of the image display scenes into multiple levels based on the types of the image display scenes; When determining that the scene complexity is at a first level, rendering an image at a first resolution range; as well as When determining that the scene complexity is at a second level, rendering the image at a second resolution range; wherein the second resolution range is lower than the first resolution range; and The second level of scene complexity is higher than the first level of scene complexity.

21. A computer program product comprising a non-transitory tangible computer readable medium having computer readable instructions thereon, the computer readable instructions executable by a processor to cause the processor to perform: Recognize the image to display scene information; Performing scene complexity detection based on the image display scene information; grouping scene complexity of the image display scenes into multiple levels based on the types of the image display scenes; When determining that the scene complexity is at a first level, rendering an image at a first resolution range; as well as When determining that the scene complexity is at a second level, rendering the image at a second resolution range; wherein the second resolution range is lower than the first resolution range; and The second level of scene complexity is higher than the first level of scene complexity.

Images