Display method and electronic device

Abstract

The present application provides a display method and an electronic device. The display method comprises: on the basis of a first image outputted by an SOC, an electronic device displays a second image on a first display screen. The resolution of the first image is less than that of the second image, the first image comprises K first areas, the second image comprises K second areas, different first areas among the K first areas correspond to different second areas among the K second areas, and the position of each first area in the first image is the same as that of the corresponding second area in the second image, wherein the resolution of a first target area among the K first areas is the same as that of a corresponding second target area among the K second areas, the content of the first target area is the same as that of the second target area, and the content of each of second areas other than the first target area among the K second areas is obtained by performing up-sampling on the content of the corresponding first area.

Description

Display method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 15, 2023, with application number 202311204134.0 and application name “Display Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal devices, and in particular to a display method and electronic device. Background Art

[0003] With the development of terminal display technology, the application scenarios of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies are increasing. Users can experience different effects such as VR, AR, or MR by wearing head-mounted display devices.

[0004] However, in the actual process of wearing the head-mounted display device, it was found that the head-mounted display device has a display delay, which causes the user to have the risk of dizziness when wearing it, seriously affecting the user experience.

[0005] Summary of the Invention

[0006] The present application provides a display method and electronic device to reduce display latency, thereby avoiding the risk of dizziness when the user wears the device and improving the comfort of the user when wearing a head-mounted display device.

[0007] In a first aspect, the present application provides a display method applied to an electronic device, the electronic device including a first display screen, the method comprising: when the first display screen receives a first image, the first display screen displays a second image.

[0008] Exemplarily, the electronic device is a head-mounted display device.

[0009] In the embodiments of the present application, the device that outputs the first image is referred to as a processing device. For example, the first image received by the first display screen may be an image output by a system on chip (SOC), that is, the processing device is the SOC. It should be noted that the present application does not limit the specific form between the SOC and the electronic device. For example, in some embodiments, the SOC is included in the electronic device. For another example, in some embodiments, the SOC is not included in the electronic device. That is, the SOC and the electronic device are independent of each other.

[0010] Specifically, in the embodiment of the present application, the resolution of the second image output by the first display screen is the same as the resolution of the first image.

[0011] Specifically, in the embodiment of the present application, the first image and the second image output by the first display screen each include K regions. In the embodiment of the present application, the K regions included in the first image are referred to as K first regions, and the K regions included in the second image output by the first display screen are referred to as K second regions.

[0012] For example, both the first image and the second image can be divided into nine (i.e., K is 9) areas in three rows and three columns: a central area located at the center of the image, an upper-left area located at the upper-left corner of the central area, an upper transition area located above the central area, an upper-right area located at the upper-right corner of the central area, a left transition area located to the left of the central area, a right transition area located to the right of the central area, a lower-left area located at the lower-left corner of the central area, a lower transition area located below the central area, and a lower-right area located at the lower-right corner of the central area.

[0013] Specifically, different first areas among the K first areas correspond to different second areas among the K second areas, and the position of each first area in the first image is the same as the position of the corresponding second area in the second image. In other words, in the embodiment of the present application, the K first areas correspond one-to-one with the K second areas, and a first area in the first image and its corresponding second area in the second image are in the same position in each image. In other words, in the embodiment of the present application, the first image and the second image output by the first display screen are aligned.

[0014] For example, taking the above-mentioned first image and second image as an example, both of which can be divided into nine areas (i.e., K is 9) of three rows and three columns, it can be considered that: the upper left corner area in the first image corresponds to the upper left corner area in the second image, the upper transition area in the first image corresponds to the upper transition area in the second image, the upper right corner area in the first image corresponds to the upper right corner area in the second image, the left transition area in the first image corresponds to the left transition area in the second image, the center area in the first image corresponds to the center area in the second image, the right transition area in the first image corresponds to the right transition area in the second image, the lower left corner area in the first image corresponds to the lower left corner area in the second image, the lower transition area in the first image corresponds to the lower transition area in the second image, and the lower right corner area in the first image corresponds to the lower right corner area in the second image.

[0015] Specifically, in an embodiment of the present application, the resolution of the first target area in the K first areas is the same as the resolution of the second target area in the K second areas, and the content of the first target area is the same as the content of the second target area, and the resolution of each second area in the K second areas except the second target area is higher than the resolution of the corresponding first area (for example, the content of each second area in the K second areas except the second target area is obtained by upsampling the content of the corresponding first area).

[0016] In other words, in an embodiment of the present application, the second image displayed on the first display screen includes a second target area, and the content of the second target area is exactly the same as the content of the first target area in the first image output by the SOC, that is, the second target area displayed on the first display screen is not upsampled with respect to the content of the first target area, and the original resolution of the first target area is maintained.

[0017] It should be understood that in this display method, by making the resolution of the first image smaller than the resolution of the second image output by the first display screen, it is possible to ensure smoothness of display when the transmission bandwidth of the processing device (such as SOC) that outputs the first image is insufficient. Since when the second image is displayed to the user on the first display screen, the second target area included in the second image is not upsampled to the corresponding first target area in the SOC, the clarity of the first target area can be guaranteed. In a specific implementation, the first target area is usually the user's gaze area. In addition, in the display method provided by the embodiment of the present application, since the first image output by the processing device and the second image output by the first display screen are aligned, at this time, there is no need to first complete the transmission of the first target area before starting to display, and data can be transmitted while being displayed on the first display screen, thereby reducing display latency.

[0018] In combination with the first aspect, in some embodiments, each second region except the second target region in the K second regions is obtained by upsampling the corresponding first region, and the upsampling ratio corresponding to the second region farther from the second target region in the K second regions is greater.

[0019] In other words, when the second image is displayed on the first display screen based on the first image, for the second region in the K second regions that is farther from the second target region, a larger upsampling ratio is used when the second region is obtained based on the corresponding first region.

[0020] It is understandable that in this implementation, because when displaying the second image on the first display screen based on the first image, the upsampling ratio used when upsampling the first region farther from the first target region among the K first regions is greater, the K second regions displayed on the first display screen have the following characteristics: the second regions farther from the second target region in the second image are less clear, while the second regions closer to the second target region are clearer. In other words, in this implementation, when displaying the K second regions on the first display screen, the clarity of the second regions at different distances from the second target region is different.

[0021] It should be understood that compared to using the same resolution for the first target area and the corresponding second target area, all other second areas are sampled with the same upsampling ratio. The second image displayed in this way is more in line with the characteristics of the user's viewing, and thus can improve the user experience.

[0022] In combination with the first aspect, in some embodiments, the electronic device also includes a second display screen, and the method also includes: when the second display screen receives the third image, the second display screen displays a fourth image, the resolution of the third image is less than the resolution of the fourth image, the third image includes K third areas, the fourth image includes K fourth areas, different third areas among the K third areas correspond to different fourth areas among the K fourth areas, and the position of each third area in the third image is the same as the position of the corresponding fourth area in the fourth image; the K third areas include a third target area, the K fourth areas include a fourth target area corresponding to the third target area, the resolution of the third target area and the fourth target area are the same, and the resolution of each fourth area except the fourth target area in the K fourth areas is higher than the resolution of the corresponding third area.

[0023] Among them, this solution can be compared to the display method of the first display screen in the first aspect, and will not be repeated here.

[0024] In conjunction with the first aspect, in some embodiments, the first image and the second image can both be divided into nine areas in three rows and three columns: a central area located at the center of the image, an upper-left area located at the upper-left corner of the central area, an upper transition area located above the central area, an upper-right area located at the upper-right corner of the central area, a left transition area located to the left of the central area, a right transition area located to the right of the central area, a lower-left area located at the lower-left corner of the central area, a lower transition area located below the central area, and a lower-right area located at the lower-right corner of the central area;

[0025] The first target area is the central area of ​​the first image, the second target area is the central area of ​​the second image, and the first target area and the second target area have the same resolution, including: the central area of ​​the second image has the same resolution as the central area of ​​the first image.

[0026] In combination with the first aspect, in some embodiments, the upper left corner area in the second image is obtained based on upsampling the upper left corner area in the first image by q1 times in the horizontal direction and by p1 times in the vertical direction, the left transition area in the second image is obtained based on upsampling the left transition area in the first image by q1 times in the horizontal direction and without upsampling in the vertical direction, and the upper transition area in the second image is obtained based on not upsampling the upper transition area in the first image in the horizontal direction and by p1 times in the vertical direction, wherein q1 and p1 are both greater than 1.

[0027] Exemplarily, when the resolution of the second image is 3840*3840, the upper left corner area in the second image is obtained by upsampling the upper left corner area in the first image by a factor of 2 in both the horizontal and vertical directions, the left transition area in the second image is obtained by upsampling the left transition area in the first image by a factor of 2 in the horizontal direction and without upsampling in the vertical direction, and the upper transition area in the second image is obtained by not upsampling the upper transition area in the first image in the horizontal direction and upsampling it by a factor of 2 in the vertical direction.

[0028] In a second aspect, the present application provides a display device, comprising a functional module for implementing the method as described in the first aspect or any possible implementation thereof.

[0029] In a third aspect, the present application provides a display device comprising: a memory and a processor; the memory is used to store program instructions; the processor is used to execute the program instructions in the memory to implement the method described in the first aspect or any possible implementation method thereof.

[0030] In a fourth aspect, the present application provides an electronic device comprising one or more display screens, one or more memories, and one or more processors; wherein the one or more memories store one or more programs; when the one or more processors execute the one or more programs, the electronic device executes the method described in the first aspect or any possible implementation method thereof.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer instructions, which, when executed on the electronic device, enables the electronic device to execute the method described in the first aspect or any possible implementation thereof.

[0032] In a sixth aspect, the present application provides a computer program product, which includes instructions for implementing the method described in the first aspect or any possible implementation thereof.

[0033] It can be understood that the effects that can be obtained in the second, third, fourth, fifth and sixth aspects can be referred to the description in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG1 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

[0035] FIG2 is a structural diagram of a VR system provided in an embodiment of the present application;

[0036] FIG3 is a structural diagram of a head-mounted device provided by the present application;

[0037] FIG4 is a structural diagram of another head-mounted device provided by the present application;

[0038] FIG5 is a schematic diagram of a software structure used by the head mounted device provided in this application;

[0039] FIG6 is a schematic diagram of an existing display method;

[0040] FIG7 is a schematic diagram of another existing display method;

[0041] FIG8 is a schematic diagram of the process of existing SOC transmission and display screen display;

[0042] FIG9 is an example of a display method provided by the present application;

[0043] FIG10 is an example of a display method provided by the present application;

[0044] FIG11 is a schematic diagram of the process of SOC transmission and display screen display provided by the present application;

[0045] FIG12 is another example of the display method provided by the present application;

[0046] FIG13 is another example of the display method provided by the present application;

[0047] FIG14 is another example of the display method provided by the present application;

[0048] FIG15 is a schematic diagram of a display device provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0050] To facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0051] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0052] In addition, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, where a, b, c can be single or multiple.

[0053] The processing method of the terminal device provided in the embodiment of the present application can also be called a method for identifying a third-party frame drawing thread, and can be applied to a terminal device with a display function. The terminal device can be a mobile phone, a tablet computer, a personal computer (PC), a smart screen, a car device, a wearable terminal device such as a smart watch, and can also be various teaching auxiliary tools (such as learning machines, early education machines), smart toys, portable robots, personal digital assistants (PDAs), augmented reality technology (AR) devices, virtual reality (VR) devices, etc., and can also be a device with mobile office functions, a device with smart home functions, a device with audio and video entertainment functions, a device that supports smart travel, etc. It should be understood that the embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0054] With the development of terminal display technology, the application scenarios of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies are increasing. Users can experience different effects such as VR, AR, or MR by wearing head-mounted display devices.

[0055] Refer to Figure 1, which is a schematic diagram of the application scenario provided by this application. As shown in Figure 1, the user can wear a head-mounted display device on his head to watch the content played on the head-mounted display device. For example, a user wearing a VR head-mounted display device (such as VR glasses, VR helmets, etc.) can see the VR game interface, and through gestures, handles, etc., can interact with the VR game interface as if they were in the game. For another example, a user wearing an AR transparent glasses head-mounted display device can see the real environment around him through the lenses of the glasses, and virtual objects can also be displayed on the lenses, so that the user can see both real objects and virtual objects.

[0056] VR technology is a form of human-computer interaction created with the help of computers and sensor technology. It integrates various scientific and technological advances, including computer graphics, computer simulation, sensor technology, and display technology, to create virtual environments. These environments consist of two- or three-dimensional computer-generated virtual objects that play dynamically in real time, simulating visual and other sensory experiences for the user, making them feel as if they are actually there. In addition to the visual perception generated by computer graphics, other senses, such as hearing, touch, force, movement, and even smell and taste, are also included, also known as multi-sensory perception. Furthermore, the user's head movements, eye movements, gestures, and other human movements can be detected. The computer processes the data corresponding to the user's movements, responds to them in real time, and provides feedback to the user's five senses, thus forming a virtual environment.

[0057] AR technology involves overlaying computer-generated virtual objects on real-world scenes, thereby enhancing the real world. In other words, AR technology involves capturing real-world scenes and then adding a virtual environment to them. Therefore, the difference between VR and AR is that VR creates a completely virtual environment, where users see only virtual objects; whereas AR overlays virtual objects on the real world, allowing users to see both real-world and virtual objects.

[0058] MR technology enhances the user experience by introducing real-world scene information (or real scene information) into a virtual environment, creating a bridge of interactive feedback between the virtual environment, the real world, and the user. Specifically, real objects are virtualized (for example, by using a camera to scan real objects for 3D reconstruction and generate virtual objects). This virtualized real object is then introduced into the virtual environment, allowing the user to see the real object in the virtual environment.

[0059] Below, for ease of understanding, the technical solution of this application is introduced by taking a user wearing a VR head-mounted display device as an example. However, it should be noted that the technical solution provided in the embodiments of this application is not limited to VR head-mounted display devices. For example, it can be applied to VR, AR or MR, or it can also be applied to other scenarios or electronic devices other than VR, AR or MR that need to display a three-dimensional environment to the user. The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0060] Referring to Figure 2, Figure 2 shows a schematic diagram of a VR system when presenting a VR effect to a user. As shown in Figure 2, the VR system includes a VR head-mounted display device 100 and a processing device 200. The VR system can be called a VR split machine. Specifically, the VR head-mounted display device 100 can be connected to the processing device 200. The connection between the VR head-mounted display device 100 and the processing device 200 includes a wired or wireless connection. The wireless connection can be Bluetooth (BT), traditional Bluetooth or low-power BLE Bluetooth, wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Zigbee, frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), or general 2.4G / 5G frequency band wireless communication connection, etc.

[0061] Among them, the processing device 200 can perform processing calculations. For example, the processing device 200 can generate an image and process the image, and then send the processed image to the VR head-mounted display device 100 for display.

[0062] In some embodiments, the processing device 100 can be, for example, a host (e.g., a VR host) or a server (e.g., a VR server). The VR host or VR server can be a device with significant computing power. For example, the VR host can be a mobile phone, tablet computer, laptop computer, or other device, and the VR server can be a cloud server, etc.

[0063] Among them, the VR head-mounted display device 100 includes a display screen, and the display screen of the VR head-mounted display device 100 can display images to the human eye.

[0064] In some embodiments, the VR head-mounted display device 100 includes only one display screen, and the user's two eyes view the images on the one display screen.

[0065] In some embodiments, the VR head-mounted display device 100 may be provided with two display screens (e.g., VR glasses, etc.). For example, in the VR system shown in FIG2 , the display screen 110 and the display screen 120 are enclosed within the VR glasses, so the arrows indicating the display screens 110 and 120 in FIG2 are represented by dotted lines.

[0066] Optionally, the VR head-mounted display device 100 also has functions such as image generation and processing, that is, the VR head-mounted display device 100 does not need the processing device 200 in Figure 1. Such a VR head-mounted display device 100 can be called a VR all-in-one machine.

[0067] 3 , which shows a schematic diagram of a VR head-mounted display device 100 including two display screens. As shown in FIG3 , the VR head-mounted display device 100 includes a display module 1 and a display module 2 .

[0068] Display module 1 is used to display images to the user's right eye and includes a display screen 110 and an optical device 130. Display screen 110 is located on the side of optical device 130 facing away from the right eye. Display module 2 is used to display images to the user's left eye and includes a display screen 120 and an optical device 140. Display screen 120 is located on the side of optical device 140 facing away from the left eye.

[0069] Specifically, in a scenario where a user wears the VR head-mounted display device 100, when the display screen 110 is displaying an image, the light emitted by the display screen 110 is converged to the user's right eye through the optical device 330; when the display screen 120 is displaying an image, the light emitted by the display screen 120 is converged to the user's left eye through the optical device 140.

[0070] It should be noted that the VR head-mounted display device 100 shown in Figure 3 may also include other components, for example, a support part 30 and a bracket 40, wherein the support part 30 is used to support the VR head-mounted display device on the bridge of the nose, and the bracket 40 is used to support the VR head-mounted display device on the ears to ensure that the VR head-mounted display device is stably worn.

[0071] Optionally, in some embodiments, optical device 130 or optical device 140 may be one or more of a reflective mirror, a transmissive mirror, or an optical waveguide, respectively, to increase the field of view. For example, optical device 130 or optical device 140 may be an eyepiece composed of multiple transmissive mirrors. Exemplary optical devices may be Fresnel lenses and / or aspherical lenses, etc., which are not limited in this embodiment of the present application. Generally, optical device 130 and optical device 140 are respectively aligned with the user's two eyes.

[0072] Optionally, in some embodiments, the optical device 130 and the optical device 140 are symmetrical with respect to the middle plane C. The symmetry can be, for example, strict symmetry or slight deviation. For example, the optical device 130 and the optical device 140 can be strictly symmetrical with respect to the middle plane C, or the optical device 130 and the optical device 140 can be basically symmetrical with respect to the middle plane C, and the basic symmetry can have a certain deviation, which is within a small range. Optionally, in some embodiments, the VR head-mounted display device 100 can be a bilaterally symmetrical structure, and the support portion 30 and / or the bracket 40 can be respectively bilaterally symmetrical with respect to the middle plane C, and the support portion 30 can fix the position of the human face, which is conducive to aligning the optical device 130 and the optical device 140 with the user's left eye and right eye, respectively.

[0073] It is understandable that the VR head-mounted display device 100 may further include more devices. Refer to Figure 4. Figure 4 is a structural diagram of a head-mounted display device 100 provided in an embodiment of the present application. The head-mounted display device may be a VR head-mounted display device, an AR head-mounted display device, an MR head-mounted display device, and the like. Taking a VR head-mounted display device as an example, as shown in Figure 4, the VR head-mounted display device 100 may include a processor 101, a memory 102, a sensor module 103 (for example, which can be used to obtain the user's posture, etc.), a microphone 104, a button 150, an input and output interface 160, a communication module 170, a camera 180, a battery 190, an optical display module 106, and an eye tracking module 105, etc.

[0074] The processor 101 is generally used to control the overall operation of the VR head-mounted display device 100 and may include one or more processing units. For example, the processor 101 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0075] Processor 101 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 101 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 101. If processor 101 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 101's latency, and thus improves system efficiency.

[0076] In some embodiments of the present application, the processor 101 can be used to control the optical focal length of the VR head-mounted display device 100. For example, the processor 101 can be used to control the optical focal length of the optical display module 106 to achieve the function of adjusting the optical focal length of the head-mounted display device 100. For example, the processor 101 can adjust the relative positions between the various optical devices (such as lenses, etc.) in the optical display module 106 so that the optical focal length of the optical display module 106 is adjusted, and then the position of the corresponding virtual image surface of the optical display module 106 when imaging the human eye can be adjusted. Thereby achieving the effect of controlling the optical focal length of the head-mounted display device 100.

[0077] In some embodiments, the processor 101 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0078] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 101 may include multiple I2C buses.

[0079] A UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 101 and the communication module 170. For example, the processor 101 communicates with the Bluetooth module in the communication module 170 via the UART interface to implement Bluetooth functionality.

[0080] The MIPI interface can be used to connect the processor 101 with the display screen in the optical display module 106 , the camera 180 and other peripheral devices.

[0081] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 101 with the camera 180, the display screen in the optical display module 106, the communication module 170, the sensor module 103, the microphone 104, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. In some embodiments, the camera 180 can capture images including real objects, and the processor 101 can fuse the images captured by the camera with virtual objects to obtain an image fused with the reality of the optical display module 106. In some embodiments, the camera 180 can also capture images including human eyes. The processor 101 performs eye tracking through the image.

[0082] The USB interface is an interface that complies with USB standard specifications, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the VR head-mounted display device 100, and can also be used to transmit data between the VR head-mounted display device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as mobile phones. The USB interface can be USB3.0, which is used to be compatible with high-speed display port (DP) signal transmission and can transmit high-speed video and audio data.

[0083] It should be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the head-mounted display device 100. In other embodiments of this specification, the head-mounted display device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0084] In addition, the VR head-mounted display device 100 may include a wireless communication function. For example, the VR head-mounted display device 100 may receive images from other electronic devices (such as a VR host) for display, or the VR head-mounted display device 100 may directly obtain data from a base station or other site. The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication function can be implemented through an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The VR head-mounted display device 100 may include multiple antennas, each of which can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0085] The mobile communication module can provide wireless communication solutions for the VR head-mounted display device 100, including solutions for second-generation (2G) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, and sixth-generation (6G) networks. The mobile communication module can include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves from an antenna, filter, amplify, and perform other processing on the received electromagnetic waves, and transmit them to a modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some of the functional modules of the mobile communication module can be provided in the processor 101. In some embodiments, at least some of the functional modules of the mobile communication module can be provided in the same device as at least some of the modules of the processor 101.

[0086] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, etc.) or displays an image or video through a display screen in the optical display module 106. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 101 and be set in the same device as the mobile communication module or other functional modules.

[0087] The wireless communication module can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the VR head-mounted display device 100. The wireless communication module can be one or more devices that integrate at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 101. The wireless communication module can also receive the signal to be sent from the processor 101, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0088] In some embodiments, the antenna of the VR head mounted display device 100 is coupled to the mobile communication module so that the VR head mounted display device 100 can communicate with a network and other devices via wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), 5G, 6G, BT, GNSS, WLAN, NFC, FM, and / or IR technology. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0089] The VR head-mounted display device 100 implements display functions through a GPU, an optical display module 106, and an application processor. The GPU is a microprocessor for image processing that connects the optical display module 106 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 101 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0090] The memory 102 can be used to store computer executable program code, which includes instructions. The processor 101 executes various functional applications and data processing of the VR head-mounted display device 100 by running the instructions stored in the memory 102. The memory 102 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the head-mounted display device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0091] The VR head-mounted display device 100 can implement audio functions through an audio module, a speaker, a microphone 104, a headphone interface, and an application processor. For example, music playback, recording, etc. The audio module is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module can also be used to encode and decode audio signals. In some embodiments, the audio module can be set in the processor 101, or some functional modules of the audio module can be set in the processor 101. The speaker, also known as a "speaker", is used to convert audio electrical signals into sound signals. The head-mounted display device 100 can listen to music or listen to hands-free calls through the speaker.

[0092] The microphone 104, also known as a "microphone" or "microphone," is used to convert sound signals into electrical signals. The VR head-mounted display device 100 may be provided with at least one microphone 104. In other embodiments, the VR head-mounted display device 100 may be provided with two microphones 104, which, in addition to collecting sound signals, may also implement noise reduction. In other embodiments, the VR head-mounted display device 100 may also be provided with three, four, or more microphones 104 to collect sound signals, reduce noise, identify sound sources, implement directional recording functions, and the like.

[0093] The headphone jack is used to connect wired headphones. The headphone jack can be a USB port or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard port or a Cellular Telecommunications Industry Association of the USA (CTIA) standard port.

[0094] In some embodiments, the VR head-mounted display device 100 may include one or more buttons 150, which can control the VR head-mounted display device and provide the user with the function of interacting with the VR head-mounted display device 100. The button 150 can be in the form of a button, a switch, a dial, and a touch or near-touch sensing device (such as a touch sensor). Specifically, for example, the user can turn on the optical display module 106 of the VR head-mounted display device 100 by pressing a button. The button 150 includes a power button, a volume button, etc. The button 150 can be a mechanical button. It can also be a touch button. The head-mounted display device 100 can receive button input and generate key signal input related to user settings and function control of the head-mounted display device 100.

[0095] In some embodiments, the VR head-mounted display device 100 may include an input / output interface 160, which may connect other devices to the VR head-mounted display device 100 through appropriate components. The components may include, for example, audio / video jacks, data connectors, and the like.

[0096] The optical display module 106 is used to present images to the user under the control of the processor 101. The optical display module 106 can convert real pixel images into virtual images projected near the eye through one or more optical devices such as reflectors, transmissive mirrors, or optical waveguides, enabling a virtual interactive experience or a combined virtual and real interactive experience. For example, the optical display module 106 receives image data information sent by the processor 101 and presents the corresponding image to the user.

[0097] For example, the optical display module 106 may have the structure shown in FIG. 3 . For example, the optical display module 106 includes two display screens, namely, a display screen 110 and a display screen 120. For another example, the optical display module 106 includes a display module 1 and a display module 2. Display module 1 includes a display screen 110 and an optical device 130, and display module 2 includes a display screen 120 and an optical device 140.

[0098] In some embodiments, the VR head-mounted display device 100 may further include an eye tracking module 105, which is used to track the movement of the human eye and thereby determine the gaze point of the human eye. For example, the pupil position can be located by image processing technology, the pupil center coordinates can be obtained, and then the person's gaze point can be calculated. In some embodiments, the eye tracking system can determine the user's gaze point position (or determine the user's line of sight direction) by methods such as a video eye diagram method, a photodiode response method, or a pupil corneal reflection method, thereby achieving eye tracking of the user.

[0099] It should be noted that in some embodiments, a corresponding eye tracking system can be set up for each of the user's eyes to track the eye movements of both eyes synchronously or asynchronously. In other embodiments, an eye tracking system can be set up only near one of the user's eyes. The eye tracking system can obtain the gaze direction of the corresponding eye and determine the gaze direction or gaze point position of the user's other eye based on the relationship between the two eyes' gaze points (e.g., when a user observes an object through both eyes, the gaze points of the two eyes are generally close or the same) and the distance between the user's eyes.

[0100] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the VR head-mounted display device 100. In other embodiments of this specification, the VR head-mounted display device 100 may include more or fewer components than those shown in Figure 4, or combine or separate certain components, or arrange the components differently, which is not limited in the embodiment of the present application.

[0101] FIG5 is a block diagram of the software structure of the VR head-mounted display device 100 according to an embodiment of the present application. As shown in FIG5 , the software structure of the VR head-mounted display device 100 can be a layered architecture. For example, the software can be divided into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the layers are divided into five layers, from top to bottom, namely, the application layer 510, the application framework layer (framework, FWK) 520, the Android runtime (Android runtime) and system library 530, the kernel layer 540, and the hardware layer 550.

[0102] The application layer 510 may include a series of application packages. For example, as shown in FIG5 , the application layer includes a gallery application, a game application, and the like.

[0103] The application framework layer 520 provides an application programming interface (API) and a programming framework for the applications of the application layer. The application framework layer may include some predefined functions. As shown in Figure 5, the application framework layer may include a resource manager, a view system, etc. For example, the view system 5 includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a message notification icon may include a view for displaying text and a view for displaying pictures. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

[0104] The Android runtime consists of core libraries and a virtual machine (VM). The Android runtime is responsible for scheduling and management of the Android system. The core libraries consist of two parts: one for Java-based functions and the other for the Android core library. The application layer and application framework layer run in the VM. The VM executes Java files from the application layer and application framework layer as binary files. The VM is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0105] The system library can include multiple functional modules. For example: surface manager 5, media libraries, three-dimensional graphics processing library, such as OpenGL ES, two-dimensional graphics engine, such as SGL, etc. Among them, the surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc. The two-dimensional graphics engine is a drawing engine for two-dimensional drawing.

[0106] The kernel layer 540 is a layer between hardware and software and includes at least a display driver, a camera driver, an audio driver, a sensor driver, and the like.

[0107] The hardware layer may include display screens and various sensor modules, such as accelerometers, gravity sensors, touch sensors, etc.

[0108] It is understandable that the software structure shown in FIG5 does not constitute a specific limitation on the software structure of the VR head-mounted display device 100. For example, in other embodiments, the software structure of the VR head-mounted display device 100 may include more or fewer layers than those shown in FIG5 , such as an adaptation layer located between the application framework layer 520 and the Android runtime and system library 530 to achieve adaptation between the upper layer (i.e., the application framework layer) and the lower layer (i.e., the system library), for example, to achieve interface matching between the upper layer and the lower layer to ensure data communication between the upper layer and the lower layer.

[0109] As can be seen from the above description, when a head-mounted display device displays an image on a display screen, it must first receive image data sent by a processing device. Alternatively, it can be expressed this way: when a head-mounted display device displays an image to a user through a display screen, it must be driven by a processing device.

[0110] In some embodiments, the display screen is included in the head-mounted device, and the processing device and the head-mounted display device are deployed separately as shown in FIG2 .

[0111] In some embodiments, the processing device and the display screen are both integrated into the head-mounted display device. In this case, the head-mounted display device can also be called an all-in-one device.

[0112] Generally, head-mounted display devices have high display requirements, which are mainly reflected in the following three aspects: 1) higher resolution; 2) larger field of view (FoV); 3) higher refresh rate and lower latency.

[0113] Here, resolution refers to the resolution of the image output by each display screen.

[0114] Among them, the refresh rate refers to the number of times the display screen is refreshed per second, also known as the output frame rate, and the unit is usually Hertz (Hz).

[0115] For example, for a certain head-mounted display device, its display index (also called output capability) is 1800x1920x2@90Hz, which means that when a single display screen is displayed, the image resolution is 1800x1920 and the refresh rate is 90HZ (that is, 90 frames are output per second).

[0116] In some embodiments, silicon-based organic light emitting diodes (OLEDs) can achieve high resolution in a compact and lightweight form factor due to their ultra-high PPI (pixels per inch) characteristics. They also offer advantages such as a wide color gamut and high brightness. Therefore, silicon-based OLEDs can be used for display to achieve higher resolutions. For example, when using silicon-based OLEDs for display, each display can achieve a 4K resolution.

[0117] However, the development of processing equipment that drives display screens is slower than that of display screens, which is mainly reflected in the insufficient transmission bandwidth of processing equipment, resulting in the display screen's display effect not being fully utilized.

[0118] In some embodiments, the processing device is a system on chip (SOC).

[0119] For example, if the output capability of each display is 3840*3840@90Hz, meaning each display outputs an image with a resolution of 3840*3840 and a refresh rate of 90Hz (i.e., 90 frames per second), then to achieve optimal display quality on each display, the bandwidth required for each display is 3840*3840*90Hz*24bits / (1024^3)=29.66Gbps. However, if the maximum transmission bandwidth supported by the SOC is 17Gbps / panel, the SOC transmission bandwidth cannot meet the display's requirements, resulting in an inability to transmit a 3840*3840*90Hz image to the display.

[0120] Currently, when the SOC transmission bandwidth cannot meet the needs of the display, there are two commonly used display solutions:

[0121] The first commonly used display solution is: while ensuring the refresh rate remains unchanged, the resolution of the image output by the SOC is made smaller than the resolution of the image output by each display to ensure the maximum transmission bandwidth supported by the SOC. The head-mounted display device then upsamples the image transmitted by the SOC to obtain an image with the same resolution as the display.

[0122] In some embodiments, the head-mounted display device may up-sample the image transmitted by the SOC by a display driver IC (DDIC) in the head-mounted display device.

[0123] For example, assuming the maximum transmission bandwidth supported by the SOC is 17 Gbps / panel and each display has an output capability of 3840*3840@90Hz, if the SOC meets the transmission bandwidth requirement of less than 17 Gbps / panel, as shown in Figure 6, the SOC can output an image to the display at 2880*2880@90Hz. The DDIC then upsamples the image by a factor of 1.33 to display a 3840*3840 image on the display.

[0124] It should be noted that 2880*2880 in FIG6 is only an example and does not constitute a limitation of the present application.

[0125] For example, in some embodiments, the resolution of the image output by the SOC can be x1*x1, as long as x1*x1*90Hz*24bits / (1024^3) is less than or equal to 17Gbps; accordingly, the DDIC upsamples the image transmitted by the SOC by 3840 / x1 in both the horizontal and vertical directions to display a 3840*3840 image on the display screen.

[0126] For example, in some embodiments, the resolution of the image output by the SOC can be x1*y1, as long as x1*y1*90Hz*24bits / (1024^3) is less than or equal to 17Gbps; accordingly, after receiving the image transmitted by the SOC, the DDIC performs upsampling by 3840 / x1 in the horizontal direction and by 3840 / y1 in the vertical direction to display a 3840*3840 image on the display screen.

[0127] The second common display solution is to reduce the refresh rate of the SOC, that is, reduce the frequency of the SOC output image. For example, if the SOC meets the transmission bandwidth of less than 17Gbps / panel, the SOC can output images to each display at 3840*3840@50Hz. Accordingly, each display outputs 50 frames per second.

[0128] Similarly, the number 50 in this example is only an example and does not constitute a limitation of this application. For example, in some embodiments, the output frame rate of the SOC can be set to z1, as long as 3840*3840*z1*24bits / (1024^3) is less than or equal to 17Gbps.

[0129] It should be understood that the first approach may result in a problem of reduced clarity of the image output by each display screen, while the second implementation approach may result in a problem of reduced smoothness during display.

[0130] To this end, people in this field have proposed another display solution, also known as a foveation rendering solution. The basic concept of this solution is: since the FOV of the human eye is very small, for example, it is in the range of 20 degrees to 30 degrees. Therefore, this display solution includes: the image output by the SOC is composed of two parts stacked and spliced ​​together, wherein the upper part is the gaze area part that needs to be displayed on the display screen, and the lower part is the background area part that needs to be displayed on the display screen, and then the image is transmitted to the display screen; accordingly, when the head-mounted display device displays through the display screen, it does not upsample the gaze area part, but upsamples the background area part, combines the human eye tracking technology to identify the coordinate position of the gaze area part on the display screen, and displays the gaze area part at the corresponding coordinate position to ensure the clarity of the gaze area.

[0131] It can be seen that this display solution ensures the SOC transmission bandwidth by reducing the resolution of the SOC output image, while ensuring that the clarity of the gaze area and the refresh rate are not affected.

[0132] As an example, as shown in Figure 7, the SOC output image includes two parts, the upper part is the gaze area part, and the lower part is the background area part. The resolution of the gaze area part is 1536*1536, and the resolution of the background area part is also 1536*1536. Then the SOC transmits the image in a top-to-bottom transmission order. Accordingly, when displayed on the display screen, after the specified position of the gaze area part on the display screen is identified by combining the human eye tracking technology (the center position of the gaze area on the display screen in the example of Figure 7), the gaze area part is not upsampled and is displayed at the specified position, while the background area is upsampled 2.5 times. That is, when the display screen is displayed, the resolution of the gaze area part is still 1536*1536, ensuring the clarity of the gaze area part.

[0133] It should be noted that the 1536*1536 resolution of the gaze area and background area output by the SOC in Figure 7 is only an example and does not constitute a limitation. For example, assuming that the maximum output frame rate of the display is z2HZ and the maximum bandwidth supported by the SOC is S Gbps / panel, the resolution of the gaze area and the background area output by the SOC can also be x2*x2, as long as x2*2x2*z2*24bits / (1024^3) is less than or equal to S Gbps / panel. When then transmitted to the display for display, when the display upsamples the background area, the upsampling ratio in the horizontal and vertical directions can be changed to 3840 / x2.

[0134] However, when wearing a head-mounted display device using the above-mentioned foveated rendering solution, it was found that users were at risk of dizziness, which seriously affected the user experience.

[0135] After analysis, it was found that the reasons why users are at risk of dizziness when wearing the headset are as follows: In the above-mentioned gaze point rendering scheme, when the head-mounted display device displays each frame of image, it starts to judge whether each pixel of the display screen displays data of the gaze area or the background area after the SOC has completed transmitting the entire gaze area part and starts transmitting the background area part, so as to start displaying line by line. That is, in the above-mentioned gaze point rendering scheme, the display screen must start displaying only after the data of the gaze area part is transmitted. As a result, a fixed display delay is generated when each frame of image is displayed, which makes the user at risk of dizziness when wearing the headset.

[0136] For example, taking Figure 7 as an example, the delay required to transmit the data in the gaze area is 1 / 2.5 = 0.4 frames. If the refresh rate of the display is 90 Hz, that is, the time required to display a frame of image is 11.1 ms, then the display delay of each frame is 4.4 ms (11.1 * 0.4), which may cause dizziness to the user.

[0137] It should be understood that under this solution, if the resolution of the gaze area output by the SOC is greater, the display delay of the entire device will be greater.

[0138] For ease of understanding, Figure 8 shows a schematic diagram of the SOC transmission and screen display process, where the resolution of the gaze area and background area output by the SOC is 1536*1536, and the resolution of the display output image is 3840*3840. As shown in Figure 8, the horizontal axis represents time, and the vertical axis represents the number of screen rows. Line 1 represents the value range of the top pixel of the gaze area, and Line 2 represents the value range of the bottom pixel of the gaze area. It takes 0.4 frames to transmit the data of the gaze area for each frame of the image. After the data of the gaze area is transmitted, the SOC begins to transmit the data of the background area (represented by line 3 in the figure). Accordingly, the head-mounted display screen begins to display line by line (represented by line 4 in the figure).

[0139] In view of this, the embodiments of the present application provide a display method, so as to achieve that when a head-mounted display device displays on a display screen, while ensuring the clarity and smoothness of the gaze area, it can also reduce the display delay of the entire device, thereby avoiding the risk of dizziness that may occur when the user wears the head-mounted display device and improving the user experience.

[0140] Below, an example is first used to illustrate the display solution provided by the embodiment of the present application.

[0141] In this example, a commonly used display screen with an output capability of 3840*3840*90HZ and a maximum transmission bandwidth supported by the SOC of 17Gbps / panel is used as an example for description.

[0142] Specifically, the display method proposed in the embodiment of the present application is conceived as follows: aligning the image output by the SOC and the image output by the display screen. For example, as shown in FIG9 , the image can be divided into the following regions according to the user's demand for the display clarity of the entire device: region 5, region 1 located at the upper left corner of region 5, region 2 located above region 5, region 3 located at the upper right corner of region 5, region 4 located to the left of region 5, region 6 located to the right of region 5, region 7 located at the lower left corner of region 5, region 8 located below region 5, and region 9 located at the lower right corner of region 5. Then the images output by the SOC and the display screen both include the above 9 regions, and for the same region, the position in the image output by the SOC and the position in the image output by the display screen are the same. For example, the position of region 5 included in the image output by the SOC and the position of region 5 included in the image output by the display screen are the same, and the region 1 included in the image output by the SOC and the region 1 included in the image output by the display screen are both in the upper left corner of their respective regions 5.

[0143] Among them, area 5 can be a central area, area 1, area 3, area 7, and area 9 can be corner areas, and area 2, area 4, area 6, and area 8 can be transition areas. In this case, area 5 can be called central area 5, area 1, area 3, area 7, and area 9 can be called upper left corner area 1, upper right corner area 3, lower left corner area 7, and lower right corner area 9, respectively, and area 2, area 4, area 6, and area 8 can be transition areas, which can be called upper transition area 2, left transition area 4, right transition area 6, and lower transition area 8, respectively.

[0144] Specifically, in the embodiment of the present application, the image output by the SOC (which can be considered as the first image) is shown in Figure 9. The resolution of area 5 is A2*B2, the resolution of areas 1, 3, 7, and 9 is A1*B1, the resolution of areas 4 and 6 is A1*B2, and the resolution of areas 2 and 8 is A2*B1. Among them, A1+A2+A1=m, B1+B2+B1=n, m represents the horizontal resolution of the first image output by the SOC, and n represents the vertical resolution of the image output by the SOC.

[0145] It should be noted that in the embodiments of this application, there is no restriction on the values ​​of m and n, as long as they can keep the SOC transmission bandwidth less than 17 Gbps / panel. In other words, the values ​​of m and n must satisfy the following: m*n*90*24 bits / (1024^3) is less than or equal to 17 Gbps.

[0146] Specifically, in an embodiment of the present application, when it is necessary to display on a display screen, for an image transmitted by the SOC, at least one area transmitted by the SOC will be upsampled so that a 3840*3840 image (that is, it can be considered as a second image) is displayed on the display screen.

[0147] It should be noted that the embodiments of the present application do not limit the specific upsampling ratio used, as long as it satisfies: A1*q1+A2*q2+A1*q1=3840, B1*p1+B2*p2+B1*p1=3840. Among them, q1 is the upsampling ratio used in the horizontal direction when upsampling regions 1, 4, 7, 3, 6, and 9 in the SOC image, q2 is the upsampling ratio used in the horizontal direction when upsampling regions 2, 5, and 8 in the SOC image, p1 is the upsampling ratio used in the vertical direction when upsampling regions 1, 2, 3, 7, 8, and 9 in the SOC image, and p2 is the upsampling ratio used in the vertical direction when upsampling regions 4, 5, and 6 in the SOC image.

[0148] In an embodiment of the present application, when an image transmitted by the SOC is displayed on a display screen, upsampling of the gaze area transmitted by the SOC may not be performed to ensure clarity of the gaze area.

[0149] For example, if the gaze area in FIG9 is area 5, then q2 and p2 may be set to 1 to ensure the clarity of area 5 when displayed on the display screen.

[0150] In some embodiments, when upsampling is performed on the display screen for areas other than the gaze area transmitted by the SOC, a larger upsampling factor is used when upsampling for areas farther away from the gaze area.

[0151] Still taking FIG. 9 as an example, as an example, assuming that the gaze area is area 5, then q1 can be 2, q2 can be 1, p1 can be 2, and p2 can be 1.

[0152] For example, when q1 is 2, q2 is 1, p1 is 2, and p2 is 1, the resolutions of the various regions in the images output by the SOC and the display are as shown in FIG10. Referring to FIG10, in the image output by the SOC, the resolution of region 5 is 1536*1536, the resolution of region 1 is 1152 / 2*1152 / 2, the resolution of region 2 is 1536*1152 / 2, the resolution of region 3 is 1152 / 2*1152 / 2, the resolution of region 4 is 1152 / 2*1536, the resolution of region 6 is 1152 / 2*1536, the resolution of region 7 is 1152 / 2*1152 / 2, the resolution of region 8 is 1536*1152 / 2, and the resolution of region 9 is 1152 / 2*1152 / 2. Accordingly, when the head-mounted display device displays on the display screen, the horizontal and vertical directions of area 1 of the SOC transmission are upsampled by a factor of 2, the horizontal direction of area 2 of the SOC transmission is not upsampled but the vertical direction is upsampled by a factor of 2, the horizontal and vertical directions of area 3 of the SOC transmission are upsampled by a factor of 2, the horizontal direction of area 4 of the SOC transmission is upsampled by a factor of 2 but the vertical direction is not upsampled, the horizontal and vertical directions of area 5 of the SOC transmission are not upsampled, the horizontal direction of area 6 of the SOC transmission is upsampled by a factor of 2 but the vertical direction is not upsampled, the horizontal and vertical directions of area 7 of the SOC transmission are upsampled by a factor of 2, the horizontal direction of area 8 of the SOC transmission is not upsampled but the vertical direction is upsampled by a factor of 2, and the horizontal and vertical directions of area 9 of the SOC transmission are upsampled by a factor of 2.

[0153] It should be understood that by using a larger upsampling ratio when upsampling areas farther away from the gaze area, the clarity of the gaze area is the highest, and the clarity of the gaze area is greater than that of the transition area, and the clarity of the transition area is greater than that of the corner area.

[0154] As can be seen, in the display method provided by the embodiments of the present application, the SOC ensures smooth display even when the SOC's transmission bandwidth is insufficient by reducing the resolution of the image it outputs. Furthermore, because the focus area included in the image displayed on the display screen is an area that has not been upsampled from the central area transmitted by the SOC, the clarity of the focus area is guaranteed.

[0155] Optionally, in some embodiments, when the image transmitted by the SOC is displayed on a display screen, the gaze area may also be upsampled using a lower upsampling ratio to ensure the clarity of the gaze area; and when other areas other than the gaze area transmitted by the SOC are displayed on the display screen, the upsampling ratio used is greater than the upsampling ratio used for the gaze area, and the farther away the area is from the gaze area, the greater the upsampling ratio used when upsampling.

[0156] In addition, in the display method provided in the embodiment of the present application, since the image output by the SOC and the image output by the display screen are aligned in position, there is no need to first transmit the gaze area before starting to display. Instead, the SOC can transmit and display the display screen at the same time (as shown in Figure 11), thereby reducing the display delay.

[0157] It should be noted that FIG9 illustrates the display method of the embodiment of the present application by taking the common fixation area being the center area (i.e., the fixation area of ​​the human eye is in the center area of ​​the display screen) as an example. However, it should be understood that sometimes the fixation area may not be the center area.

[0158] Therefore, for the display method provided by the embodiments of the present application, in some embodiments, when the head-mounted display device does not have a gaze point function, the gaze area can be considered to be the central area. When the head-mounted display device has a gaze point function, the gaze area may include any one or more of the multiple divided areas (that is, the area that is not upsampled may be any one or more of the multiple divided areas), such as any one or more of the above-mentioned 9 areas. In other words, when the head-mounted display device has a gaze point function, the area that is not upsampled is the gaze area, which changes with eye movement, and when the head-mounted display device does not have a gaze point function, the area that is not upsampled is the central area.

[0159] For example, taking Figure 9 as an example, if the gaze area of ​​the human eye becomes the upper left corner area 1, then when displayed on the display screen, the upper left corner area 1 will not be upsampled, and for other areas, upsampling can be performed under the constraints of: A1+A2+A1=m, B1+B2+B1=n, m*n*90*24bits / (1024^3) is less than or equal to 17Gbps, and A1*q1+A2*q2+A1*q1=3840, B1*p1+B2*p2+B1*p1=3840.

[0160] In the above, the display method provided by the embodiment of the present application is illustrated by taking the display output capacity of 3840*3840*90HZ, the maximum transmission bandwidth supported by the SOC as 17Gbps / panel, and the user's gaze area being located in the center area of ​​the display when viewing the display as an example.

[0161] It should be noted that the above division of the image output by the SOC into 9 areas is only an example and does not constitute a limitation of the present application.

[0162] For example, in some embodiments, the number of regions may be even greater.

[0163] Exemplarily, the image output by the SOC may also include 25 areas as shown in FIG12 (areas 21 to 35 in FIG12 ), where A3+A4+A5+A6+A7=m2, and B3+B4+B5+B6+B7=n2.

[0164] Similarly, the embodiments of the present application do not limit the values ​​of m2 and n2, and the specific values ​​of A3, A4, A5, A6, A7, B3, B4, B5, B6, B7, q3, q4, q5, q6, q7, p3, p4, p5, p6, and p7, as long as the relationship between them can satisfy the following relationship: m2*n2*90*24bits / (1024^3) is less than or equal to 17Gbps, A3*q3+A4*q4+A5*q5+A6*q6+A7*q7=3840, and B3*p3+B4*p4+B5*q5+B6*p6+B7*p7=3840.

[0165] Accordingly, when displaying on the display screen, for the image transmitted by the SOC, the gaze area is not upsampled, and at least one other area except the gaze area is upsampled using the corresponding upsampling ratio, so that a 3840*3840 image is displayed on the display screen.

[0166] For example, if the user's gaze area is area 23, then q5 and p5 can be set to 1 to ensure the clarity of the gaze area.

[0167] Optionally, in this example, when areas other than area 23 of the SOC transmission are displayed by upsampling, a larger upsampling ratio is used when upsampling an area farther away from area 23 .

[0168] For example, as shown in FIG13 , q3 takes a value of 6, q4 takes a value of 3, q5 takes a value of 1, q6 takes a value of 3, q7 takes a value of 6, p3 takes a value of 6, p4 takes a value of 3, p5 takes a value of 1, p6 takes a value of 3, and p7 takes a value of 6. Accordingly, when the head-mounted device displays on the display screen, the horizontal and vertical directions of the SOC transmission areas 11, 15, 31, and 35 are upsampled by a factor of 6, the horizontal direction of the SOC transmission areas 12, 14, 32, and 34 are upsampled by a factor of 3 and the vertical direction is upsampled by a factor of 6, the horizontal direction of the SOC transmission areas 16, 20, 26, and 30 are upsampled by a factor of 6 and the vertical direction is upsampled by a factor of 3, and the horizontal direction of the SOC transmission areas 13 and 33 are not upsampled, but the vertical direction is upsampled by a factor of 6. Upsampling: the horizontal and vertical directions of areas 17, 19, 27, and 29 of SOC transmission are upsampled by 3 times, the horizontal direction of areas 21 and 25 of SOC transmission are upsampled by 6 times, and the vertical direction is not upsampled. The horizontal direction of areas 18 and area 28 of SOC transmission are not upsampled, but the vertical direction is upsampled by 3 times. The horizontal direction of areas 22 and area 24 of SOC transmission are upsampled by 3 times, but the vertical direction is not upsampled. No upsampling is performed on the horizontal and vertical directions of area 23 of SOC transmission.

[0169] For another example, in some embodiments, the number of regions may be even smaller.

[0170] Exemplarily, the image output by the SOC may also include four regions as shown in FIG14 (regions 130 to 133 in FIG14 ), where A8+A9=m3 and B8+B9=n3.

[0171] Similarly, the embodiments of the present application do not limit the values ​​of m3 and n3, and the specific values ​​of A8, A9, B8, B9, q8, q9, p8, and p9, as long as the relationship between them can satisfy the following relationship: m3*n3*90*24bits / (1024^3) is less than or equal to 17Gbps, A8*q8+A9*q9=3840, and B8*p8+B9*p9=3840.

[0172] Accordingly, when the head-mounted device displays on the display screen, for the image transmitted by the SOC, the gaze area is not upsampled, and at least one other area other than the gaze area is upsampled using the corresponding upsampling ratio, so that a 3840*3840 image is displayed on the display screen.

[0173] For example, if the user's gaze area is area 132, then q8 and p9 can be set to 1 to ensure the clarity of the gaze area.

[0174] Optionally, in this example, when areas other than area 132 transmitted by the SOC are displayed by upsampling, a larger upsampling ratio is used when upsampling an area farther away from area 132 .

[0175] It should be noted that the above description is based on an example of a display screen with an output capacity of 3840*3840*90HZ and a maximum transmission bandwidth supported by the SOC of 17Gbps / panel. However, it should be understood that this example does not constitute a limitation of this application. For example, it can be extended to any scenario where a display screen displays data transmitted by the SOC.

[0176] Specifically, assuming that the display screen that displays images for the user is called a first display screen, and the first display screen is included in an electronic device, the display method provided in an embodiment of the present application includes: displaying a second image on the first display screen based on the first image output by the SOC.

[0177] Specifically, the resolution of the first image output by the SOC is smaller than the resolution of the second image output on the first display screen. It should be understood that the resolution of the first image output by the SOC is smaller than the resolution of the second image output by the first display screen, which can ensure the refresh rate when the maximum bandwidth supported by the SOC is insufficient to drive the first display screen.

[0178] Specifically, the first image output by the SOC is divided into K areas (referred to as K first areas). Accordingly, when displaying, the electronic device will not upsample the content of the gaze area (also referred to as the first target area) transmitted by the SOC to ensure the clarity of the gaze area, but will upsample each area other than the gaze area transmitted by the SOC using a corresponding upsampling ratio (including a horizontal upsampling ratio and / or a vertical upsampling ratio) to obtain K second areas corresponding to the K first areas.

[0179] That is, in the embodiment of the present application, the second image displayed on the first display screen includes K second regions, the K second regions include a second target region, the content of the second target region is the same as the content of the first target region and the position of the second target region in the image to which they belong is the same. Furthermore, the second regions other than the second target region in the Kth second region are obtained based on the corresponding first region transmitted by the SOC using the corresponding upsampling ratio.

[0180] More specifically, in an embodiment of the present application, different first areas among the K first areas correspond to different second areas among the K second areas, and the position of each first area in the first image is the same as the position of the corresponding second area in the second image; or in other words, different first areas among the K first areas correspond to different second areas among the K second areas, and the positional relationship between each first area and the gaze area in the first image is the same as the positional relationship between the corresponding second area and the gaze area in the second image. Or in other words, in an embodiment of the present application, the K first areas correspond one-to-one with the K second areas, and the positions of a first area in the first image and its corresponding second area in the second image are the same in their respective images. Or in other words, in an embodiment of the present application, the areas in the first image output by the SOC and the second image output by the first display screen are aligned in position.

[0181] It should be understood that, through this approach, there is no need to first complete the transmission of the first target area before starting to display, and the SOC is able to transmit data while displaying on the first display screen, thereby reducing display delay.

[0182] Furthermore, when displaying the second image on the first display screen based on the first image, a larger upsampling ratio is used when upsampling a first region farther from the first target region among the K first regions. In other words, when displaying the second image on the first display screen based on the first image, a larger upsampling ratio is used when obtaining a second region farther from the second target region among the K second regions based on the corresponding first region.

[0183] It is understandable that in this implementation, because when displaying the second image on the first display screen based on the first image, the upsampling ratio used when upsampling the first region farther from the first target region among the K first regions is greater, the K second regions displayed on the first display screen have the following characteristics: the second regions farther from the second target region in the second image are less clear, while the second regions closer to the second target region are clearer. In other words, in this implementation, when displaying the K second regions on the first display screen, the clarity of the second regions at different distances from the second target region is different.

[0184] In some embodiments, the electronic device further includes a second display screen. In this scenario, the method of the embodiment of the present application further includes: displaying a fourth image on the second display screen based on the third image output by the SOC, wherein the resolution of the third image is smaller than the resolution of the fourth image, the third image includes K third areas, the fourth image includes K fourth areas, different third areas in the K third areas correspond to different fourth areas in the K fourth areas, and the position of each third area in the third image is the same as the position of the corresponding fourth area in the fourth image; wherein the resolution of the third target area in the K third areas is the same as the resolution of the corresponding fourth target area in the K fourth areas, and the content of the third target area is the same as the content of the fourth target area, and the content of each fourth area in the K fourth areas except the fourth target area is obtained by upsampling the content of the corresponding third area.

[0185] Among them, this solution can be compared to the display method of the first display screen in the first aspect, and will not be repeated here.

[0186] It should be noted that the embodiments of the present application do not limit the specific form between the SOC and the electronic device.

[0187] For example, in some embodiments, the SOC is included in an electronic device.

[0188] For another example, in another possible implementation, the SOC is not included in the electronic device, that is, the SOC and the electronic device are independent of each other.

[0189] Figure 15 is a schematic diagram of the structure of a processing device of a terminal device provided in another embodiment of the present application. As shown in Figure 15, processing device 1500 may include a processor 1501 and an interface circuit 1502. Processor 1501 and interface circuit 1502 are coupled to each other. It is understood that interface circuit 1502 may be a transceiver or an input / output interface. Optionally, processing device 1500 may also include a memory 1503 for storing instructions executed by processor 1501, storing input data required by processor 1501 to execute instructions, or storing data generated after processor 1501 executes instructions.

[0190] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0191] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also be present in a network device or a terminal device as discrete components.

[0192] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive.

[0193] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0194] It is understood that the various numbers used in the embodiments of this application are merely for ease of description and are not intended to limit the scope of the embodiments of this application. The order of the sequence numbers of the above-mentioned processes does not necessarily imply a specific order of execution; the order of execution of the processes should be determined by their functions and inherent logic.

Claims

1. A display method, characterized in that: Applied to an electronic device, the electronic device includes a first display screen, and the method includes: When the first display screen receives the first image, the first display screen displays the second image, the resolution of the first image is smaller than the resolution of the second image, the first image includes K first areas, the second image includes K second areas, different first areas among the K first areas correspond to different second areas among the K second areas, and the position of each first area in the first image is the same as the position of the corresponding second area in the second image; The K first areas include a first target area, the K second areas include a second target area corresponding to the first target area, the first target area and the second target area have the same resolution, and the resolution of each of the K second areas except the second target area is higher than the resolution of the corresponding first area.

2. The method according to claim 1, characterized in that Each second region among the K second regions except the second target region is obtained by upsampling the first region corresponding to the respective second regions. The farther the second region among the K second regions is from the second target region, the greater the upsampling ratio corresponding to the second region.

3. The method according to claim 1 or 2, characterized in that: The electronic device also includes a second display screen, and the method further includes: When the second display screen receives the third image, the second display screen displays a fourth image, the resolution of the third image is smaller than the resolution of the fourth image, the third image includes K third areas, the fourth image includes K fourth areas, different third areas among the K third areas correspond to different fourth areas among the K fourth areas, and the position of each third area in the third image is the same as the position of the corresponding fourth area in the fourth image; The K third areas include a third target area, the K fourth areas include a fourth target area corresponding to the third target area, the third target area and the fourth target area have the same resolution, and the resolution of each fourth area among the K fourth areas except the fourth target area is higher than the resolution of the corresponding third area.

4. The method according to claim 3, characterized in that The first image received by the first display screen is an image output by a system-on-chip (SOC).

5. The method according to claim 4, characterized in that The SOC is included in the electronic device.

6. The method according to claim 4, characterized in that The SOC is not included in the electronic device.

7. The method according to any one of claims 1 to 6, characterized in that Both the first image and the second image can be divided into nine areas in three rows and three columns: a central area located at the center of the image, an upper left corner area located at the upper left corner of the central area, an upper transition area located above the central area, an upper right corner area located at the upper right corner of the central area, a left transition area located to the left of the central area, a right transition area located to the right of the central area, a lower left corner area located at the lower left corner of the central area, a lower transition area located below the central area, and a lower right corner area located at the lower right corner of the central area; The first target area is the central area of ​​the first image, the second target area is the central area of ​​the second image, and the first target area and the second target area have the same resolution, including: the central area of ​​the second image has the same resolution as the central area of ​​the first image.

8. The method according to claim 7, characterized in that Based on upsampling the upper left corner area in the first image by q1 times in the horizontal direction and by p1 times in the vertical direction, the upper left corner area in the second image is obtained; based on upsampling the left transition area in the first image by q1 times in the horizontal direction and without upsampling in the vertical direction, the left transition area in the second image is obtained; based on not upsampling the upper transition area in the horizontal direction and upsampling it by p1 times in the vertical direction in the upper transition area in the first image, the upper transition area in the second image is obtained, wherein both q1 and p1 are greater than 1.

9. The method according to any one of claims 1 to 8, characterized in that The electronic device is a head-mounted display device.

10. A display device, characterized in that: include: Memory and processor; The memory is used to store program instructions; The processor is configured to execute program instructions in the memory to implement the method according to any one of claims 1 to 9.

11. An electronic device, characterized in that: It includes one or more display screens, one or more memories, and one or more processors; wherein the one or more memories store one or more programs; it is characterized in that when the one or more processors execute the one or more programs, the electronic device executes the display method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the display method as claimed in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises instructions for implementing the method according to any one of claims 1 to 9.

Images