Display method and electronic equipment

Abstract

The invention provides a display method and electronic equipment. The display method comprises the steps that the electronic device displays a second image on a first display screen based on a first image output by an SOC, the resolution ratio of the first image is smaller than that of the second image, the first image comprises K first areas, and the second image comprises K second areas, different first areas in the K first areas correspond to different second areas in the K second areas, and the position of each first area in the first image is the same as the position of the corresponding second area in the second image; wherein the resolution ratio of a first target area in the K first areas is the same as the resolution ratio of a corresponding second target area in the K second areas, and the content of the first target area is the same as the content of the second target area; the content of each second area except the first target area in the K second areas is obtained by up-sampling the content of the corresponding first area.

Description

Technical Field

[0001] The present application relates to the field of terminal devices, and in particular to a display method and electronic device. Background Art

[0002] With the development of terminal display technology, the application scenarios of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies are increasing. Users can experience different effects such as VR, AR, or MR by wearing head-mounted display devices.

[0003] However, in the actual process of wearing the head-mounted display device, it was found that the head-mounted display device has a display delay, which causes the user to have the risk of dizziness when wearing it, seriously affecting the user experience. Summary of the invention

[0004] The present application provides a display method and an electronic device, in order to reduce display latency, thereby avoiding the risk of dizziness when the user wears the device and improving the comfort of the user when wearing a head-mounted display device.

[0005] In a first aspect, the present application provides a display method, which is applied to an electronic device, the electronic device comprising a first display screen, the method comprising: when the first display screen receives a first image, the first display screen displays a second image.

[0006] Exemplarily, the electronic device is a head-mounted display device.

[0007] In an embodiment of the present application, the device that outputs the first image is referred to as a processing device. For example, the first image received by the first display screen may be an image output by a system on chip (SOC), that is, the processing device is the SOC. It is noted that the present application does not limit the specific form between the SOC and the electronic device. For example, in some embodiments, the SOC is included in the electronic device. For another example, in some embodiments, the SOC is not included in the electronic device. That is, the SOC and the electronic device are independent of each other.

[0008] Specifically, in the embodiment of the present application, the resolution of the second image output by the first display screen is the same as the resolution of the first image.

[0009] Specifically, in the embodiment of the present application, the first image and the second image output by the first display screen both include K regions. In the embodiment of the present application, the K regions included in the first image are referred to as K first regions, and the K regions included in the second image output by the first display screen are referred to as K second regions.

[0010] For example, both the first image and the second image can be divided into nine (i.e., K is 9) areas in three rows and three columns: a central area located at the center of the image, an upper left area located at the upper left corner of the central area, an upper transition area located above the central area, an upper right area located at the upper right corner of the central area, a left transition area located to the left of the central area, a right transition area located to the right of the central area, a lower left area located at the lower left corner of the central area, a lower transition area located below the central area, and a lower right area located at the lower right corner of the central area.

[0011] Specifically, different first areas among the K first areas correspond to different second areas among the K second areas, and the position of each first area in the first image is the same as the position of the corresponding second area in the second image. Or in other words, in the embodiment of the present application, the K first areas correspond one-to-one with the K second areas, and the positions of a first area in the first image and its corresponding second area in the second image are the same in their respective images. Or in other words, in the embodiment of the present application, the first image and the second image output by the first display screen are aligned.

[0012] For example, taking the above-mentioned first image and second image that can both be divided into nine areas (i.e., K is 9) of three rows and three columns as an example, it can be considered that: the upper left corner area in the first image corresponds to the upper left corner area in the second image, the upper transition area in the first image corresponds to the upper transition area in the second image, the upper right corner area in the first image corresponds to the upper right corner area in the second image, the left transition area in the first image corresponds to the left transition area in the second image, the central area in the first image corresponds to the central area in the second image, the right transition area in the first image corresponds to the right transition area in the second image, the lower left corner area in the first image corresponds to the lower left corner area in the second image, the lower transition area in the first image corresponds to the lower transition area in the second image, and the lower right corner area in the first image corresponds to the lower right corner area in the second image.

[0013] Specifically, in an embodiment of the present application, the resolution of the first target area among the K first areas is the same as the resolution of the second target area among the K second areas, and the content of the first target area is the same as the content of the second target area, and the resolution of each second area among the K second areas except the second target area is higher than the resolution of the corresponding first area (for example, the content of each second area among the K second areas except the second target area is obtained by upsampling the content of the corresponding first area).

[0014] In other words, in an embodiment of the present application, the second image displayed on the first display screen includes a second target area, and the content of the second target area is exactly the same as the content of the first target area in the first image output by the SOC, that is, the second target area displayed on the first display screen is not upsampled with respect to the content of the first target area, and the original resolution of the first target area is maintained.

[0015] It should be understood that in the display method, by making the resolution of the first image smaller than the resolution of the second image output by the first display screen, it is possible to ensure the smoothness of the display when the transmission bandwidth of the processing device (such as SOC) that outputs the first image is insufficient. Since the second target area included in the second image is not upsampled to the corresponding first target area in the SOC when the second image is displayed to the user on the first display screen, the clarity of the first target area can be guaranteed. In a specific implementation, the first target area is usually the user's gaze area. In addition, in the display method provided by the embodiment of the present application, since the first image output by the processing device and the second image output by the first display screen are aligned, at this time, it is not necessary to first transmit the first target area before starting to display, and data can be transmitted while being displayed on the first display screen, thereby reducing the display delay.

[0016] In combination with the first aspect, in some embodiments, each second region except the second target region among the K second regions is obtained by upsampling the corresponding first region, and the upsampling ratio corresponding to the second region farther from the second target region among the K second regions is greater.

[0017] In other words, when the second image is displayed on the first display screen based on the first image, a second area farther from the second target area among the K second areas has a larger upsampling ratio when it is obtained based on the corresponding first area.

[0018] It can be understood that in this implementation, when the second image is displayed on the first display screen based on the first image, the upsampling ratio used when upsampling the first area farther from the first target area among the K first areas is larger, therefore, the K second areas displayed on the first display screen have the following characteristics: the second area farther from the second target area in the second image is less clear, while the second area closer to the second target area is clearer. That is, in this implementation, when the K second areas are displayed on the first display screen, the clarity of the second areas at different distances from the second target area is different.

[0019] It should be understood that compared with using the same resolution for the first target area and the corresponding second target area, all other second areas are sampled with the same upsampling ratio. The second image displayed in this way is more in line with the characteristics of the user's viewing, thereby improving the user experience.

[0020] In combination with the first aspect, in some embodiments, the electronic device also includes a second display screen, and the method also includes: when the second display screen receives the third image, the second display screen displays a fourth image, the resolution of the third image is less than the resolution of the fourth image, the third image includes K third areas, the fourth image includes K fourth areas, different third areas among the K third areas correspond to different fourth areas among the K fourth areas, and the position of each third area in the third image is the same as the position of the corresponding fourth area in the fourth image; the K third areas include a third target area, the K fourth areas include a fourth target area corresponding to the third target area, the resolution of the third target area and the fourth target area is the same, and the resolution of each fourth area among the K fourth areas except the fourth target area is higher than the resolution of the corresponding third area.

[0021] Among them, this solution can be compared to the display method of the first display screen in the first aspect, and will not be repeated here.

[0022] In combination with the first aspect, in some embodiments, the first image and the second image can be divided into nine areas in three rows and three columns: a central area located at the center of the image, an upper left corner area located at the upper left corner of the central area, an upper transition area located above the central area, an upper right corner area located at the upper right corner of the central area, a left transition area located to the left of the central area, a right transition area located to the right of the central area, a lower left corner area located at the lower left corner of the central area, a lower transition area located below the central area, and a lower right corner area located at the lower right corner of the central area;

[0023] The first target area is the central area of ​​the first image, the second target area is the central area of ​​the second image, and the first target area and the second target area have the same resolution, including: the central area of ​​the second image has the same resolution as the central area of ​​the first image.

[0024] In combination with the first aspect, in some embodiments, the upper left corner area in the second image is obtained based on upsampling the upper left corner area in the first image by q1 times in the horizontal direction and by p1 times in the vertical direction, the left transition area in the second image is obtained based on upsampling the left transition area in the first image by q1 times in the horizontal direction and without upsampling in the vertical direction, and the upper transition area in the second image is obtained based on not upsampling the upper transition area in the first image in the horizontal direction and by p1 times in the vertical direction, wherein q1 and p1 are both greater than 1.

[0025] Exemplarily, when the resolution of the second image is 3840*3840, the upper left corner area in the second image is obtained by upsampling the upper left corner area in the first image by a factor of 2 in both the horizontal and vertical directions, the left transition area in the second image is obtained by upsampling the left transition area in the first image by a factor of 2 in the horizontal direction and without upsampling in the vertical direction, and the upper transition area in the second image is obtained by not upsampling the upper transition area in the first image in the horizontal direction and upsampling it by a factor of 2 in the vertical direction.

[0026] In a second aspect, the present application provides a display device, comprising a functional module for implementing the method as described in the first aspect or any possible implementation manner thereof.

[0027] In a third aspect, the present application provides a display device, comprising: a memory and a processor; the memory is used to store program instructions; the processor is used to execute the program instructions in the memory to implement the method described in the first aspect or any possible implementation method thereof.

[0028] In a fourth aspect, the present application provides an electronic device, comprising one or more display screens, one or more memories, and one or more processors; wherein the one or more memories store one or more programs; when the one or more processors execute the one or more programs, the electronic device executes the method described in the first aspect or any possible implementation method thereof.

[0029] In a fifth aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when executed on the electronic device, enable the electronic device to execute the method described in the first aspect or any possible implementation thereof.

[0030] In a sixth aspect, the present application provides a computer program product, the computer program product comprising instructions for implementing the method as described in the first aspect or any possible implementation manner thereof.

[0031] It can be understood that the effects that can be obtained in the second, third, fourth, fifth and sixth aspects can refer to the description in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0033] Figure 2 A structural schematic diagram of a VR system provided in an embodiment of the present application;

[0034] Figure 3 A structural schematic diagram of a head-mounted device provided in this application;

[0035] Figure 4 A structural schematic diagram of another head-mounted device provided in this application;

[0036] Figure 5 A schematic diagram of a software structure used by the head mounted device provided in this application;

[0037] Figure 6 A schematic diagram of an existing display method;

[0038] Figure 7 A schematic diagram of another existing display method;

[0039] Figure 8 A schematic diagram of the process of existing SOC transmission and display screen display;

[0040] Fig. 9 An example of the display method provided by this application;

[0041] Fig.10 An example of the display method provided by this application;

[0042] Fig.11 Schematic diagram of the SOC transmission and display process provided by this application;

[0043] Fig.12 Another example of the display method provided by the present application;

[0044] Fig.13 Another example of the display method provided by the present application;

[0045] Fig.14 Another example of the display method provided by the present application;

[0046] Fig.15 A schematic diagram of a display device provided for an embodiment of the application. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0049] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0050] In addition, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, where a, b, c can be single or multiple.

[0051] The processing method of the terminal device provided in the embodiment of the present application can also be called a method for identifying a third-party drawing frame thread, which can be applied to a terminal device with a display function. The terminal device can be a wearable terminal device such as a mobile phone, a tablet computer, a personal computer (PC), a smart screen, a car machine device, and a smart watch. It can also be various teaching auxiliary tools (such as learning machines, early education machines), smart toys, portable robots, personal digital assistants (PDA), augmented reality technology (AR) equipment, virtual reality (VR) equipment, etc. It can also be a device with mobile office functions, a device with smart home functions, a device with audio and video entertainment functions, a device that supports smart travel, etc. It should be understood that the embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0052] With the development of terminal display technology, the application scenarios of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies are increasing. Users can experience different effects such as VR, AR, or MR by wearing head-mounted display devices.

[0053] refer to Figure 1 , Figure 1 This is a schematic diagram of the application scenario provided by this application. Figure 1 As shown, the user can wear the head-mounted display device on the head to watch the content played on the head-mounted display device. For example, a user wearing a VR head-mounted display device (such as VR glasses, VR helmets, etc.) can see the VR game interface, and can interact with the VR game interface through gestures, handles, etc., as if they are in the game. For another example, a user wearing an AR transparent glasses head-mounted display device can see the real environment around through the lenses of the glasses, and virtual objects can also be displayed on the lenses, so that the user can see both real objects and virtual objects.

[0054] Among them, VR technology is a means of human-computer interaction created with the help of computer and sensor technology. VR technology integrates a variety of scientific technologies such as computer graphics technology, computer simulation technology, sensor technology, display technology, etc., and can create a virtual environment. The virtual environment includes two-dimensional or three-dimensional virtual objects generated by computers and played dynamically in real time, providing users with simulations of senses such as vision, making users feel as if they are in the scene. Moreover, in addition to the visual perception generated by computer graphics technology, there are also perceptions such as hearing, touch, force, movement, and even smell and taste, which are also called multi-perception. In addition, the user's head rotation, eyes, gestures, or other human behavior movements can also be detected, and the computer will process the data adapted to the user's movements, respond to the user's movements in real time, and feedback to the user's five senses respectively, thereby forming a virtual environment.

[0055] AR technology refers to superimposing computer-generated virtual objects on real-world scenes to enhance the real world. In other words, AR technology requires collecting real-world scenes and then adding virtual environments to the real world. Therefore, the difference between VR technology and AR technology is that VR technology creates a completely virtual environment, and all users see are virtual objects; while AR technology superimposes virtual objects on the real world, so users can see both real-world objects and virtual objects.

[0056] MR technology introduces real scene information (or real scene information) into the virtual environment, building a bridge of interactive feedback information between the virtual environment, the real world and the user, thereby enhancing the sense of reality of the user experience. Specifically, the real object is virtualized (for example, a camera is used to scan the real object for three-dimensional reconstruction and generate a virtual object), and the virtualized real object is introduced into the virtual environment, so that the user can see the real object in the virtual environment.

[0057] Below, for ease of understanding, the technical solution of the present application is introduced by taking a user wearing a VR head-mounted display device as an example. However, it should be noted that the technical solution provided in the embodiment of the present application is not limited to VR head-mounted display devices. For example, it can be applied to VR, AR or MR, or it can also be applied to other scenes or electronic devices other than VR, AR or MR that need to display a three-dimensional environment to the user. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0058] refer to Figure 2 , Figure 2 A schematic diagram of a VR system presenting a VR effect to a user is given. Figure 2 As shown, the VR system includes a VR head-mounted display device 100 and a processing device 200, and the VR system can be called a VR split machine. Specifically, the VR head-mounted display device 100 can be connected to the processing device 200. The connection between the VR head-mounted display device 100 and the processing device 200 includes a wired or wireless connection, and the wireless connection can be Bluetooth (bluetooth, BT), which can be traditional Bluetooth or low-power BLE Bluetooth, wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), Zigbee, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), or general 2.4G / 5G frequency band wireless communication connection, etc.

[0059] Among them, the processing device 200 can perform processing calculations. For example, the processing device 200 can generate an image and process the image, and then send the processed image to the VR head-mounted display device 100 for display.

[0060] In some embodiments, the processing device 100 may be in the form of a host (e.g., a VR host) or a server (e.g., a VR server). The VR host or VR server may be a device with greater computing power. For example, the VR host may be a mobile phone, a tablet computer, a laptop computer, or the like, and the VR server may be a cloud server, or the like.

[0061] Among them, the VR head-mounted display device 100 includes a display screen, and the display screen of the VR head-mounted display device 100 can display images to the human eye.

[0062] In some embodiments, the VR head mounted display device 100 includes only one display screen, and the user's eyes view the images on the one display screen.

[0063] In some embodiments, the VR head mounted display device 100 may be provided with two display screens (such as VR glasses, etc.). Figure 2 In the VR system shown in FIG. 1 , the display screen 110 and the display screen 120 are wrapped inside the VR glasses, so Figure 2 The arrows indicating the display screen 110 and the display screen 120 are represented by dotted lines.

[0064] Optionally, the VR head mounted display device 100 also has image generation and processing functions, that is, the VR head mounted display device 100 does not need Figure 1 The processing device 200 in the VR head mounted display device 100 can be called a VR all-in-one machine.

[0065] refer to Figure 3 , Figure 3 A schematic diagram of a VR head mounted display device 100 including two display screens is shown. Figure 3 As shown, the VR head mounted display device 100 includes a display module 1 and a display module 2.

[0066] The display module 1 is used to display images to the right eye of the user, and the display module 1 includes a display screen 110 and an optical device 130, and the display screen 110 is located on the side of the optical device 130 away from the right eye. The display module 2 is used to display images to the left eye of the user, and the display module 2 includes a display screen 120 and an optical device 140, and the display screen 120 is located on the side of the optical device 140 away from the left eye.

[0067] Specifically, in a scenario where a user wears the VR head-mounted display device 100, when the display screen 110 is displaying an image, the light emitted by the display screen 110 is converged to the user's right eye through the optical device 330, and when the display screen 120 is displaying an image, the light emitted by the display screen 120 is converged to the user's left eye through the optical device 140.

[0068] It should be noted that Figure 3 The VR head-mounted display device 100 shown may also include other components, for example, a support portion 30 and a bracket 40, wherein the support portion 30 is used to support the VR head-mounted display device on the bridge of the nose, and the bracket 40 is used to support the VR head-mounted display device on both ears to ensure that the VR head-mounted display device is stably worn.

[0069] Optionally, in some embodiments, the optical device 130 or the optical device 140 may be one or more optical devices such as a reflector, a transmission mirror, or an optical waveguide, and the field of view may also be improved. For example, the optical device 130 or the optical device 140 may be an eyepiece composed of multiple transmission mirrors. For example, the optical device may be a Fresnel lens and / or an aspherical lens, etc., which is not limited in the embodiments of the present application. Generally, the optical device 130 and the optical device 140 are respectively aligned with the two eyes of the user.

[0070] Optionally, in some embodiments, the optical device 130 and the optical device 140 are symmetrical with respect to the middle plane C. The symmetry may be, for example, strict symmetry or slight deviation. For example, the optical device 130 and the optical device 140 may be strictly symmetrical with respect to the middle plane C, or the optical device 130 and the optical device 140 may be basically symmetrical with respect to the middle plane C, and the basic symmetry may have a certain deviation, which is within a small range. Optionally, in some embodiments, the VR head-mounted display device 100 may be a left-right symmetrical structure, the support portion 30 and / or the bracket 40 may be respectively left-right symmetrical with respect to the middle plane C, and the support portion 30 may fix the position of the face, which is beneficial for the optical device 130 and the optical device 140 to align the left eye and the right eye of the user, respectively.

[0071] It is understandable that the VR head mounted display device 100 may further include more components. Figure 4 . Figure 4 The structure diagram of a head mounted display device 100 provided in an embodiment of the present application is shown in FIG. The head mounted display device may be a VR head mounted display device, an AR head mounted display device, an MR head mounted display device, and the like. Taking a VR head mounted display device as an example, Figure 4 As shown, the VR head-mounted display device 100 may include a processor 101, a memory 102, a sensor module 103 (for example, which can be used to obtain the user's posture, etc.), a microphone 104, a button 150, an input and output interface 160, a communication module 170, a camera 180, a battery 190, an optical display module 106, and an eye tracking module 105, etc.

[0072] The processor 101 is generally used to control the overall operation of the VR head mounted display device 100, and may include one or more processing units, for example: the processor 101 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a video processing unit (video processing unit, VPU) controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices, or they may be integrated in one or more processors.

[0073] The processor 101 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. The memory may store instructions or data that the processor 101 has just used or cyclically used. If the processor 101 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 101, and thus improves the efficiency of the system.

[0074] In some embodiments of the present application, the processor 101 can be used to control the optical focal length of the VR head-mounted display device 100. Exemplarily, the processor 101 can be used to control the optical focal length of the optical display module 106 to achieve the function of adjusting the optical focal length of the head-mounted display device 100. For example, the processor 101 can adjust the relative positions between the various optical devices (such as lenses, etc.) in the optical display module 106 so that the optical focal length of the optical display module 106 is adjusted, and then the position of the corresponding virtual image surface of the optical display module 106 can be adjusted when the optical display module 106 is imaged to the human eye. Thereby achieving the effect of controlling the optical focal length of the head-mounted display device 100.

[0075] In some embodiments, the processor 101 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0076] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 101 may include multiple I2C buses.

[0077] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 101 and the communication module 170. For example, the processor 101 communicates with the Bluetooth module in the communication module 170 through the UART interface to implement the Bluetooth function.

[0078] The MIPI interface can be used to connect the processor 101 with the display screen in the optical display module 106 , the camera 180 and other peripheral devices.

[0079] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 101 with the camera 180, the display screen in the optical display module 106, the communication module 170, the sensor module 103, the microphone 104, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. In some embodiments, the camera 180 can capture images including real objects, and the processor 101 can fuse the images captured by the camera with virtual objects to obtain images obtained by the actual fusion of the optical display module 106. In some embodiments, the camera 180 can also capture images including human eyes. The processor 101 performs eye tracking through the image.

[0080] The USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the VR head-mounted display device 100, and can also be used to transmit data between the VR head-mounted display device 100 and peripheral devices. It can also be used to connect headphones and play audio through the headphones. The interface can also be used to connect other electronic devices, such as mobile phones. The USB interface can be USB3.0, which is used to be compatible with high-speed display port (DP) signal transmission, and can transmit high-speed audio and video data.

[0081] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the head-mounted display device 100. In other embodiments of the present specification, the head-mounted display device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0082] In addition, the VR head-mounted display device 100 may include a wireless communication function. For example, the VR head-mounted display device 100 may receive images from other electronic devices (such as a VR host) for display, or the VR head-mounted display device 100 may directly obtain data from sites such as base stations. The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication function can be implemented through an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The VR head-mounted display device 100 may include multiple antennas, each of which can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example: antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0083] The mobile communication module can provide solutions for wireless communications including the second generation (2G) network / third generation (3G) network / fourth generation (4G) network / fifth generation (5G) network / sixth generation (6G) network, etc., applied to the VR head mounted display device 100. The mobile communication module may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module may receive electromagnetic waves from an antenna, filter, amplify, and process the received electromagnetic waves, and transmit them to a modulation and demodulation processor for demodulation. The mobile communication module may also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some functional modules of the mobile communication module may be arranged in the processor 101. In some embodiments, at least some functional modules of the mobile communication module may be arranged in the same device as at least some modules of the processor 101.

[0084] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, etc.), or displays an image or video through a display screen in the optical display module 106. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 101 and be arranged in the same device as the mobile communication module or other functional modules.

[0085] The wireless communication module can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the VR head-mounted display device 100. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 101. The wireless communication module can also receive the signal to be sent from the processor 101, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0086] In some embodiments, the antenna of the VR head mounted display device 100 is coupled to the mobile communication module, so that the VR head mounted display device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), 5G, 6G, BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0087] The VR head mounted display device 100 implements display functions through a GPU, an optical display module 106, and an application processor. The GPU is a microprocessor for image processing, connecting the optical display module 106 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 101 may include one or more GPUs that execute program instructions to generate or change display information.

[0088] The memory 102 can be used to store computer executable program codes, which include instructions. The processor 101 executes various functional applications and data processing of the VR head-mounted display device 100 by running the instructions stored in the memory 102. The memory 102 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the head-mounted display device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0089] The VR head-mounted display device 100 can implement audio functions through an audio module, a speaker, a microphone 104, a headphone interface, and an application processor. For example, music playback, recording, etc. The audio module is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module can also be used to encode and decode audio signals. In some embodiments, the audio module can be set in the processor 101, or some functional modules of the audio module can be set in the processor 101. The speaker, also known as a "speaker", is used to convert audio electrical signals into sound signals. The head-mounted display device 100 can listen to music or listen to hands-free calls through the speaker.

[0090] Microphone 104, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. VR head-mounted display device 100 can be provided with at least one microphone 104. In other embodiments, VR head-mounted display device 100 can be provided with two microphones 104, which can not only collect sound signals but also realize noise reduction function. In other embodiments, VR head-mounted display device 100 can also be provided with three, four or more microphones 104 to realize sound signal collection, noise reduction, and can also identify the sound source, realize directional recording function, etc.

[0091] The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0092] In some embodiments, the VR head-mounted display device 100 may include one or more buttons 150, which can control the VR head-mounted display device and provide the user with the function of interacting with the VR head-mounted display device 100. The button 150 can be in the form of a button, a switch, a dial, and a touch or near-touch sensing device (such as a touch sensor). Specifically, for example, the user can turn on the optical display module 106 of the VR head-mounted display device 100 by pressing a button. The button 150 includes a power button, a volume button, etc. The button 150 can be a mechanical button. It can also be a touch button. The head-mounted display device 100 can receive button input and generate key signal input related to user settings and function control of the head-mounted display device 100.

[0093] In some embodiments, the VR head mounted display device 100 may include an input and output interface 160, which may connect other devices to the VR head mounted display device 100 through appropriate components. The components may include, for example, audio / video jacks, data connectors, and the like.

[0094] The optical display module 106 is used to present images to the user under the control of the processor 101. The optical display module 106 can convert the real pixel image display into a virtual image display of near-eye projection through one or more optical devices such as a reflector, a transmission mirror or an optical waveguide, so as to realize a virtual interactive experience or an interactive experience combining virtual and reality. For example, the optical display module 106 receives the image data information sent by the processor 101 and presents the corresponding image to the user.

[0095] For example, the optical display module 106 can refer to the above Figure 3 The structure shown, for example, includes two display screens in the optical display module 106, namely, the display screen 110 and the display screen 120. For another example, the optical display module 106 includes a display module 1 and a display module 2, wherein the display module 1 includes the display screen 110 and the optical device 130, and the display module 2 includes the display screen 120 and the optical device 140.

[0096] In some embodiments, the VR head mounted display device 100 may further include an eye tracking module 105, which is used to track the movement of the human eye, and then determine the gaze point of the human eye. For example, the pupil position can be located by image processing technology, the pupil center coordinates can be obtained, and then the gaze point of the person can be calculated. In some embodiments, the eye tracking system can determine the user's gaze point position (or determine the user's line of sight direction) by methods such as video eye diagram method, photodiode response method, or pupil corneal reflection method, thereby realizing the user's eye tracking.

[0097] It should be noted that, in some embodiments, a corresponding eye tracking system may be set for each eye of the user, so as to track the eye movements of the eyes synchronously or asynchronously. In other embodiments, an eye tracking system may be set only near one eye of the user, and the sight direction of the corresponding eye may be obtained through the eye tracking system, and the sight direction or the sight point position of the other eye of the user may be determined based on the relationship between the gaze points of the two eyes (e.g., when the user observes an object through both eyes, the gaze points of the two eyes are generally close or the same), combined with the distance between the two eyes of the user.

[0098] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the VR head mounted display device 100. In other embodiments of the present specification, the VR head mounted display device 100 may include Figure 4 The embodiments of the present application are not limited to more or fewer components, or combining certain components, or splitting certain components, or different arrangements of components.

[0099] Figure 5 1 is a software structure diagram of the VR head mounted display device 100 according to an embodiment of the present application. Figure 5 As shown, the software structure of the VR head mounted display device 100 can be a layered architecture, for example, the software can be divided into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, it is divided into five layers, from top to bottom, namely, application layer 510, application framework layer (framework, FWK) 520, Android runtime (Androidruntime) and system library 530, kernel layer 540 and hardware layer 550.

[0100] The application layer 510 may include a series of application packages. For example, Figure 5 As shown, the application layer includes gallery applications, game applications, etc.

[0101] The application framework layer 520 provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer may include some predefined functions. Figure 5 As shown, the application framework layer may include a resource manager, a view system, etc. For example, the view system 5 includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface may be composed of one or more views. For example, a display interface including a message notification icon may include a view for displaying text and a view for displaying pictures. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

[0102] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] The system library can include multiple functional modules. For example: surface manager 5, media libraries, three-dimensional graphics processing library, such as OpenGL ES, two-dimensional graphics engine, such as SGL, etc. Among them, the surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc. The two-dimensional graphics engine is a drawing engine for two-dimensional drawing.

[0104] The kernel layer 540 is a layer between hardware and software. The kernel layer 540 at least includes a display driver, a camera driver, an audio driver, a sensor driver, and the like.

[0105] The hardware layer may include display screens and various sensor modules, such as accelerometers, gravity sensors, touch sensors, etc.

[0106] Understandably, Figure 5 The software structure shown does not constitute a specific limitation on the software structure of the VR head mounted display device 100. For example, in other embodiments, the software structure of the VR head mounted display device 100 may include: Figure 5 More or fewer layers, for example, also include an adaptation layer, which is located between the application framework layer 520 and the Android runtime and system library 530, and is used to achieve adaptation between the upper layer (i.e., the application framework layer) and the lower layer (i.e., the system library), for example, to achieve interface matching between the upper layer and the lower layer to ensure that the upper layer and the lower layer can communicate data.

[0107] From the above description, it can be known that when the head mounted display device displays an image on the display screen, it needs to first receive the image data sent by the processing device. Alternatively, it can be expressed as follows: when the head mounted display device displays an image to the user through the display screen, it needs to be driven by the processing device.

[0108] In some embodiments, the display screen is included in the head-mounted device, and the processing device is connected to the head-mounted display device such as Figure 2 As shown, they are deployed separately.

[0109] In some embodiments, the processing device and the display screen are integrated into the head-mounted display device. In this case, the head-mounted display device can also be called an all-in-one device.

[0110] Generally, head-mounted display devices have high requirements for display, which are mainly reflected in the following three aspects: 1) higher resolution; 2) larger field of view (FoV); 3) higher refresh rate and lower latency.

[0111] The resolution refers to the resolution of the image output by each display screen.

[0112] Among them, the refresh rate refers to the number of times the display screen is refreshed per second, also known as the output frame rate, and the unit is usually Hertz (Hz).

[0113] For example, for a certain head-mounted display device, its display indicator (also called output capability) is 1800x1920x2@90Hz, which means that when a single display screen is displayed, the image resolution is 1800x1920 and the refresh rate is 90HZ (that is, 90 frames are output per second).

[0114] In some embodiments, silicon-based organic light emitting diodes (OLEDs) can be compatible with small and thin form factors and high resolution due to their ultra-high PPI (pixels per inch) characteristics, and also have advantages such as high color gamut and high brightness. Therefore, silicon-based OLEDs can be used for display to achieve higher resolutions. For example, when silicon-based OLEDs are used for display, each block can achieve a resolution of 4K.

[0115] However, the development of processing equipment that drives display screens is slower than that of display screens, which is mainly reflected in the insufficient transmission bandwidth of processing equipment, resulting in the inability to fully display the display effect of the display screen.

[0116] In some embodiments, the processing device is a system on chip (SOC).

[0117] For example, if the output capacity of each display is 3840*3840@90Hz, it means that the resolution of the image output by each display is 3840*3840 and the refresh rate is 90HZ (i.e., 90 frames of images can be output per second). If each display is to achieve the best display effect, the bandwidth required for each display is 3840*3840*90Hz*24bits / (1024^3)=29.66Gbps. However, if the maximum transmission bandwidth supported by the SOC is 17Gbps / panel, the problem of being unable to transmit 3840*3840*90Hz images to the display arises because the SOC transmission bandwidth cannot meet the needs of the display.

[0118] At present, when the SOC transmission bandwidth cannot meet the needs of the display screen, there are two commonly used display solutions:

[0119] The first commonly used display solution is: while ensuring the refresh rate remains unchanged, the resolution of the image output by the SOC is made smaller than the resolution of the image output by each display screen to ensure the maximum transmission bandwidth supported by the SOC. The head-mounted display device then upsamples the image transmitted by the SOC to obtain an image with the same resolution as the display screen.

[0120] In some embodiments, the head mounted display device may upsample the image transmitted by the SOC by a display driver IC (DDIC) in the head mounted display device.

[0121] For example, the maximum transmission bandwidth supported by the above SOC is 17Gbps / panel, and the output capacity of each display is 3840*3840@90Hz. If the SOC meets the transmission bandwidth of less than 17Gbps / panel, Figure 6 As shown, the SOC can output an image to the display screen in a 2880*2880@90Hz output mode, and then the DDIC upsamples the image by a factor of 1.33 to display a 3840*3840 image on the display screen.

[0122] It is noted here that the Figure 6 The 2880*2880 is only an example and does not constitute a limitation of the present application.

[0123] For example, in some embodiments, the resolution of the image output by the SOC can be x1*x1, as long as x1*x1*90Hz*24bits / (1024^3) is less than or equal to 17Gbps; accordingly, the DDIC upsamples the image transmitted by the SOC by 3840 / x1 in both the horizontal and vertical directions to display a 3840*3840 image on the display screen.

[0124] For example, in some embodiments, the resolution of the image output by the SOC can be x1*y1, as long as x1*y1*90Hz*24bits / (1024^3) is less than or equal to 17Gbps; accordingly, after receiving the image transmitted by the SOC, the DDIC performs upsampling by 3840 / x1 in the horizontal direction and by 3840 / y1 in the vertical direction to display a 3840*3840 image on the display screen.

[0125] The second commonly used display solution is to reduce the refresh rate of the SOC, that is, reduce the frequency of the SOC output image. For example, when the SOC meets the transmission bandwidth of less than 17Gbps / panel, the SOC can output images to each display screen through the output mode of 3840*3840@50Hz. Accordingly, each display screen outputs 50 frames of images per second.

[0126] Similarly, the number 50 in this example is only an example and does not constitute a limitation of the present application. For example, in some embodiments, the output frame rate of the SOC can be z1, as long as 3840*3840*z1*24bits / (1024^3) is less than or equal to 17Gbps.

[0127] It should be understood that the first method will result in a problem of reduced clarity of the image output by each display screen, while the second implementation method will result in a problem of reduced smoothness during display.

[0128] To this end, people in this field have proposed another display solution, also known as a gaze point rendering solution. The basic concept of this solution is: since the FOV of the human eye is very small, for example, it is in the range of 20 to 30 degrees. Therefore, this display solution includes: the image output by the SOC is composed of two parts stacked and spliced ​​up and down, wherein the upper part is the gaze area part that needs to be displayed on the display screen, and the lower part is the background area part that needs to be displayed on the display screen, and then the image is transmitted to the display screen; accordingly, when the head-mounted display device is displayed through the display screen, the gaze area part is not upsampled, but the background area part is upsampled, and the coordinate position of the gaze area part on the display screen is identified by combining the human eye tracking technology, and the gaze area part is displayed at the corresponding coordinate position to ensure the clarity of the gaze area.

[0129] It can be seen that this display solution reduces the resolution of the SOC output image to ensure the SOC transmission bandwidth while ensuring that the clarity of the gaze area and the refresh rate are not damaged.

[0130] As an example, Figure 7As shown, the SOC output image includes two parts, the upper part is the fixation area, and the lower part is the background area. The resolution of the fixation area is 1536*1536, and the resolution of the background area is also 1536*1536. Then the SOC transmits the image in a top-to-bottom transmission order. Accordingly, when the display is displayed on the display screen, after the specified position of the fixation area on the display screen is identified by combining the human eye tracking technology ( Figure 7 In the example, the fixation area is at the center of the display screen. No upsampling is performed on the fixation area and the fixation area is displayed at the specified position, while the background area is upsampled 2.5 times. That is, when the display screen is displayed, the resolution of the fixation area is still 1536*1536, ensuring the clarity of the fixation area.

[0131] It is to be noted here that Figure 7 The 1536*1536 of the fixation area and background area output by the SOC in the example is only an example and does not constitute a limitation. For example, assuming that the maximum output frame rate of the display screen is z2 HZ and the maximum bandwidth supported by the SOC is S Gbps / panel, then the resolution of the fixation area and the resolution of the background area output by the SOC can also be x2*x2, as long as x2*2x2*z2*24bits / (1024^3) is less than or equal to SGbps / panel, and then when it is transmitted to the display screen for display, when the display screen upsamples the background area, the upsampling ratio in the horizontal and vertical directions becomes 3840 / x2.

[0132] However, when wearing a head-mounted display device using the above-mentioned foveated rendering solution, it is found that there is a risk of dizziness when the user wears it, which seriously affects the user experience.

[0133] After analysis, it was found that the reasons why users are at risk of dizziness when wearing the headset are as follows: in the above-mentioned gaze point rendering scheme, when the head-mounted display device displays each frame of image, it is only when the SOC has completed the transmission of the gaze area part and starts to transmit the background area part that it begins to judge whether each pixel of the display screen displays the data of the gaze area or the background area, and starts to display line by line. That is, in the above-mentioned gaze point rendering scheme, the display screen must start displaying only after the data of the gaze area part is transmitted. This results in a fixed display delay when each frame of image is displayed, which makes the user at risk of dizziness when wearing the headset.

[0134] For example, Figure 7 For example, the delay required to transmit the data in the gaze area is 1 / 2.5=0.4 frames. If the refresh rate of the display is 90HZ, that is, the time required to display a frame of image is 11.1ms, then the display delay of each frame is 4.4ms (11.1*0.4), which may cause dizziness to the user.

[0135] It should be understood that under this solution, if the resolution of the gaze area output by the SOC is greater, the display delay of the entire device will be greater.

[0136] For ease of understanding, Figure 8 The figure shows the process of SOC transmission and screen display in a scenario where the resolution of the gaze area and background area output by the SOC is 1536*1536 and the resolution of the image output by the display screen is 3840*3840. Figure 8 As shown, the horizontal axis represents time, the vertical axis represents the number of screen rows of the display screen, line 1 represents the value range of the pixel points of the gaze area at the top of the display screen, and line 2 represents the value range of the pixel points of the gaze area at the bottom of the display screen. It takes 0.4 frames to transmit the data of the gaze area of ​​each frame of the image. After the data of the gaze area is transmitted, the SOC starts to transmit the data of the background area (indicated by line 3 in the figure), and accordingly, the display screen of the head-mounted display device starts to display line by line (indicated by line 4 in the figure).

[0137] In view of this, an embodiment of the present application provides a display method, so that when a head-mounted display device is displayed on a display screen, while ensuring the clarity and smoothness of the gaze area, the display delay of the entire device can be reduced, thereby avoiding the risk of dizziness that occurs when the user wears the head-mounted display device and improving the user experience.

[0138] Below, an example is first used to illustrate the display solution provided by the embodiment of the present application.

[0139] In this example, a commonly used display screen with an output capacity of 3840*3840*90HZ and a maximum transmission bandwidth supported by the SOC of 17Gbps / panel is used as an example for description.

[0140] Specifically, the display method proposed in the embodiment of the present application is to align the image output by the SOC with the image output by the display screen. Fig. 9As shown, according to the user's demand for the display clarity of the whole machine, the image can be divided into the following regions: region 5, region 1 located at the upper left corner of region 5, region 2 located above region 5, region 3 located at the upper right corner of region 5, region 4 located to the left of region 5, region 6 located to the right of region 5, region 7 located at the lower left corner of region 5, region 8 located below region 5, and region 9 located at the lower right corner of region 5. Then the images output by the SOC and the display screen both include the above 9 regions, and for the same region, the position in the image output by the SOC and the position in the image output by the display screen are the same. For example, the position of region 5 included in the image output by the SOC and the position of region 5 included in the image output by the display screen are the same, and the region 1 included in the image output by the SOC and the region 1 included in the image output by the display screen are both in the upper left corner of their respective regions 5.

[0141] Among them, area 5 can be a central area, area 1, area 3, area 7 and area 9 can be corner areas, and area 2, area 4, area 6 and area 8 can be transition areas. At this time, area 5 can be called central area 5, area 1, area 3, area 7 and area 9 can be called upper left corner area 1, upper right corner area 3, lower left corner area 7 and lower right corner area 9 respectively, and area 2, area 4, area 6 and area 8 can be transition areas and can be called upper transition area 2, left transition area 4, right transition area 6 and lower transition area 8 respectively.

[0142] Specifically, in the embodiment of the present application, the image output by the SOC (that is, it can be considered as the first image) is as follows: Fig. 9 As shown, the resolution of area 5 is A2*B2, the resolution of area 1, area 3, area 7 and area 9 is A1*B1, the resolution of area 4 and area 6 is A1*B2, and the resolution of area 2 and area 8 is A2*B1. Wherein, A1+A2+A1=m, B1+B2+B1=n, m represents the horizontal resolution of the first image output by the SOC, and n represents the vertical resolution of the image output by the SOC.

[0143] It should be noted that in the embodiments of the present application, there is no restriction on the values ​​of m and n, as long as they can make the transmission bandwidth of the SOC less than 17Gbps / panel. That is, the values ​​of m and n need to satisfy: m*n*90*24bits / (1024^3) is less than or equal to 17Gbps.

[0144] Specifically, in an embodiment of the present application, when it is necessary to display on a display screen, for an image transmitted by the SOC, at least one area transmitted by the SOC will be upsampled so that a 3840*3840 image (that is, it can be considered as a second image) is displayed on the display screen.

[0145] It is noted here that the embodiments of the present application do not limit the specific upsampling ratio used, as long as it satisfies: A1*q1+A2*q2+A1*q1=3840, B1*p1+B2*p2+B1*p1=3840. Wherein, q1 is the upsampling ratio used in the horizontal direction when upsampling regions 1, 4, 7, 3, 6 and 9 in the SOC image, q2 is the upsampling ratio used in the horizontal direction when upsampling regions 2, 5 and 8 in the SOC image, p1 is the upsampling ratio used in the vertical direction when upsampling regions 1, 2, 3, 7, 8 and 9 in the SOC image, and p2 is the upsampling ratio used in the vertical direction when upsampling regions 4, 5 and 6 in the SOC image.

[0146] In an embodiment of the present application, when an image transmitted by the SOC is displayed on a display screen, the gaze area transmitted by the SOC may not be upsampled to ensure the clarity of the gaze area.

[0147] For example, if Fig. 9 If the fixation area in is area 5, then q2 and p2 can be set to 1 to ensure the clarity of area 5 when displayed on the display screen.

[0148] In some embodiments, when upsampling is performed on the display screen for areas other than the gaze area transmitted by the SOC, a larger upsampling factor is used when upsampling for areas farther away from the gaze area.

[0149] Still Fig. 9 For example, as an example, assuming that the gaze area is area 5, at this time q1 can be 2, q2 can be 1, p1 can be 2, and p2 can be 1.

[0150] For example, when q1 is 2, q2 is 1, p1 is 2, and p2 is 1, the resolution of each area in the image output by the SOC and the display screen is, for example, Fig.10 Reference Fig.10In the image output by SOC, the resolution of area 5 is 1536*1536, the resolution of area 1 is 1152 / 2*1152 / 2, the resolution of area 2 is 1536*1152 / 2, the resolution of area 3 is 1152 / 2*1152 / 2, the resolution of area 4 is 1152 / 2*1536, the resolution of area 6 is 1152 / 2*1536, the resolution of area 7 is 1152 / 2*1152 / 2, the resolution of area 8 is 1536*1152 / 2, and the resolution of area 9 is 1152 / 2*1152 / 2. Accordingly, when the head-mounted display device displays on the display screen, area 1 of SOC transmission is upsampled by 2 times in the horizontal and vertical directions, area 2 of SOC transmission is not upsampled in the horizontal direction but is upsampled by 2 times in the vertical direction, area 3 of SOC transmission is upsampled by 2 times in the horizontal and vertical directions, area 4 of SOC transmission is upsampled by 2 times in the horizontal direction but not in the vertical direction, area 5 of SOC transmission is not upsampled in the horizontal and vertical directions, area 6 of SOC transmission is upsampled by 2 times in the horizontal direction but not in the vertical direction, area 7 of SOC transmission is upsampled by 2 times in the horizontal and vertical directions, area 8 of SOC transmission is not upsampled in the horizontal direction but is upsampled by 2 times in the vertical direction, and area 9 of SOC transmission is upsampled by 2 times in the horizontal and vertical directions.

[0151] It should be understood that by using a larger upsampling factor when upsampling the area farther away from the gaze area, the clarity of the gaze area is the highest, and the clarity of the gaze area is greater than that of the transition area, and the clarity of the transition area is greater than that of the corner area.

[0152] It can be seen that in the display method provided by the embodiment of the present application, the SOC can ensure the smoothness of the display when the transmission bandwidth of the SOC is insufficient by making the resolution of the output image smaller than the resolution of the image output by the display screen. And because when the image is displayed to the user on the display screen, the fixation area included in the image displayed on the display screen is the area where the central area transmitted by the SOC is not upsampled, the clarity of the fixation area can be guaranteed.

[0153] Optionally, in some embodiments, when the image transmitted by the SOC is displayed on a display screen, the gaze area may be upsampled using a lower upsampling ratio to ensure the clarity of the gaze area; and when other areas other than the gaze area transmitted by the SOC are displayed on the display screen, the upsampling ratio used is greater than the upsampling ratio used for the gaze area, and the farther the area is from the gaze area, the greater the upsampling ratio used when upsampling.

[0154] In addition, in the display method provided in the embodiment of the present application, since the image output by the SOC and the image output by the display screen are aligned in position, it is not necessary to first transmit the gaze area before starting to display, but the SOC can transmit and display the display screen at the same time (such as Fig.11 As shown), the display delay can be reduced.

[0155] It should be noted that Fig. 9 The display method of the embodiment of the present application is described by taking the common fixation area as the central area (ie, the fixation area of ​​the human eye is in the central area of ​​the display screen) as an example. However, it should be understood that sometimes the fixation area may not be the central area.

[0156] Therefore, for the display method provided by the embodiments of the present application, in some embodiments, when the head-mounted display device does not have a gaze point function, the gaze area can be considered to be the central area. When the head-mounted display device has a gaze point function, the gaze area may include any one or more of the multiple divided areas (that is, the area that is not upsampled may be any one or more of the multiple divided areas), such as any one or more of the above 9 areas. In other words, when the head-mounted display device has a gaze point function, the area that is not upsampled is the gaze area, which changes with eye movement, and when the head-mounted display device does not have a gaze point function, the area that is not upsampled is the central area.

[0157] For example, Fig. 9 For example, if the gaze area of ​​the human eye becomes the upper left corner area 1, then when displaying on the display screen, the upper left corner area 1 will not be upsampled, and for other areas, upsampling is performed under the constraints of A1+A2+A1=m, B1+B2+B1=n, m*n*90*24bits / (1024^3) is less than or equal to 17Gbps, and A1*q1+A2*q2+A1*q1=3840, B1*p1+B2*p2+B1*p1=3840.

[0158] In the above, the display method provided in the embodiment of the present application is illustrated by taking the display screen output capacity of 3840*3840*90HZ, the maximum transmission bandwidth supported by the SOC of 17Gbps / panel, and the user's gaze area being located in the center area of ​​the display screen when viewing the display screen as an example.

[0159] It should be noted that the above division of the image output by the SOC into 9 areas is only an example and does not constitute a limitation of the present application.

[0160] For example, in some embodiments, the number of regions may be greater.

[0161] For example, the image output by the SOC may also include: Fig.12 The 25 regions shown ( Fig.12 Area 21 to Area 35), where A3+A4+A5+A6+A7=m2, B3+B4+B5+B6+B7=n2.

[0162] Similarly, the embodiments of the present application do not limit the values ​​of m2 and n2 and the specific values ​​of A3, A4, A5, A6, A7, B3, B4, B5, B6, B7, q3, q4, q5, q6, q7, p3, p4, p5, p6, and p7, as long as the relationship between them can satisfy the following relationship:

[0163] m2*n2*90*24bits / (1024^3) is less than or equal to 17Gbps, A3*q3+A4*q4+A5*q5+A6*q6+A7*q7=3840 and B3*p3+B4*p4+B5*q5+B6*p6+B7*p7=3840.

[0164] Accordingly, when displaying on the display screen, for the image transmitted by the SOC, the gaze area is not upsampled, and at least one other area except the gaze area is upsampled using a corresponding upsampling ratio, so that a 3840*3840 image is displayed on the display screen.

[0165] Exemplarily, if the user's gaze area is area 23, then q5 and p5 can be set to 1 to ensure the clarity of the gaze area.

[0166] Optionally, in this example, when other regions except region 23 of the SOC transmission are displayed by upsampling, a larger upsampling ratio is used when upsampling a region farther away from region 23 .

[0167] For example, Fig.13As shown, q3 takes the value of 6, q4 takes the value of 3, q5 takes the value of 1, q6 takes the value of 3, q7 takes the value of 6, p3 takes the value of 6, p4 takes the value of 3, p5 takes the value of 1, p6 takes the value of 3, and p7 takes the value of 6. Accordingly, when the head mounted device is displayed on the display screen, the horizontal and vertical directions of the areas 11, 15, 31, and 35 of the SOC transmission are upsampled by 6 times, the horizontal directions of the areas 12, 14, 32, and 34 of the SOC transmission are upsampled by 3 times and the vertical directions are upsampled by 6 times, the horizontal directions of the areas 16, 20, 26, and 30 of the SOC transmission are upsampled by 6 times and the vertical directions are upsampled by 3 times, and the horizontal directions of the areas 13 and 33 of the SOC transmission are not upsampled, but the vertical directions are upsampled by 6 times. Upsampling: the horizontal and vertical directions of areas 17, 19, 27 and 29 of SOC transmission are upsampled by 3 times, the horizontal direction of areas 21 and 25 of SOC transmission are upsampled by 6 times, and the vertical direction is not upsampled, the horizontal direction of areas 18 and area 28 of SOC transmission are not upsampled, but the vertical direction is upsampled by 3 times, the horizontal direction of areas 22 and area 24 of SOC transmission are upsampled by 3 times, and the vertical direction is not upsampled, and the horizontal and vertical directions of area 23 of SOC transmission are not upsampled.

[0168] For another example, in some embodiments, the number of regions may be even smaller.

[0169] For example, the image output by the SOC may also include: Fig.14 The four areas shown ( Fig.14 Area 130 to area 133), where A8+A9=m3, B8+B9=n3.

[0170] Similarly, the embodiments of the present application do not limit the values ​​of m3 and n3 and the specific values ​​of A8, A9, B8, B9, q8, q9, p8, and p9, as long as the relationship between them can satisfy the following relationship: m3*n3*90*24bits / (1024^3) is less than or equal to 17Gbps, A8*q8+A9*q9=3840, and B8*p8+B9*p9=3840.

[0171] Accordingly, when the head mounted device displays on the display screen, for the image transmitted by the SOC, the gaze area is not upsampled, while at least one other area except the gaze area is upsampled using the corresponding upsampling ratio, so that a 3840*3840 image is displayed on the display screen.

[0172] Exemplarily, if the user's gaze area is area 132, then q8 and p9 may be set to 1 to ensure clarity of the gaze area.

[0173] Optionally, in this example, when other regions except region 132 of the SOC transmission are displayed by upsampling, a larger upsampling ratio is used when upsampling a region farther away from region 132.

[0174] It should be noted that the above description is based on the example that the display screen output capacity is 3840*3840*90HZ and the maximum transmission bandwidth supported by the SOC is 17Gbps / panel. However, it should be understood that this example does not constitute a limitation of the present application. For example, it can be extended to any scenario where a display screen displays data transmitted by receiving the SOC.

[0175] Specifically, assuming that the display screen that displays images for the user is called a first display screen, and the first display screen is included in an electronic device, the display method provided in an embodiment of the present application includes: displaying a second image on the first display screen based on the first image output by the SOC.

[0176] Specifically, the resolution of the first image output by the SOC is smaller than the resolution of the second image output on the first display screen. It should be understood that the resolution of the first image output by the SOC is smaller than the resolution of the second image output by the first display screen, which can ensure the refresh rate when the maximum bandwidth supported by the SOC is insufficient to drive the first display screen.

[0177] Specifically, the first image output by the SOC is divided into K areas (referred to as K first areas). Accordingly, when displaying, the electronic device will not upsample the content of the gaze area (also referred to as the first target area) transmitted by the SOC to ensure the clarity of the gaze area, but will upsample each area other than the gaze area transmitted by the SOC using a corresponding upsampling ratio (including an upsampling ratio in the horizontal direction and / or an upsampling ratio in the vertical direction) to obtain K second areas corresponding to the K first areas.

[0178] That is to say, in the embodiment of the present application, the second image displayed on the first display screen includes K second areas, the K second areas include a second target area, the content of the second target area is the content of the first target area and the position of the second target area in the image to which they belong is the same. And, the second area other than the second target area in the K second area is obtained based on the corresponding first area transmitted by the SOC using the corresponding upsampling ratio.

[0179] More specifically, in an embodiment of the present application, different first areas among the K first areas correspond to different second areas among the K second areas, and the position of each first area in the first image is the same as the position of the corresponding second area in the second image; or in other words, different first areas among the K first areas correspond to different second areas among the K second areas, and the positional relationship between each first area and the gaze area in the first image is the same as the positional relationship between the corresponding second area and the gaze area in the second image. Or in other words, in an embodiment of the present application, the K first areas correspond one-to-one with the K second areas, and the positions of a first area in the first image and its corresponding second area in the second image are the same in their respective images. Or in other words, in an embodiment of the present application, the first image output by the SOC and the respective areas in the second image output by the first display screen are aligned in position.

[0180] It should be understood that, through this method, there is no need to complete the transmission of the first target area before starting to display, so that the SOC can display on the first display screen while transmitting data, thereby reducing the display delay.

[0181] Further, when the second image is displayed on the first display screen based on the first image, a larger upsampling ratio is used when upsampling the first area farther from the first target area among the K first areas. In other words, when the second image is displayed on the first display screen based on the first image, a larger upsampling ratio is used when the second area farther from the second target area among the K second areas is obtained based on the corresponding first area.

[0182] It can be understood that in this implementation, when the second image is displayed on the first display screen based on the first image, the upsampling ratio used when upsampling the first area farther from the first target area among the K first areas is larger, therefore, the K second areas displayed on the first display screen have the following characteristics: the second area farther from the second target area in the second image is less clear, while the second area closer to the second target area is clearer. That is, in this implementation, when the K second areas are displayed on the first display screen, the clarity of the second areas at different distances from the second target area is different.

[0183] In some embodiments, the electronic device also includes a second display screen. In this scenario, the method of the embodiment of the present application also includes: based on the third image output by the SOC, displaying a fourth image on the second display screen, the resolution of the third image is less than the resolution of the fourth image, the third image includes K third areas, the fourth image includes K fourth areas, different third areas in the K third areas correspond to different fourth areas in the K fourth areas, and the position of each third area in the third image is the same as the position of the corresponding fourth area in the fourth image; wherein the resolution of the third target area in the K third areas is the same as the resolution of the corresponding fourth target area in the K fourth areas, and the content of the third target area is the same as the content of the fourth target area, and the content of each fourth area in the K fourth areas except the fourth target area is obtained by upsampling the content of the corresponding third area.

[0184] Among them, this solution can be compared to the display method of the first display screen in the first aspect, and will not be repeated here.

[0185] It should be noted that the embodiments of the present application do not limit the specific form between the SOC and the electronic device.

[0186] For example, in some embodiments, the SOC is included in an electronic device.

[0187] For another example, in another possible implementation, the SOC is not included in the electronic device, that is, the SOC and the electronic device are independent of each other.

[0188] Fig.15 A schematic diagram of the structure of a processing device of a terminal device provided in another embodiment of the present application. Fig.15 As shown, the processing device 1500 may include a processor 1501 and an interface circuit 1502. The processor 1501 and the interface circuit 1502 are coupled to each other. It is understood that the interface circuit 1502 may be a transceiver or an input-output interface. Optionally, the processing device 1500 may also include a memory 1503 for storing instructions executed by the processor 1501 or storing input data required by the processor 1501 to execute instructions or storing data generated after the processor 1501 executes instructions.

[0189] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0190] The method steps in the embodiments of the present application can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also be present in a network device or a terminal device as discrete components.

[0191] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc; it may also be a semiconductor medium, for example, a solid-state hard disk.

[0192] In the various embodiments of the present application, unless otherwise specified or provided for in any logical conflict, the terms and / or descriptions between the different embodiments are consistent and may be referenced to each other, and the technical features in the different embodiments may be combined to form new embodiments according to their inherent logical relationships.

[0193] It is understood that the various numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic.

Claims

1. A display method, characterized in that: Applied to an electronic device, the electronic device includes a first display screen, and the method includes: When the first display screen receives the first image, the first display screen displays the second image, the resolution of the first image is smaller than the resolution of the second image, the first image includes K first areas, the second image includes K second areas, different first areas among the K first areas correspond to different second areas among the K second areas, and the position of each first area in the first image is the same as the position of the corresponding second area in the second image; The K first areas include a first target area, the K second areas include a second target area corresponding to the first target area, the first target area and the second target area have the same resolution, and the resolution of each of the K second areas except the second target area is higher than the resolution of the corresponding first area.

2. The method according to claim 1, characterized in that Each second region among the K second regions except the second target region is obtained by upsampling the first region corresponding to the respective second regions. The farther the second region among the K second regions is from the second target region, the greater the upsampling ratio corresponding to the second region.

3. The method according to claim 1 or 2, characterized in that: The electronic device also includes a second display screen, and the method further includes: When the second display screen receives the third image, the second display screen displays a fourth image, the resolution of the third image is smaller than the resolution of the fourth image, the third image includes K third areas, the fourth image includes K fourth areas, different third areas among the K third areas correspond to different fourth areas among the K fourth areas, and the position of each third area in the third image is the same as the position of the corresponding fourth area in the fourth image; The K third areas include a third target area, the K fourth areas include a fourth target area corresponding to the third target area, the third target area and the fourth target area have the same resolution, and the resolution of each fourth area among the K fourth areas except the fourth target area is higher than the resolution of the corresponding third area.

4. The method according to claim 3, characterized in that The first image received by the first display screen is an image output by a system-on-chip (SOC).

5. The method according to claim 4, characterized in that The SOC is included in the electronic device.

6. The method according to claim 4, characterized in that The SOC is not included in the electronic device.

7. The method according to any one of claims 1 to 6, characterized in that Both the first image and the second image can be divided into nine areas in three rows and three columns: a central area located at the center of the image, an upper left corner area located at the upper left corner of the central area, an upper transition area located above the central area, an upper right corner area located at the upper right corner of the central area, a left transition area located to the left of the central area, a right transition area located to the right of the central area, a lower left corner area located at the lower left corner of the central area, a lower transition area located below the central area, and a lower right corner area located at the lower right corner of the central area; The first target area is the central area of ​​the first image, the second target area is the central area of ​​the second image, and the first target area and the second target area have the same resolution, including: the central area of ​​the second image has the same resolution as the central area of ​​the first image.

8. The method according to claim 7, characterized in that Based on upsampling the upper left corner area in the first image by q1 times in the horizontal direction and by p1 times in the vertical direction, the upper left corner area in the second image is obtained; based on upsampling the left transition area in the first image by q1 times in the horizontal direction and without upsampling in the vertical direction, the left transition area in the second image is obtained; based on not upsampling the upper transition area in the horizontal direction and upsampling it by p1 times in the vertical direction in the upper transition area in the first image, the upper transition area in the second image is obtained, wherein both q1 and p1 are greater than 1.

9. The method according to any one of claims 1 to 8, characterized in that The electronic device is a head-mounted display device.

10. A display device, characterized in that: include: Memory and processor; The memory is used to store program instructions; The processor is configured to execute program instructions in the memory to implement the method according to any one of claims 1 to 9.

11. An electronic device, characterized in that: It includes one or more display screens, one or more memories, and one or more processors; wherein the one or more memories store one or more programs; it is characterized in that when the one or more processors execute the one or more programs, the electronic device executes the display method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the display method as claimed in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises instructions for implementing the method according to any one of claims 1 to 9.

Images