Display method and device and electronic equipment

Abstract

The invention provides a display method and device and electronic equipment. The method comprises the following steps: acquiring an image cache of a to-be-rendered image after secondary rendering, wherein the rendering frame rate of secondary rendering is a first frame rate; determining a first frame image according to the image cache; sending and displaying a first frame image; performing displacement compensation on the first frame image to obtain a second frame image; and displaying the first frame image and the second frame image at a second frame rate, wherein the second frame rate is greater than the first frame rate. According to the display method, in the scene of reducing the rendering frame rate, in order to avoid display lagging possibly caused by the fact that the rendering frame rate is smaller than the display frame rate, displacement compensation is carried out on the image at the screen end before the image is displayed, so that a good display effect is guaranteed while time delay and energy consumption are saved.

Description

Technical Field

[0001] The present application relates to the field of electronic devices, and in particular to a display method, device and electronic device. Background Art

[0002] With the development of science and technology, in electronic devices that can realize display functions, a certain display refresh rate is usually required to meet the user's requirements for display effects. However, in order to meet a certain display refresh rate, electronic devices usually need to perform a large number of operations. If electronic devices render and display frame by frame, it will cause delays and high energy consumption.

[0003] Therefore, how to reduce latency and save energy while ensuring good display effects is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a display method, device and electronic device. The method can reduce the rendering frame rate while reducing the display delay and energy consumption, ensuring a non-stuttering display and providing users with a good display effect.

[0005] In a first aspect, a display method is provided, comprising: obtaining an image cache after secondary rendering of an image to be rendered, the rendering frame rate of the secondary rendering being a first frame rate; determining a first frame image according to the image cache; displaying the first frame image; obtaining a second frame image by displacement compensation for the first frame; and displaying the first frame image and the second frame image at a second frame rate, the second frame rate being greater than the first frame rate.

[0006] It should be noted that the first frame rate is the rendering frame rate, and the second frame rate is the display frame rate. A lower rendering frame rate of the secondary rendering can reduce latency and save power consumption.

[0007] Exemplarily, the first frame may be an odd frame, and the second frame may be an even frame. The first frame image obtained after rendering may be directly displayed on the screen. After the first frame image is sent to the display driver integrated circuit chip (DDIC), displacement compensation processing is performed to obtain the second frame image, thereby displaying the second frame image after the compensation processing of the first frame image. It can be seen that the second frame image can reuse the image cache of the previous frame. The second frame image is not a new image obtained by re-rendering, but is obtained by performing displacement compensation on the screen end based on the first frame image.

[0008] Based on the above method, in the scenario of reducing the rendering frame rate, in order to avoid display stuttering caused by the rendering frame rate being lower than the display frame rate, the image is displacement compensated on the screen before displaying the image, so as to save latency and energy consumption while ensuring a good display effect.

[0009] In combination with the first aspect, in certain implementations of the first aspect, before performing displacement compensation on the first frame image, the method further includes: predicting a compensation displacement amount of the first frame image.

[0010] In combination with the first aspect, in certain implementations of the first aspect, predicting the compensation displacement of the first frame image includes: predicting the compensation displacement according to the head angular velocity, and the compensation displacement includes a horizontal displacement and / or a longitudinal displacement.

[0011] Exemplarily, a displacement sensor such as an inertial measurement unit (IMU) can be used to predict the compensation displacement frame by frame according to the angular velocity of the head movement. The display method in the present application can be applied to an electronic device, which can be an augmented reality (AR) glasses device. When a user wears an AR glasses device, the compensation displacement can be predicted by the user's head angular velocity. Exemplarily, the displacement compensation amount can also be predicted by parameters such as the user's head movement velocity and head movement displacement.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the image cache includes a first layer, a second layer, and a third layer, the first layer includes first coding information, and the first layer also includes second coding information and / or third coding information; the first coding information is used to indicate whether the displacement compensation is enabled; wherein the second coding information is used to indicate the horizontal displacement amount, and the third coding information is used to indicate the vertical displacement amount; the second layer is used to represent the rendering image of the left eye, and the third layer is used to represent the rendering image of the right eye.

[0013] Exemplarily, the first encoding information is Orbit_En, the second encoding information is Orbit_x, and the third encoding information is Orbit_y.

[0014] It should be noted that the display method of the present application can be implemented by using image coding and decoding, wherein the first layer is a coding image, and the first layer includes a plurality of coding information.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the first layer includes first data, wherein determining the first frame image based on the image cache includes: merging the first data, the second layer, and the third layer to obtain a fourth layer; decoding the fourth layer to obtain first encoding information of the first data, wherein the first encoding information of the first data is used to indicate that the displacement compensation is not enabled; and superimposing the second layer and the third layer to obtain the first frame image.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the first layer includes second data, wherein the second frame image is obtained by performing displacement compensation on the first frame image, including: merging the second data, the second layer and the third layer to obtain a fifth layer; decoding the fifth layer to obtain first coding information of the second data, the second coding information and / or the third coding information, wherein the first coding information of the second data is used to indicate enabling the displacement compensation; performing the displacement compensation on the first frame image according to the second coding information and / or the third coding information to obtain the second frame image.

[0017] It should be noted that before the first frame image and the second frame image are displayed, the corresponding layers need to be decoded. Before the first frame image is displayed, the fourth layer is decoded; before the second frame image is displayed, the fifth layer is decoded. According to the encoding information obtained from the decoded layer, it is determined whether displacement compensation is required. The screen does not enable displacement compensation for odd frames and can directly display the first frame image; the screen enables compensation for even frames and displays the second frame image after displacement compensation.

[0018] Based on the above scheme, when the rendering frame rate is lower than the display frame rate, the higher display frame rate is maintained unchanged, and the odd frames are directly displayed on the screen. For even frames, the image cache of the previous frame can be reused, and corresponding compensation is performed according to the predicted compensation displacement, and then the displacement-compensated even frames are displayed on the screen, making the screen display image more smooth.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the image cache includes a second layer and a third layer, the second layer is used to represent a rendering image for a left eye, and the third layer is used to represent a rendering image for a right eye; the method also includes: obtaining configuration information of a third frame, the configuration information of the third frame includes first configuration information, and the first configuration information is used to indicate that the displacement compensation is not enabled; wherein the third frame is before the first frame; determining the first frame image based on the image cache includes: determining the first frame image based on the configuration information of the third frame and a sixth layer, the sixth layer being a combination of the second layer and the third layer.

[0020] It should be noted that the display method of the present application can be implemented by register writing. In this method, multiple layers obtained after secondary rendering can be merged, and displacement compensation can be enabled for the merged layers respectively, or displacement compensation can not be enabled. For frames that need to enable displacement compensation, the layer data after the previous frame rendering can be directly reused without re-rendering, which further saves energy consumption.

[0021] Exemplarily, the first configuration information is Orbit_En=0.

[0022] It should be noted that there is a certain delay in the effectiveness of configuration information, so the configuration information that needs to take effect in the current frame needs to be obtained in the previous frame.

[0023] In combination with the first aspect, some implementations of the first aspect

[0024] Exemplarily, the second configuration information is Orbit_En=1, the third configuration information is Orbit_x=x1, and the third coding information is Orbit_y=y1.

[0025] In combination with the first aspect, in some implementations of the first aspect, obtaining the configuration information of the first frame includes: writing the configuration information of the first frame serially; or writing the configuration information of the first frame in parallel.

[0026] In combination with the first aspect, in some implementations of the first aspect, obtaining the configuration information of the third frame includes: writing the configuration information of the third frame serially; or writing the configuration information of the third frame in parallel.

[0027] It should be noted that the specific methods of using registers to write configuration information include but are not limited to serial writing and parallel writing. Among them, serial writing can be understood as writing to the register during the porch idle time at the end of a frame. Parallel writing can be understood as using the serial communication (inter-integrated circuit, I2C) interface to write to the register while the mobile industry processor interface (mobile industry processor interface, MIPI) transmits data normally.

[0028] In combination with the first aspect, in some implementations of the first aspect, the method is applied to an augmented reality (AR) device or a virtual reality (VR) device.

[0029] In a second aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, cause the electronic device to perform the following steps: obtaining an image cache after secondary rendering of an image to be rendered, the rendering frame rate of the secondary rendering being a first frame rate; determining a first frame image based on the image cache; displaying the first frame image; obtaining a second frame image by compensating for the displacement of the first frame; and displaying the first frame image and the second frame image at a second frame rate, the second frame rate being greater than the first frame rate.

[0030] In combination with the second aspect, in certain implementations of the second aspect, when the instruction is executed by the one or more processors, the electronic device performs the following steps: before performing displacement compensation on the first frame image, predicting the compensated displacement amount of the first frame image.

[0031] In combination with the second aspect, in certain implementations of the second aspect, the predicted compensation displacement of the first frame image, when the instruction is executed by the one or more processors, causes the electronic device to perform the following steps: predicting the compensation displacement based on the head angular velocity, the compensation displacement including a horizontal displacement and / or a longitudinal displacement.

[0032] In combination with the second aspect, in certain implementations of the second aspect, the image cache includes a first layer, a second layer and a third layer, the first layer includes first coding information, and the first layer also includes second coding information and / or third coding information; the first coding information is used to indicate whether the displacement compensation is enabled; wherein the second coding information is used to indicate the horizontal displacement amount, and the third coding information is used to indicate the vertical displacement amount; the second layer is used to represent the rendering image of the left eye, and the third layer is used to represent the rendering image of the right eye.

[0033] In combination with the second aspect, in certain implementations of the second aspect, the first layer includes first data, and the first frame image is determined based on the image cache. When the instruction is executed by the one or more processors, the electronic device performs the following steps: merging the first data, the second layer, and the third layer to obtain a fourth layer; decoding the fourth layer to obtain first encoding information of the first data, and the first encoding information of the first data is used to indicate that the displacement compensation is not enabled; superimposing the second layer and the third layer to obtain the first frame image.

[0034] In combination with the second aspect, in certain implementations of the second aspect, the first layer includes second data, and the second frame image is obtained by performing displacement compensation on the first frame image. When the instruction is executed by the one or more processors, the electronic device performs the following steps: merging the second data, the second layer and the third layer to obtain a fifth layer; decoding the fifth layer to obtain first coding information of the second data, the second coding information and / or the third coding information, wherein the first coding information of the second data is used to indicate enabling the displacement compensation; performing the displacement compensation on the first frame image according to the second coding information and / or the third coding information to obtain the second frame image.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the image cache includes a second layer and a third layer, the second layer is used to represent a rendering image for a left eye, and the third layer is used to represent a rendering image for a right eye; when the instruction is executed by the one or more processors, the electronic device performs the following steps: obtaining configuration information of a third frame, the configuration information of the third frame includes first configuration information, and the first configuration information is used to indicate that the displacement compensation is not enabled; wherein the third frame is before the first frame; determining the first frame image based on the image cache includes: determining the first frame image based on the configuration information of the third frame and a sixth layer, the sixth layer being a combination of the second layer and the third layer.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the image cache includes a second layer and a third layer, the second layer is used to represent a rendering image for a left eye, and the third layer is used to represent a rendering image for a right eye; when the instruction is executed by the one or more processors, the electronic device performs the following steps: obtaining configuration information of the first frame, the configuration information of the first frame includes second configuration information, and the second configuration information is used to indicate enabling the displacement compensation; the configuration information of the first frame also includes third configuration information and / or fourth configuration information; wherein the third configuration information is used to indicate the horizontal displacement amount, and the fourth configuration information is used to indicate the vertical displacement amount; performing displacement compensation on the first frame image to obtain a second frame image includes: determining the second frame image according to the configuration information of the first frame and a sixth layer, and the sixth layer is obtained by merging the second layer and the third layer.

[0037] In combination with the second aspect, in certain implementations of the second aspect, the configuration information of the first frame is obtained. When the instruction is executed by the one or more processors, the electronic device performs the following steps: serially writes the configuration information of the first frame; or writes the configuration information of the first frame in parallel.

[0038] In combination with the second aspect, in certain implementations of the second aspect, the configuration information of the third frame is obtained. When the instruction is executed by the one or more processors, the electronic device performs the following steps: serially writes the configuration information of the third frame; or writes the configuration information of the third frame in parallel.

[0039] In combination with the second aspect, in some implementations of the second aspect, the electronic device is an augmented reality (AR) device or a virtual reality (VR) device.

[0040] In a third aspect, a display device is provided, comprising: a processor coupled to a memory, the memory being used to store a computer program, the processor being used to run the computer program, so that the device executes the method in the first aspect and any possible implementation thereof.

[0041] In combination with the third aspect, in some implementations of the third aspect, one or more of the memory and the transceiver are also included, and the transceiver is used to receive signals and / or send signals.

[0042] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer implements the method in the first aspect and any possible implementation thereof.

[0043] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method in the first aspect and any possible implementation thereof.

[0044] In a sixth aspect, a chip is provided, characterized in that the chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method in the first aspect and any possible implementation thereof.

[0045] In combination with the sixth aspect, in a possible implementation manner, the processor is coupled to the memory through an interface.

[0046] In combination with the sixth aspect, in a possible implementation, the chip system also includes a memory, in which a computer program or computer instructions are stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a structural diagram of an electronic device.

[0048] Figure 2 It is a software structure diagram of an electronic device.

[0049] Figure 3 It is a schematic diagram of a scenario provided in an embodiment of the present application.

[0050] Figure 4 It is a flow chart of a display method provided in an embodiment of the present application.

[0051] Figure 5 It is a schematic flow chart of a display method provided in an embodiment of the present application.

[0052] Figure 6 This is a schematic diagram of an orbit function provided in an embodiment of the present application.

[0053] Figure 7 This is a schematic flow chart of a method for implementing a display method using an image encoding and decoding method provided in an embodiment of the present application.

[0054] Figure 8It is a set of layer schematic diagrams provided in an embodiment of the present application.

[0055] Fig. 9 It is a set of layer schematic diagrams provided in an embodiment of the present application.

[0056] Fig.10 It is a schematic flowchart of a display method implemented by register writing provided in an embodiment of the present application.

[0057] Fig.11 It is a schematic diagram of a frame structure provided in an embodiment of the present application.

[0058] Fig.12 It is a schematic diagram of a display effect provided in an embodiment of the present application.

[0059] Fig.13 It is a schematic flow chart of a display method provided in an embodiment of the present application.

[0060] Fig.14 It is a schematic block diagram of a display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0062] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0063] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0064] The following describes an electronic device, a user interface for such an electronic device, and embodiments for using such an electronic device. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a wearable electronic device with wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to devices equipped with Or a portable electronic device with other operating systems. The portable electronic device may also be other portable electronic devices, such as a laptop computer. It should also be understood that in some other embodiments, the electronic device may not be a portable electronic device, but a desktop computer. In some other embodiments, the electronic device may be an instrument for measurement, such as an oscilloscope.

[0065] For example, Figure 11 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0066] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0067] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0068] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0069] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0070] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0071] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0072] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0073] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0074] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0075] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0076] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0077] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0078] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0079] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0080] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0081] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0082] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0083] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0084] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0085] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0086] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0087] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0088] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0089] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0090] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0091] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0092] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the alarm application icon, an instruction to create a new alarm is executed.

[0093] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc. For example, when the mobile phone detects a user's touch operation on the lock screen interface, the mobile phone can collect the user's fingerprint information through the fingerprint sensor 180H, and match the collected fingerprint information with the fingerprint information preset in the mobile phone. If the match is successful, the mobile phone can enter the non-lock screen interface from the lock screen interface.

[0094] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0095] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0096] like Figure 2 As shown, the application layer may include camera, settings, skin module, user interface (UI), third-party applications, etc. Among them, the third-party applications may include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0097] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer may include some predefined functions.

[0098] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0099] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0100] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can be composed of one or more views.

[0101] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0102] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0103] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0104] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0105] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0106] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0107] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0108] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0109] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0110] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0111] A 2D graphics engine is a drawing engine for 2D drawings.

[0112] In addition, the system library may also include status monitoring service modules, such as a physical status recognition module for analyzing and identifying user gestures; a sensor service module for monitoring sensor data uploaded by various sensors at the hardware layer to determine the physical status of the electronic device 100.

[0113] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0114] The hardware layer can include various types of sensors, such as Figure 1 Various types of sensors introduced in.

[0115] Combined with the above Figure 1 and Figure 2 The electronic device introduced, in the embodiment of the present application, can be a mobile phone terminal, a tablet computer, or other device that can support display functions, or a device that supports functions such as augmented reality (AR) and virtual reality (VR). The physical devices involved in the electronic device 100 mainly include hardware components such as sensors, decision support systems (DSS) display chips, touch screens, fingerprint recognition modules, etc.; as well as kernel software layers such as screen management modules, display drivers, fingerprint drivers, and anti-mistouch; application framework layer functions such as anti-mistouch input, screen control, always on display (AOD) services, and power management; as well as application layer services such as special adaptation applications (camera), third-party applications, system sleep, and AOD.

[0116] With the development of science and technology, electronic devices can provide users with a good visual experience through display functions. Especially in the field of AR or VR, such as Figure 3As shown, a use scenario of AR glasses is shown. AR glasses can display a user interface for users, providing users with a novel visual experience and interaction method. Before AR glasses display the user interface, the image needs to be rendered once, rendered twice, and sent to the display. In other words, rendering the image frame by frame before sending the image to the display will cause high latency and energy consumption, but if the rendering frame rate is reduced, the display effect will also be affected.

[0117] Take AR or VR as an example. Figure 4 As shown in the figure, the intermediate frame generation technology is adopted. An asynchronous time warp (ATW) thread is added. This thread can call the previous frame data and predict the new frame based on the head movement changes. The previous frame (Nth frame) and the new frame (N+1th frame) are both rendered twice and sent to the display, keeping the sending frame rate consistent with the display frame rate. Figure 4 As can be seen from the figure, the intermediate frame technology can reduce the frame rate of the first rendering through the ATW thread, but the secondary rendering after the ATW thread (including anti-distortion, geometric rasterization, shading and other steps) is still processed at the same frame rate as the display frame rate, which occupies a lot of computing power of the electronic device. Since the secondary rendering frame rate is consistent with the display frame rate, although it can ensure a good display effect, it still leads to higher latency and energy consumption.

[0118] It can be seen that in order to reduce latency and energy consumption, the secondary rendering frame rate needs to be reduced. However, when the rendering frame rate is lower than the display frame rate (for example, the secondary rendering frame rate is 30Hz and the display frame rate is 60Hz), it will cause display stuttering and other problems. Therefore, in order to solve the above problems, the present application provides a display method, which compensates for the display effect while reducing latency and energy consumption at a low rendering frame rate, thereby ensuring a display without stuttering.

[0119] like Figure 5 As shown, an exemplary flow chart of a display method provided by an embodiment of the present application is shown. The method 500 can be applied to the AR or VR field, and can specifically include the following steps.

[0120] S501, input compensation displacement.

[0121] S502, secondary rendering.

[0122] S503, orbit is closed and sent to display.

[0123] S504, orbit is enabled and sent for display.

[0124] It should be noted that Figure 5 Not all of the steps shown need to be performed.

[0125] It should be noted that the final display mode of the image may be interpolation display. For example, the Nth frame is an odd frame, and the N+1th frame is an even frame. It can also be understood that the odd frame is the previous frame.

[0126] For odd-numbered frames, they may be processed through primary rendering, secondary rendering, and display sending, and finally displayed on the user interface.

[0127] For even-numbered frames, the previous frame image buffer can be reused, that is, the odd-numbered frames can be reused. The screen end (e.g., a display or a module capable of realizing a display function) is enabled to move the image along a specified direction and / or track (orbit), and the even-numbered frames are compensated according to the displacement compensation amount, and finally the compensated even-numbered frames are displayed on the user interface.

[0128] The rendering frame rate used for the second rendering of the odd and even frames in step S502 is the same, and the rendering frame rate is less than the display frame rate, for example, the rendering frame rate is 30 Hz and the display frame rate is 60 Hz. After the second rendering, there is no need to perform displacement compensation on the odd frames, and they can be directly sent to the display, while the even frames need to be subjected to displacement compensation before being sent to the display.

[0129] Specifically, the compensation displacement amount may be input before step S502.

[0130] After the second rendering of the image in step S502, the odd-numbered frames and the even-numbered frames are processed respectively.

[0131] Step S503 is executed for odd frames, and no displacement compensation is performed according to the displacement compensation amount input in step S501, that is, the orbit function is turned off, the images are directly sent to the screen, and the odd frames are displayed on the screen.

[0132] Step 504 is executed for the even-numbered frames, and displacement compensation needs to be performed according to the displacement compensation amount input in step S501, that is, the orbit function is enabled, the displacement-compensated even-numbered frames are sent to the display end, and the compensated even-numbered frames are displayed on the screen end.

[0133] It should be noted that the orbit function can be understood as the movement of the screen along a specified direction and / or track, and the movement direction can be horizontal and / or vertical (x / y lines). For ease of understanding, Figure 6 As shown, a schematic diagram of orbit function is shown.

[0134] like Figure 6 (a) in FIG. 1 shows an image area, and the specific content displayed in the image area is three rows of "ABCDEF" (referred to as: content a). Figure 6 The content a shown in (a) in the figure is moved upward to obtain Figure 6The image area shown in (b).

[0135] Since content a moves upward, such as Figure 6 The chicken leg content displayed in the image area shown in (b) is not a complete three-line parallel "ABCDEF" (referred to as: content b). Figure 6 The image area shown in (b) includes area 601 and area 602. Area 601 is the portion of content b that is not displayed compared to content a; area 602 indicates that no content is displayed.

[0136] Similarly, Figure 6 The content a shown in (a) in the figure is moved to the left, and the result is as follows Figure 6 The image area shown in (c) is Figure 6 The content a shown in (a) in the figure is moved to the right, and the result is as follows Figure 6 The image area shown in (d) is Figure 6 The content a shown in (a) in the figure is moved downward to obtain Figure 6 The image area shown in (e). Figure 6 The image area shown in (c) Figure 6 The image area shown in (d) and Figure 6 The image area shown in (e) is Figure 6 The specific contents displayed in the image area shown in (b) are similar, both cannot fully display content a, and both include areas 601 and 602.

[0137] It is understandable that the maximum displacement of the orbit function is related to the parameters of the screen (e.g., the pixels of the screen). Generally speaking, the maximum displacement can be set before the screen leaves the factory. The maximum displacement can be determined according to the following formula (1).

[0138] X=v / f*ppd (1)

[0139] Wherein, X represents the maximum displacement; v represents the head angular velocity, in units of degrees / s; f represents the display frame rate, in units of Hz; ppd represents the display resolution (pixel per degree).

[0140] According to the above Figure 5 As can be seen from the introduction, even frames need to be displacement compensated before being sent for display. The compensation displacement amount needs to be input, which involves related processes such as transmission compensation displacement amount and effective compensation displacement amount. The related processes can be implemented in different ways. The following introduces several specific implementation methods.

[0141] Method 1: Image encoding and decoding method.

[0142] like Figure 7As shown, a schematic flow chart of a method for realizing display by using image coding and decoding is shown. The method can be applied to the field of AR or VR, and the method is specifically introduced by taking 45Hz input of data source and 90Hz display as an example.

[0143] S701, predicting the compensation displacement.

[0144] For example, the compensation displacement can be predicted by using a displacement sensor such as an inertial measurement unit (IMU) in combination with the head angular velocity, wherein the compensation displacement includes a horizontal displacement and / or a longitudinal displacement (x / y).

[0145] S702, secondary rendering to obtain layer 1, layer 2 and layer 3.

[0146] For example, layer 1 may be a coded image generated according to the compensation displacement predicted in step S701. Layer 2 and layer 3 represent rendering images for the left and right eyes, respectively.

[0147] Exemplarily, layer 1, layer 2, and layer 3 constitute an image buffer. These three layers can be understood as layers of the SurfaceFlinger thread.

[0148] For ease of understanding, Figure 8 As shown, a set of layer schematic diagrams are shown. Figure 8 Layer 1, layer 2 and layer 3 are shown respectively.

[0149] Specifically, each grid in the layer 1 coding map represents a pixel. In addition, the coding information in layer 1 includes Orbit_En (used to indicate whether the orbit function is enabled), Orbit_x (used to indicate the predicted horizontal displacement) and Orbit_y (used to indicate the predicted vertical displacement). The coding map includes first data and second data, and each row of data includes the above coding information. When Orbit_En is 1, it means that displacement compensation is required; when Orbit_En is 0, it means that displacement compensation is not required. Layer 2 and layer 3 represent the renderings of the left and right eyes respectively. Exemplarily, the first data of the coding map (layer 1) can correspond to layer 2, and the second data can correspond to layer 3. Among them, in the first data, Orbit_En is 0, or the grid corresponding to Orbit_En is red; the four grids corresponding to Orbit_x are red, indicating that there is no compensation displacement in the horizontal direction; the four grids corresponding to Orbit_y are green, indicating that there is compensation displacement in the vertical direction, and the y coordinate uses 4 bits, which can be moved down 16 pixels relative to the previous frame. In the second data, Orbit_En is 1, or the grid corresponding to Orbit_En is green; the four grids corresponding to Orbit_x are red; and the four grids corresponding to Orbit_y are green. It can be seen that the coding diagram can reflect the compensation displacement predicted in step S701. In layer 1, the Orbit_En in different rows of data may be different, but the contents represented by Orbit_x and Orbit_y are consistent. Among them, the coding in layer 1 uses red to correspond to 0 and green to correspond to 1 is only an example. In actual application, grayscale or any other color can be used.

[0150] S703, merging the layers to obtain layer 4.

[0151] In one implementation, after the image cache enters the image processing task (SurfaceFlinger) thread, the layers may be cropped and merged.

[0152] Among them, the SurfaceFlinger thread is a special process of the Android system, which is mainly responsible for synthesizing Surface to Framebuffer, and the screen side reads Framebuffer and displays it.

[0153] For example, for odd frames, the first row of layer 1 is cropped and covered on the black area after distortion correction in the upper left corner of layer 2 and layer 3, and in order to ensure that the display is not affected, the layers are merged to obtain layer 4. Layer 4 includes the first data of layer 2, layer 3 and layer 1. Since odd frames do not need to be displaced, Orbit_En in the first data of layer 1 is 0.

[0154] S704, merging the layers to obtain layer 5.

[0155] For example, for even-numbered frames, the image cache obtained in step S702 can be directly reused, the second row of layer 1 can be cropped, and covered on layer 2 and the black area in the upper left corner after distortion correction of layer 3, and in order to ensure that the display is not affected, the layers are merged to obtain layer 5. Layer 5 includes layer 2, layer 3, and the second data of layer 1. There are even-numbered frames that need to be displaced, so Orbit_En in the second data of layer 1 is 1.

[0156] For ease of understanding, a set of layer schematic diagrams are shown. Fig. 9 Layer 4 and layer 5 are shown respectively.

[0157] Specifically, the first data of layer 1 is cropped and covered in the distortion-corrected black area (white dotted frame) in the upper left corner of layer 2 and layer 3, and layer 2 and layer 3 covered with the first data of layer 1 are merged to obtain layer 4. By enlarging the white dotted frame, it can be clearly seen that the encoding information of the first data of layer 1 includes Orbit_En, Orbit_x and Orbit_y.

[0158] Similarly, the second data of layer 1 is cropped and covered on the distortion-corrected black area (white dotted frame) in the upper left corner of layer 2 and layer 3, and layer 2 and layer 3 covered with the second data of layer 1 are merged to obtain layer 5. By enlarging the white dotted frame, it can be clearly seen that the encoding information of the second data of layer 1 includes Orbit_En, Orbit_x and Orbit_y. Among them, layer 4 corresponds to odd frames, layer 5 corresponds to even frames, and the Orbit_En in the encoding information of layer 4 is different from the Orbit_En in the encoding information of layer 5. Fig. 9 It can also be seen that the grayscale of the grid corresponding to Orbit_En of layer 4 is different from the grayscale of the grid corresponding to Orbit_En of layer 5.

[0159] S705, orbit is closed and sent to display.

[0160] It should be noted that the layer 4 obtained by merging the layers in step S703 is sent to the display driver integrated circuit chip (DDIC), and the DDIC decodes the layer 4 to obtain the orbit setting information, and sets the register according to the orbit setting information. Among them, the orbit setting information indicates that no displacement compensation is performed. The register does not enable the orbit function according to the orbit setting information, and directly displays the odd frame. In other words, even if the Orbit_y in the first data of layer 1 indicates that there is a compensation displacement in the longitudinal direction, since Orbit_En is 0, no displacement compensation processing will be performed. It can be understood that the register is carried on the screen end. The screen end can support the orbit function.

[0161] S706, orbit is enabled and sent for display.

[0162] It should be noted that the layer 5 obtained by merging the layers in step S704 is sent to the DDIC for display. The DDIC decodes the layer 5 to obtain the orbit setting information, and sets the register according to the orbit setting information. The orbit setting information indicates that displacement compensation is to be performed. The register starts the orbit function according to the orbit setting information, and moves the even frame up or down by 16 pixels relative to the previous frame (odd frame).

[0163] It should be noted that in step S706, the screen end supporting the orbit function can be used to compensate for the displacement of the even-numbered frames and then interpolate and display them.

[0164] Based on the above solution, the rendering frame rate is reduced, the compensation displacement is predicted, and the predicted compensation displacement is transmitted and validated by image encoding and decoding. The orbit function is enabled on the screen to perform displacement compensation on even frames, and finally insert frames for display on the screen. While reducing the rendering frame rate, saving time and energy consumption, it also avoids the problem of display freeze caused by low rendering frame rate.

[0165] Method 2: register writing method.

[0166] like Fig.10 As shown, a schematic flow chart of a method for realizing display by register writing is shown. The method can be applied to the field of AR or VR. Taking 45Hz data source input and 90Hz display as an example, the method is specifically introduced below.

[0167] S1001, prediction of compensation displacement.

[0168] Exemplarily, the compensation displacement can be predicted by IMU in combination with the head angular velocity. The compensation displacement includes the horizontal displacement and / or the longitudinal displacement (x / y). The compensation displacement can be used as a variable transmission parameter.

[0169] S1002, secondary rendering to obtain layer 6 and layer 7.

[0170] Exemplarily, layer 6 and layer 7 represent the renderings of the left and right eyes respectively. Layer 6 is similar to layer 2 obtained in step S702 above, and layer 7 is similar to layer 3 obtained in step S702 above. Layer 6 and layer 7 constitute an image buffer. These two layers can be understood as the layers of the SurfaceFlinger thread.

[0171] S1003, merging the layers to obtain layer 8.

[0172] In one implementation, when the image cache enters the SurfaceFlinger thread, the layers can be merged.

[0173] Exemplarily, layer 6 and layer 7 representing the left and right eye renderings respectively are merged into layer 8.

[0174] S1004, write configuration information A.

[0175] It should be noted that, for odd frames, while the layer 8 obtained by merging in step S1003 is sent to DDCI via the mobile industry processor interface (MIPI), configuration information A is written by register writing. Configuration information A will take effect in the next frame. In other words, configuration information A written in an odd frame will take effect in the next frame (even frame) of the odd frame. It can be seen that configuration information A is written for even frames.

[0176] Exemplarily, the configuration information includes content similar to the coding information included in layer 1 (coding map) above. The configuration information includes Orbit_En, Orbit_x and Orbit_y.

[0177] Specifically, for configuration information A, Orbit_En=1, Orbit_x=x1, and Orbit_y=y1 in configuration information A. Configuration information A indicates that displacement compensation is required for the next frame of the odd frame, and the compensation displacement in the horizontal direction is x1 and the compensation displacement in the vertical direction is y1.

[0178] S1005, write configuration information B.

[0179] Exemplarily, for an even-numbered frame, data of a previous frame (odd-numbered frame) may be directly multiplexed.

[0180] It should be noted that for even frames, while layer 8 obtained by merging in step S1003 is sent to DDCI via MIPI, configuration information B is written by register writing. Configuration information B will take effect in the next frame. In other words, configuration information B written in an even frame will take effect in the next frame (odd frame) of the even frame. It can be seen that configuration information B is written for odd frames.

[0181] Specifically, for the configuration information B, Orbit_En in the configuration information B is 0. The configuration information B indicates that for the next frame of the even frame, it is not necessary to compensate for the displacement.

[0182] It should be noted that the embodiments of the present application do not limit the specific method of register writing configuration information, including serial writing and parallel writing. Among them, parallel writing can be understood as using the I2C interface to write to the register while the MIPI transmission data is displayed normally. Serial writing can be understood as writing to the register during the porch idle time at the end of a frame. For example, Fig.11 The frame structure includes display data, which is the data displayed by the actual pixels of the display screen, commands, which refer to the commands for configuring the display screen registers, vertical synchronization signals (VS), which are frame synchronization signals, horizontal synchronization signals (HS), which are scan synchronization signals, vertical front porch (VFP), vertical back porch (VBP), horizontal back porch (HBP), and horizontal front porch (HFP). Configuration information can be written into Fig.11 The Command position shown in .

[0183] It can be understood that the porch at the end of the frame is VFP. Similarly, the proch at the head of the frame is VBP. The porch at the head of the line is HBP. The porch at the end of the line is HFP.

[0184] S1006, do not enable orbit, send and display.

[0185] It should be noted that the layer 8 obtained by merging the layers in step S1003 is sent to DDIC for display. Since the configuration information of the previous frame (even frame) of the odd frame is configuration information B, configuration information B takes effect in the odd frame, and DDCI does not enable orbit function display according to configuration information B.

[0186] S1007, start orbit and send to display.

[0187] It should be noted that the layer 8 obtained by merging the layers in step S1003 is sent to the DDIC for display. Since the configuration information of the previous frame (odd frame) of the even frame is configuration information A, configuration information A is effective in the even frame, and DDIC enables the orbit function display according to configuration information A. The image moves x pixels horizontally and y pixels vertically relative to the previous frame (odd frame).

[0188] It should be noted that the register writing method is applicable to scenarios where data transmission supports the MIPI protocol or the I2C protocol, but the embodiments of the present application are not limited to the above two transmission protocols. For example, transmission protocols such as differential signaling technology (low-voltage differential signaling, LVDS), enhanced device protocol (enhanced device protocol, EDP), and serial peripheral interface (serial peripheral interface, SPI), and transmission protocols where the writing of the register does not affect the normal display are all within the scope of protection requested by this application.

[0189] Based on the above method, the rendering frame rate is reduced, the compensation displacement is predicted, and the predicted compensation displacement is transmitted and validated by register writing. The orbit function is enabled on the screen to perform displacement compensation on the even frames, and finally the interpolated frames are displayed on the screen. While reducing the rendering frame rate, saving time and energy consumption, it also avoids the problem of display freeze caused by low rendering frame rate.

[0190] For ease of understanding, Fig.12 A schematic diagram of a display effect is shown, such as Fig.12As shown, the display effect of three consecutive frames on the screen is shown. For example, a five-pointed star pattern is displayed on the screen. The rendering frame rate is 30Hz and the display frame rate is 60Hz. Obviously, the rendering frame rate is inconsistent with the display frame rate. At the Nth frame, the image is rendered normally and sent to the video end, and a five-pointed star pattern is displayed on the left side of the screen. At the N+2th frame, the image is rendered normally and sent to the video end, and a five-pointed star pattern is displayed on the right side of the screen. In the even frame (N+1th frame), it is predicted that the five-pointed star pattern will move from the left side of the screen to the right side of the screen, and the moving rate of the five-pointed star pattern from left to right is predicted according to the user's head movement angular velocity. The moving distance (Orbit_x) of the pattern is predicted, and the displacement of the even frame is compensated according to the predicted information before the N+1th frame is displayed on the screen end, so that the image displayed on the screen end is smoother and the jamming phenomenon is avoided.

[0191] like Fig.13 As shown, a schematic flow chart of a display method provided by an embodiment of the present application is shown. The method can be applied to AR or VR devices, and the method is described in detail below.

[0192] S1301, obtaining an image cache after secondary rendering of the image to be rendered.

[0193] It should be noted that the rendering frame rate of the secondary rendering is the first frame rate. Reducing the rendering frame rate of the secondary rendering can reduce latency and save power consumption.

[0194] Exemplarily, image caching can be understood as follows Figure 8 Schematic diagram shown.

[0195] S1302: Determine a first frame image according to the image cache.

[0196] S1303, sending the first frame image for display.

[0197] S1303, obtaining a second frame of image by compensating the displacement of the first frame of image.

[0198] It should be noted that the first frame is before the second frame, and the first frame may be a previous frame of the second frame.

[0199] Exemplarily, the first frame is the odd-numbered frame described above, and the second frame is the even-numbered frame described above.

[0200] In one implementation, before performing displacement compensation on the first frame image, the compensation displacement amount of the first frame image may be predicted.

[0201] It should be noted that the compensation displacement can be predicted according to the head movement angular velocity, wherein the compensation displacement includes horizontal displacement and / or longitudinal displacement.

[0202] It is understandable that when the AR device is an AR glasses device, the user can wear the AR glasses device, and the compensation displacement can be predicted according to the user's head angular velocity. Specifically, the device for predicting the compensation displacement can be a displacement sensor such as an IMU. The IMU predicts the compensation displacement frame by frame according to the head angular velocity.

[0203] It should be noted that the compensation displacement amount may be transmitted or the compensation displacement amount may be effective by image encoding and decoding or by register writing.

[0204] In one implementation, the display method provided in the embodiment of the present application may be implemented by using image coding.

[0205] Exemplarily, the image cache includes a first layer, a second layer, and a third layer. The first layer can be understood as follows: Figure 8 In the layer 1 (coded image) shown, the first layer includes first coding information, and the first layer also includes second coding information and / or third coding information; the first coding information is used to indicate whether displacement compensation is enabled, the second coding information is used to indicate the horizontal displacement amount, and the third coding information is used to indicate the vertical displacement amount. In addition, the second layer is used to represent the rendering image of the left eye, and the third layer is used to represent the rendering image of the right eye.

[0206] The first encoding information can be understood as Orbit_En, the second encoding information can be understood as Orbit_x, and the third encoding information can be understood as Orbit_y. The second layer can be understood as follows Figure 8 The layer 2 shown, the third layer can be understood as follows Figure 8 Layer 3 shown.

[0207] When the image cache includes the first layer, the second layer, and the third layer, the specific implementation method of determining the first frame image according to the image cache in the step S1302 may include:

[0208] The first layer includes the first data. The first data, the second layer and the third layer are combined to obtain the fourth layer; the fourth layer is decoded to obtain the first coding information of the first data, and the first coding information of the first data is used to indicate that the displacement compensation is not started; the second layer and the third layer are superimposed to obtain the first frame image.

[0209] Exemplarily, the fourth layer is as follows Fig. 9 The first encoding information Orbit_En after decoding of the fourth layer is 0, which indicates that the odd-numbered frames are not enabled for displacement compensation and can be directly displayed on the screen. That is, the first frame image displayed in the subsequent step S1305 is not subjected to displacement compensation.

[0210] When the image cache includes the first layer, the second layer and the third layer, the specific implementation method of obtaining the second frame image by performing displacement compensation on the first frame image in the above step S1304 may include:

[0211] The first layer includes the second data. The second data, the second layer and the third layer are combined to obtain a fifth layer; the fifth layer is decoded to obtain the first coding information, the second coding information and / or the third coding information of the second data, wherein the first coding information of the second data is used to indicate the start of displacement compensation; the first frame image is displacement compensated according to the second coding information and / or the third coding information to obtain the second frame image.

[0212] Exemplarily, the fifth layer is as follows Fig. 9 Layer 5 shown. The first encoding information Orbit_En after decoding the fifth layer is 1, which indicates that displacement compensation needs to be enabled for even frames. When the second encoding information Orbit_x is x1, the compensation displacement is the horizontal displacement x1; when the third encoding information Orbit_y is y1, the compensation displacement is the vertical displacement y1. The second frame image obtained after displacement compensation is transmitted to the screen end and displayed. That is to say, the second frame image displayed in the subsequent step S1305 is displacement compensated.

[0213] In another implementation manner, the display method provided in the embodiment of the present application may be implemented by writing to a register.

[0214] Exemplarily, the image cache includes a second layer and a third layer, the second layer is used to represent the rendering image for the left eye, and the third layer is used to represent the rendering image for the right eye.

[0215] In the above step S1302, before determining the first frame image according to the image cache, the second layer and the third layer may be merged to obtain the sixth layer. In this implementation, the second layer may be understood as layer 6 in the above step S1002, the third layer may be understood as layer 7 in the above step S1002, and the sixth layer may be understood as layer 8 in the above step S1003.

[0216] In the register writing method, the configuration information of the third frame can be obtained, and the configuration information of the third frame includes the first configuration information, and the first configuration information is used to indicate that the displacement compensation is not enabled. The third frame is before the first frame. The first frame image can be determined according to the configuration information of the third frame and the sixth layer.

[0217] That is to say, whether the current frame needs to be subjected to displacement compensation processing is determined by the configuration of the previous frame.

[0218] Exemplarily, the first configuration information may be as follows: Fig.10In the configuration information B described in , Orbit_En in the configuration information B is 0.

[0219] In addition, the configuration information of the first frame can also be obtained, the configuration information of the first frame includes second configuration information, and the second configuration information is used to indicate the activation of displacement compensation; the configuration information of the first frame also includes third configuration information and / or fourth configuration information. Among them, the third configuration information is used to indicate the horizontal displacement amount, and the fourth configuration information is used to indicate the vertical displacement amount. The second frame image can be determined according to the configuration information of the first frame and the sixth layer.

[0220] Exemplarily, the second configuration information may be as follows: Fig.10 The configuration information A in the configuration information A, Orbit_En in the configuration information A=1. The third configuration information is similar to the second encoding information above, and the fourth configuration information is similar to the third encoding information above.

[0221] It should be noted that, in the register writing method, the specific writing method of the configuration information of the first frame and / or the configuration information of the third frame includes but is not limited to serial writing and parallel writing. The specific writing method has been introduced above and will not be repeated here.

[0222] S1305: Display the first frame image and the second frame image using a second frame rate.

[0223] It should be noted that the second frame rate is greater than the first frame rate. The second frame rate can be understood as a display frame rate, that is, the rendering frame rate is less than the display frame rate.

[0224] Exemplarily, the first frame rate is 30 Hz and the second frame rate is 60 Hz. When the display frame rate is greater than or equal to 60 Hz, it can be considered as a high display frame rate.

[0225] That is, the first frame image and the second frame image can be displayed on the screen of the electronic device in an interpolation display manner. When the first frame is an odd frame and the second frame is an even frame, the odd frame and the even frame are alternately interpolated and displayed.

[0226] It should be noted that the first frame and the second frame may be two consecutive frames (for example, the first frame is an odd frame and the second frame is an even frame), and the second frame may also be a frame after the first frame and separated from the first frame by n frames. In other words, a display method provided in an embodiment of the present application may be applicable to alternate interpolation of odd frames and even frames, and may also be applicable to predictive interpolation every n (n=1, 2, 3, 4, 5) frames, so that data sources with different frame rates can be displayed at the same display frame rate.

[0227] Based on the above solution, in the scenario where the secondary rendering frame rate is reduced, the compensation displacement is predicted. Through the displacement compensation function of the screen, the image is compensated according to the compensation displacement. This ensures that the screen display has no ghosting or lag while achieving low latency and power consumption effects brought by the low secondary rendering frame rate.

[0228] Fig.14 is a schematic block diagram of a display device according to an embodiment of the present application. Fig.14 The display device 1400 shown may include: a processor 1410, a transceiver 1420, and a memory 1430. The processor 1410, the transceiver 1420, and the memory 1430 are connected via an internal connection path, the memory 1430 is used to store instructions, the processor 1410 is used to execute the instructions stored in the memory 1430, and the transceiver 1430 receives / sends some parameters. Optionally, the memory 1430 may be coupled to the processor 1410 via an interface, or may be integrated with the processor 1410.

[0229] It should be noted that the transceiver 1420 may include but is not limited to a transceiver device such as an input / output interface to achieve communication between the display device 1400 and other devices or a communication network.

[0230] In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 1410 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1430, and the processor 1410 reads the information in the memory 1430 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0231] It should also be understood that in the embodiment of the present application, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the processor may also include a non-volatile random access memory. For example, the processor may also store information about the device type.

[0232] The embodiment of the present application provides a computer program product, which, when executed on a device, enables the device to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above method-related embodiments, and will not be described in detail here.

[0233] The embodiment of the present application provides a readable storage medium, which contains instructions. When the instructions are executed on a device, the device executes the technical solution of the above embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0234] The embodiment of the present application provides a chip, which is used to execute instructions. When the chip is running, the technical solution in the above embodiment is executed. The implementation principle and technical effect are similar and will not be repeated here.

[0235] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0236] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units (modules) can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0237] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0238] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0240] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0241] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A display method, characterized in that: include: Obtain an image cache after secondary rendering of the image to be rendered, wherein the rendering frame rate of the secondary rendering is the first frame rate; Determine a first frame of image according to the image cache; Sending the first frame of image for display; Obtaining a second frame of image by compensating the displacement of the first frame of image; The first frame image and the second frame image are displayed using a second frame rate, and the second frame rate is greater than the first frame rate.

2. The method according to claim 1, characterized in that Before performing displacement compensation on the first frame of image, the method further includes: The compensation displacement of the first frame image is predicted.

3. The method according to claim 2, characterized in that Predicting the compensation displacement of the first frame of image includes: The compensation displacement is predicted according to the head angular velocity, and the compensation displacement includes a horizontal displacement and / or a longitudinal displacement.

4. The method according to any one of claims 1 to 3, characterized in that The image cache includes a first layer, a second layer and a third layer, the first layer includes first coding information, and the first layer also includes second coding information and / or third coding information; The first coding information is used to indicate whether the displacement compensation is enabled; The second coded information is used to indicate the horizontal displacement, and the third coded information is used to indicate the vertical displacement; The second layer is used to represent the rendering image for the left eye, and the third layer is used to represent the rendering image for the right eye.

5. The method according to claim 4, characterized in that The first layer includes first data, wherein determining the first frame image according to the image cache includes: Merging the first data, the second layer and the third layer to obtain a fourth layer; Decoding the fourth layer to obtain first encoding information of the first data, where the first encoding information of the first data is used to indicate that the displacement compensation is not enabled; The second layer and the third layer are superimposed to obtain the first frame image.

6. The method according to claim 4 or 5, characterized in that: The first layer includes second data, wherein obtaining the second frame image by displacement compensation of the first frame image includes: Combining the second data, the second layer and the third layer to obtain a fifth layer; decoding the fifth layer to obtain first coding information of the second data, the second coding information and / or the third coding information, wherein the first coding information of the second data is used to indicate enabling the displacement compensation; The displacement compensation is performed on the first frame image according to the second coding information and / or the third coding information to obtain the second frame image.

7. The method according to any one of claims 1 to 3, characterized in that The image cache includes a second layer and a third layer, the second layer is used to represent a rendering image for a left eye, and the third layer is used to represent a rendering image for a right eye; The method further comprises: Acquire configuration information of a third frame, where the configuration information of the third frame includes first configuration information, where the first configuration information is used to indicate that the displacement compensation is not enabled; Wherein, the third frame is before the first frame; The determining the first frame image according to the image cache comprises: The first frame image is determined according to the configuration information of the third frame and a sixth layer, where the sixth layer is obtained by merging the second layer and the third layer.

8. The method according to any one of claims 1, 2, 3 and 7, characterized in that The image cache includes a second layer and a third layer, the second layer is used to represent a rendering image for a left eye, and the third layer is used to represent a rendering image for a right eye; The method further comprises: Acquire configuration information of the first frame, where the configuration information of the first frame includes second configuration information, where the second configuration information is used to indicate enabling the displacement compensation; The configuration information of the first frame further includes third configuration information and / or fourth configuration information; The third configuration information is used to indicate the horizontal displacement, and the fourth configuration information is used to indicate the longitudinal displacement; The performing displacement compensation on the first frame image to obtain a second frame image includes: The second frame image is determined according to the configuration information of the first frame and a sixth layer, where the sixth layer is obtained by merging the second layer and the third layer.

9. The method according to claim 7, characterized in that: The obtaining of the configuration information of the third frame includes: Serially write the configuration information of the third frame; or The configuration information of the third frame is written in parallel.

10. The method according to claim 8, characterized in that The acquiring the configuration information of the first frame includes: Serially write the configuration information of the first frame; or The configuration information of the first frame is written in parallel.

11. The method according to any one of claims 1 to 10, characterized in that The method is applied to an augmented reality (AR) device or a virtual reality (VR) device.

12. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the method of any one of claims 1 to 11.

13. A display device, characterized in that: include: A processor coupled to a memory, the memory being used to store a computer program, the processor being used to run the computer program so that the apparatus performs the method according to any one of claims 1 to 11.

14. The device according to claim 13, characterized in that Also included is one or more of the memory and the transceiver, wherein the transceiver is used to receive signals and / or send signals.

15. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the computer is caused to implement the method according to any one of claims 1 to 11.

16. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

17. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 11.

Images