Display method and electronic device

Abstract

An embodiment of the present application provides a display method and an electronic device. The method is applied to an electronic device including a display screen and includes: if it is recognized that a first target part of the user starts to rotate, sending a first request to a server, where the first request carries the rotation direction of the first target part; receiving data of a to-be-displayed screen determined by the server according to the rotation direction of the first target part; and displaying the to-be-displayed screen to the user through the display screen according to the data of the to-be-displayed screen. In this method, once it is recognized that the first target part of the user starts to rotate, a first request is sent to the server, and data of the to-be-displayed screen determined by the server according to the rotation direction of the first target part carried in the first request is received, and then the to-be-displayed screen is displayed. This method can greatly reduce the latency of screen display, thereby improving the user's visual experience.

Description

Technical Field

[0001] This application relates to the field of electronic technologies, and in particular, to a display method and an electronic device. Background Art

[0002] Virtual reality (VR) technology is a three-dimensional environment of a virtual world generated by a computer simulation system. It is a system simulation of an interactive three-dimensional dynamic visual scene and entity behavior that integrates multi-source information, enabling users to immerse themselves in this environment to experience the virtual world therein, and it has been widely applied in scenarios such as gaming, movie viewing, and remote teaching.

[0003] In traditional technologies, after a user wears a VR device, the display screen of the VR device can be changed by rotating the head; among them, when the user's head rotates and then stops, the gyroscope and gravity sensor in the VR device can identify the rotation data of the user's head and send it to the server, and the server can determine the data of the new display screen according to the rotation data of the user's head, and then display it through the VR device.

[0004] However, in traditional technologies, the time delay of the VR device for displaying the screen is relatively long, which may cause the user to have a sense of dizziness and the experience is not high. Summary of the Invention

[0005] Embodiments of this application provide a display method and an electronic device, which can greatly reduce the time delay of screen display, thereby enhancing the visual experience of users.

[0006] In a first aspect, an embodiment of this application provides a display method applied to an electronic device including a display screen. The method includes: if it is recognized that a first target part of a user starts to rotate, sending a first request to a server, where the first request carries the rotation direction of the first target part; receiving the data of the to-be-displayed screen determined by the server according to the rotation direction of the first target part; and displaying the to-be-displayed screen to the user through the display screen according to the data of the to-be-displayed screen.

[0007] In the above implementation, once the VR device recognizes that the first target part of the user starts to rotate, it sends a first request to the server, and receives the data of the to-be-displayed screen determined by the server according to the rotation direction of the first target part carried in the first request, and then displays the to-be-displayed screen; this method can greatly reduce the time delay of screen display, thereby enhancing the visual experience of users.

[0008] In combination with the first aspect, in some implementation manners of the first aspect, the above method further includes: obtaining first data, where the first data includes the rotation direction of a second target part of the user; the step of displaying the to-be-displayed screen to the user through the display screen according to the data of the to-be-displayed screen includes: if the rotation direction of the first target part is the same as the rotation direction of the second target part, then display the to-be-displayed screen to the user through the display screen according to the data of the to-be-displayed screen.

[0009] In the above implementation manner, the VR device displays the to-be-displayed screen to the user through the display screen only when the rotation directions of the user's first target part and second target part are the same, which can reduce the phenomenon that the display screen shakes due to the frequent slight movement of the user's first target part, and further improve the user's visual experience.

[0010] In combination with the first aspect and the above implementation manner, an inertial measurement unit is configured in the electronic device, and the obtaining of the first data includes: judging whether the second target part has rotated and is currently in a static state according to the measurement data of the inertial measurement unit; if so, obtaining the first data according to the measurement data.

[0011] In the above implementation manner, by judging whether the second target part has rotated and is currently in a static state to determine that the user has made a normal rotation, and then displaying the to-be-displayed screen to the user through the display screen after determination, the user's visual experience is further improved.

[0012] In combination with the first aspect and the above implementation manner, the first request further carries the rotation angle of the first target part, and the first data further includes the rotation angle of the second target part; the step of displaying the to-be-displayed screen to the user through the display screen if the rotation direction of the first target part is the same as the rotation direction of the second target part includes: if the rotation direction of the first target part is the same as the rotation direction of the second target part, and the angle difference between the rotation angle of the first target part and the rotation angle of the second target part is less than or equal to a preset threshold, then display the to-be-displayed screen to the user through the display screen according to the data of the to-be-displayed screen.

[0013] In the above implementation manner, when the VR device recognizes that the user's eyes begin to rotate, it sends a first request to the server, and then stores the data of the received to-be-displayed screen in the cache. When the rotation direction of the user's eyes is the same as the rotation direction of the head, and the angle difference between the rotation angle of the eyes and the rotation angle of the head is less than or equal to a preset threshold, the to-be-displayed screen is displayed to the user through the display screen, which can further improve the user's visual experience.

[0014] Combined with the first aspect and the above implementation manner, the first target part includes the eyeball or the pupil, and the second target part includes the user's head.

[0015] In the above implementation manner, based on the physiological phenomenon that the rotation efficiency of the human eyeball is higher than that of the head, in the VR device scenario, the user mainly makes visual traction movements. When the user intends to turn the head, the eyeball movement will be faster than the head movement. Therefore, once the VR device recognizes that the user's eyeball starts to rotate, it sends a first request to the server, which can greatly reduce the latency of the screen display, thereby improving the user's visual experience.

[0016] Combined with the first aspect and the above implementation manner, before the step of, if it is recognized that the first target part of the user starts to rotate, sending a first request to the server, the method further includes: obtaining a first image and a second image, where the first image and the second image are two adjacent frames of images, and both the first image and the second image include the first target part; respectively identifying a first position of the first target part in the first image and a second position of the first target part in the second image; if the second position is different from the first position, determining that the first target part starts to rotate.

[0017] In the above implementation manner, the VR device determines whether the first target part starts to rotate according to the positions of the first target part in two adjacent frames of images, thereby improving the accuracy of the determination result. On this basis, sending a first request to the server can improve the accuracy of the data of the display screen to be determined by the server, further improving the user's visual experience.

[0018] Combined with the first aspect and the above implementation manner, the above method further includes: determining a rotation direction of the first target part according to the second position and the first position, and generating the first request according to the rotation direction of the first target part.

[0019] In the above implementation manner, the VR device determines the rotation direction of the first target part according to the positions of the first target part in two adjacent frames of images. On this basis, sending a first request to the server can improve the accuracy of the data of the display screen to be determined by the server.

[0020] Combined with the first aspect and the above implementation manner, the first request further carries a rotation displacement of the first target part. Receiving the data of the display screen to be determined by the server according to the rotation direction of the first target part includes: receiving the data of the display screen to be determined by the server according to the rotation direction and the rotation displacement of the first target part.

[0021] In the above implementation, the server jointly determines the data of the to-be-displayed screen according to the rotation direction and rotation displacement of the first target part, which can further improve the accuracy of the data of the to-be-displayed screen determined by the server.

[0022] Combined with the first aspect and the above implementation, both the first image and the second image are eye images of the user.

[0023] In the above implementation, by identifying the eye image of the user to determine the position of the first target part, thereby improving the accuracy of the result of determining whether the first target part starts to rotate. On this basis, a first request is sent to the server, which can improve the accuracy of the data of the to-be-displayed screen determined by the server, and further enhance the user's visual experience.

[0024] In a second aspect, an embodiment of the present application provides a device, which is included in an electronic device and has the function of implementing the behavior of the electronic device in the above first aspect and any possible implementation manners of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, which includes: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other to enable the electronic device to execute any one of the methods in the technical solutions described in the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a chip, which includes a processor. The processor is configured to read and execute a computer program stored in a memory to execute the methods in the first aspect and any possible implementation manners thereof.

[0027] Optionally, the chip further includes a memory, and the memory is connected to the processor through a circuit or a wire.

[0028] Further optionally, the chip further includes a communication interface.

[0029] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor is enabled to execute any one of the methods in the technical solutions described in the first aspect.

[0030] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes: computer program code. When the computer program code runs on an electronic device, the electronic device is enabled to execute any one of the methods in the technical solutions described in the first aspect. Description of the Drawings

[0031] Figure 1 It is an application scenario diagram of a display method provided by an embodiment of the present application;

[0032] Figure 2 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0033] Figure 3 It is a software structure block diagram of the electronic device provided by an embodiment of the present application;

[0034] Figure 4 It is a schematic flow diagram of a display method provided by an embodiment of the present application;

[0035] Figure 5 It is a schematic diagram of a determined to-be-displayed screen provided by an embodiment of the present application;

[0036] Figure 6 It is a schematic flow diagram of another display method provided by an embodiment of the present application;

[0037] Figure 7 Figure (a) in it is a schematic diagram of a first position in a first image provided by an embodiment of the present application;

[0038] Figure 7 Figure (b) in it is a schematic diagram of a second position in a second image provided by an embodiment of the present application;

[0039] Figure 8 It is a schematic flow diagram of yet another display method provided by an embodiment of the present application;

[0040] Figure 9 It is a schematic flow diagram of yet another display method provided by an embodiment of the present application;

[0041] Figure 10 Figure (a) in it is a schematic diagram of a first image provided by an embodiment of the present application;

[0042] Figure 10 Figure (b) in it is a schematic diagram of a second image provided by an embodiment of the present application;

[0043] Figure 11 It is a schematic diagram of another determined to-be-displayed screen provided by an embodiment of the present application. Detailed implementation manners

[0044] Next, the technical solutions in the embodiments of the present application will be described in conjunction with the accompanying drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may represent A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. These three situations. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

[0045] Hereinafter, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first", "second", and "third" may explicitly or implicitly include one or more of such features.

[0046] The display method provided by the embodiments of the present application can be applied to a scenario as Figure 1 shown. After the user wears the VR device 1, the user can view the virtual picture through the display screen (such as the lens) on the VR device 1 and change the displayed picture by turning the head. However, in the traditional technology, usually when the user's head rotates and then stops, the VR device 1 sends the rotation data of the user's head (such as the rotation direction, rotation angle, etc.) to the server 2, and the server 2 determines the new display picture data according to the rotation data and displays it through the VR device 1. As can be seen from the above, in the traditional technology, a new display picture data is requested from the server 2 only after the user's head rotates and stops, which largely results in a relatively long delay in the VR device 1 for displaying the picture, may cause the user to have a sense of dizziness, and the user experience is relatively poor.

[0047] For example, in a 5G scenario, a large amount of data will be sent to the server 2 for calculation, such as the real-time rendering of VR content. To calculate the end-to-end delay, the delay in the case of unstable network must be considered. According to the current model estimation, in a normal network situation, this delay will reach about 100 milliseconds (ms), which is far greater than the dizziness lower limit of 20 ms, and will bring a very bad experience to the user.

[0048] Therefore, the embodiments of the present application provide a display method, which can enable the VR device 1 to request new display picture data from the server 2 before the user's head stops. When the user's head stops, the new display picture can be directly displayed, greatly reducing the delay of the picture display, thereby enhancing the user's visual experience.

[0049] The display method provided by the embodiments of this application can be applied to electronic devices that can install a display screen, such as augmented reality (AR) / virtual reality (VR) devices, mobile phones, tablet computers, wearable devices, in-vehicle devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of this application do not impose any restrictions on the specific types of electronic devices.

[0050] Exemplarily, Figure 2 FIG. 5 is a schematic structural diagram of an electronic device 100 provided by an embodiment of this application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0051] It can be understood that the structure schematically shown in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0053] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0054] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0055] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0056] The I2C interface is a two-way synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface to implement the touch function of the electronic device 100.

[0057] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple groups of I2S buses. The processor 110 may be coupled to the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the I2S interface to implement the function of answering a call through a Bluetooth headset.

[0058] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through the PCM bus interface. In some embodiments, the audio module 170 may also transmit an audio signal to the wireless communication module 160 through the PCM interface to implement the function of answering a call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0059] The UART interface is a universal serial data bus for asynchronous communication. This bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0060] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0061] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0062] The USB interface 130 is an interface that conforms to the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0063] It can be understood that the interface connection relationships between the various modules illustrated in the embodiments of the present application are only illustrative descriptions and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0064] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input from a wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0065] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery charge cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0066] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0067] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 2 The structures of the antenna 1 and the antenna 2 in [description] are only one example. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: The antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0068] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0069] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0070] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 may also receive the signal to be transmitted from the processor 110, perform frequency modulation on it, amplify it, and convert it into electromagnetic waves through the antenna 2 and radiate it out.

[0071] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0072] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0073] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0074] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0075] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0076] The camera 193 is used to capture static images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0077] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0078] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0079] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as learning from the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0080] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0081] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0082] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0083] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0084] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A.

[0085] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by bringing the receiver 170B close to the human ear.

[0086] The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak by bringing the mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0087] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0088] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0089] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and enables the lens to offset the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenarios.

[0090] The barometric pressure sensor 180C is used to measure barometric pressure. In some embodiments, the electronic device 100 calculates the altitude according to the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0091] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip, features such as automatic flip unlocking are set.

[0092] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0093] A distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0094] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear for a call, so as to automatically turn off the screen to achieve the purpose of power saving. The proximity light sensor 180G can also be used for the holster mode, and the pocket mode automatically unlocks and locks the screen.

[0095] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touch.

[0096] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access the application lock, fingerprint taking pictures, fingerprint answering calls, etc.

[0097] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds the threshold, the electronic device 100 reduces the performance of the processor near the temperature sensor 180J to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by low temperature. In some other embodiments, when the temperature is lower than yet another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0098] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as the "touch screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from that of the display screen 194.

[0099] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals of the vibrating bone mass of the human vocal part. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulsation signals. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bone mass of the human vocal part acquired by the bone conduction sensor 180M to implement the voice function. The application processor can parse out heart rate information based on the blood pressure pulsation signals acquired by the bone conduction sensor 180M to implement the heart rate detection function.

[0100] The button 190 includes a power-on button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs to generate key signal inputs related to the user settings and function control of the electronic device 100.

[0101] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playing, etc.) can correspond to different vibration feedback effects. For touch operations acting on different regions of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0102] The indicator 192 can be an indicator light, which can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0103] The SIM card interface 195 is used to connect to a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0104] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0105] Figure 3 It is the software structure block diagram of the electronic device 100 in the embodiments of this application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer can include a series of application packages.

[0106] As Figure 3 shown, the application packages can include applications such as cameras, galleries, calendars, calls, maps, navigation, WLAN, Bluetooth, music, videos, and text messages.

[0107] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0108] As Figure 3 shown, the application framework layer can include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, a Bluetooth proxy module, etc.

[0109] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0110] The content provider is used to store and obtain data, and make this data accessible to application programs. The data may include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0111] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

[0112] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call states (including answering, hanging up, etc.).

[0113] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0114] The notification manager enables application programs to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application program, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompt text information in the status bar, emit a prompt tone, the electronic device vibrates, the indicator light flashes, etc.

[0115] Android runtime includes a core library and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0116] The core library consists of two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android.

[0117] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0118] The system library may include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), etc.

[0119] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0120] The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0121] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0122] The 2D graphics engine is a drawing engine for 2D drawing.

[0123] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0124] To facilitate the understanding of the embodiments of the present application, first, a brief introduction to the process of the VR device outputting a display screen is provided.

[0125] When the VR device outputs a display screen, the resource information corresponding to the display screen is first obtained by the GPU inside the VR device. The GPU constructs and renders the resource information to form an image digital signal, and then inputs the image digital signal into the display driver IC (DDIC) of the VR device. The DDIC converts the image digital signal into an analog signal and controls the display device inside the display screen through the analog signal, so that the VR device outputs a display screen. For example, the DDIC outputs a control voltage or current to the anode and cathode of the OLED light-emitting layer, and the anode and cathode receive the control voltage or current to complete the driving of each pixel, so that the VR device outputs a corresponding display screen.

[0126] In the following embodiments of the present application, a VR device with the Figure 2 and Figure 3 shown structure will be taken as an example, and in combination with the accompanying drawings and application scenarios, the display method provided by the embodiments of the present application will be specifically described.

[0127] Figure 4 is a schematic flowchart of an example display method provided by the embodiments of the present application. The method includes:

[0128] S101. If it is recognized that the first target part of the user starts to rotate, send a first request to the server, where the first request carries the rotation direction of the first target part.

[0129] Among them, since the rotation of the human eyeball and the rotation of the head are controlled by different tissues, the rotation efficiency of the eyeball is higher than that of the head, and the rotation response speed is faster. Therefore, in the VR device scenario, the user mainly uses visual traction actions. When the user intends to rotate the head, the eyeball movement will be faster than the head movement. When the eyeball rotates, it can be understood that the eyeball is in a saccade state. Saccade means that the target of the eye has changed, or indicates that the information in the field of view is inconsistent with the user's expectation; when the eyeball is stationary, it can be understood that the eyeball is in a fixation state. The fixation point can show that the eye is looking at a certain target (with cognitive processing), which means that the eye is locked on the target. Fixation and saccade often occur alternately.

[0130] Based on the above physiological characteristics of the human body, in this embodiment, when the VR device recognizes that the user's eyeball (i.e., the first target part) starts to rotate, that is, when the eyeball changes from the fixation state to the saccade state, a first request is sent to the server. The first request carries the rotation direction of the eyeball and is used to request data of a new displayable picture from the server.

[0131] As an example rather than a limitation, a camera device, such as a camera, can be set at a position in the VR device facing the user's eyes to capture the user's eye image. The working frequency of the camera device can be 60 hertz (Hz) - 120 Hz, and the time delay per frame is 15 ms - 8 ms; then, the captured eye image is recognized to identify the position of the eyeball in the eye image. If the positions of the eyeball in multiple consecutive eye images are different, it can indicate that the eyeball starts to rotate continuously. Optionally, in this embodiment, eye images with different sampling frame numbers can be selected to determine whether the eyeball starts to rotate.

[0132] In a possible implementation manner, the above first target part can be the user's pupil. The judgment process for the rotation of the user's pupil is similar to the judgment process for the rotation of the user's eyeball and will not be elaborated here.

[0133] In another possible implementation, the above-mentioned first target part can also be the user's head. An inertial measurement unit (IMU) is usually set in a VR device. An IMU is a device that measures the target's three-axis attitude angle (or angular rate) and acceleration. Usually, an IMU includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometers measure the acceleration signals of the target on three axes in the coordinate system, and the gyroscopes measure the magnitude of the angular velocity of the target's movement in the coordinate system. Finally, the attitude of the target is calculated based on the obtained measurement data. Then, the change in the user's head attitude can be identified through the measurement data of the IMU. Once it is identified that the user's head starts to rotate, a first request is sent to the server.

[0134] S102. The server determines the data of the to-be-displayed screen according to the rotation direction of the first target part.

[0135] S103. The server sends the data of the to-be-displayed screen to the VR device.

[0136] Specifically, after receiving the first request sent by the VR device, the server can determine the data of the to-be-displayed screen according to the rotation direction of the eyeball. Since the screen presented by the VR device is a three-dimensional screen, the entire corresponding scene is equivalent to a sphere, and the data of the entire scene is stored in the server (here it can be assumed that the entire scene screen is located in an xyz coordinate system), and the screen presented by the VR device each time corresponds to a field of view (FOV) screen in the scene; optionally, when the server receives the first request, it can select the adjacent FOV screen of the current FOV screen as the to-be-displayed screen according to the rotation direction of the eyeball, and the direction of the adjacent perspective screen relative to the current perspective screen is the same as the rotation direction of the eyeball.

[0137] Exemplarily, as Figure 5 shown, assuming that the rotation direction of the eyeball is to rotate to the right, and the current FOV screen presented by the VR device is A, then the server selects the perspective screen W adjacent to the right of A from the entire scene screen as the to-be-displayed screen and sends the data of the to-be-displayed screen to the VR device; it should be noted that the FOV of the screen that the VR device can present is pre-set, such as a screen with a FOV of 60° or 90°.

[0138] S104. According to the data of the to-be-displayed screen, the to-be-displayed screen is displayed to the user through the display screen.

[0139] Specifically, when the VR device receives the data of the to-be-displayed screen, it can perform construction, rendering, signal conversion, etc. according to the above process of outputting the display screen, and display the to-be-displayed screen to the user through the display screen.

[0140] In a possible implementation, when the VR device receives the data of the picture to be displayed, it can immediately display the picture to be displayed to the user through the display screen.

[0141] In another possible implementation, when the VR device recognizes that the user's eyes no longer move, such as when it recognizes that the position of the eyes in multiple frames of eye images no longer changes, it then displays the picture to be displayed to the user through the display screen.

[0142] In the above display method, since the user's eye movement is faster than head movement, once the VR device recognizes that the user's eyes start to move, it sends a first request to the server and receives the data of the picture to be displayed determined by the server according to the eye movement direction carried in the first request, and then displays the picture to be displayed; this method can greatly reduce the latency of picture display, thus enhancing the user's visual experience.

[0143] In one embodiment, before the above S101, if it recognizes that the first target part of the user starts to move and sends a first request to the server, the VR device also needs to determine whether the first target part starts to move, such as Figure 6 As shown, the specific process may include:

[0144] S201, obtain a first image and a second image, where the first image and the second image are two adjacent frames of images, and both the first image and the second image include the first target part.

[0145] S202, respectively identify a first position of the first target part in the first image and a second position of the first target part in the second image.

[0146] Specifically, the imaging device installed inside the VR device can continuously collect the user's eye images at a fixed frame rate, and each eye image includes the eyes. After the VR device obtains two adjacent frames of eye images (including the first image and the second image), it can respectively identify the positions of the eyes in the two frames of eye images. As an example but not a limitation, the VR device can use a target detection algorithm based on a neural network to identify the eye images, such as the YOLO algorithm, the Fast-RCNN algorithm, etc., to obtain the first position of the eyes in the first image and the second position of the eyes in the second image. Optionally, the first position can be represented by the coordinates of the center of the eyes (such as the pupil) in the first image, and the second position can be represented by the coordinates of the center of the eyes in the second image. It should be noted that the first image and the second image need to be in the same reference coordinate system.

[0147] In a possible implementation, after the imaging device captures multiple frames of eye images, the VR device can identify the eye images of the interval frames or the eye images of multiple intervals to determine the position of the eyeball in each eye image.

[0148] In another possible implementation, the above-mentioned first image and second image can also be the user's face image, as long as the eyeball part can be included. This embodiment does not make specific restrictions on the first image and the second image.

[0149] S203. If the second position is different from the first position, it is determined that the first target part starts to rotate.

[0150] Among them, if the obtained second position is different from the first position, it indicates that the eyeball starts to rotate; however, in actual applications, the human eyeball may have involuntary micro-movements. Therefore, this embodiment can also set a threshold. When the coordinate difference between the second position and the first position is greater than this threshold, that is, when the eyeball has a slightly large position change, it is determined that the eyeball starts to rotate.

[0151] Then, the VR device can also determine the rotation direction of the eyeball according to the second position and the first position. Assuming that the first image and the second image are in the same XY coordinate system, if the x coordinate of the second position is greater than the x coordinate of the first position, it means that the eyeball rotates to the right. If the x coordinate of the second position is less than the x coordinate of the first position, it means that the eyeball rotates to the left; furthermore, the VR device can generate a first request according to the determined rotation direction and send it to the server.

[0152] In another possible implementation, as Figure 7 shown in figure (a) therein, the VR device can also use the distance between the center of the eyeball (such as the pupil) in the first image and a preset point as the first position; as Figure 7 shown in figure (b) therein, use the distance between the center of the eyeball in the second image and the preset point as the second position. The preset point can be the corner of the eye position point (such as Figure 7 point S therein). If the second position is different from the first position, it can indicate that the eyeball starts to rotate; in addition, if the second position is greater than the first position, that is, the pupil is farther from point S in the second image than in the first image, it means that the eyeball rotates to the left. If the second position is less than the first position, that is, the pupil is closer to point S in the second image than in the first image, it means that the eyeball rotates to the right.

[0153] In the above display method, the VR device determines whether the first target part starts to rotate according to the position of the first target part in two adjacent frames of images, thereby improving the accuracy of the determination result. On this basis, a first request is sent to the server, which can improve the accuracy of the data of the display screen to be determined by the server, and further enhance the user's visual experience.

[0154] In one embodiment, to avoid the situation where the user may have unconscious eye rotation but actually does not want to change the display screen, the rotation data of the user's head can be further obtained, and the eye rotation data is compared with the head rotation data before determining whether to display the display screen to be determined. Figure 8 It is a schematic flowchart of another display method provided by an embodiment of the present application. The method includes:

[0155] S301, if it is recognized that the first target part of the user starts to rotate, send a first request to the server, and the first request carries the rotation direction of the first target part.

[0156] S302, receive the data of the display screen to be determined by the server according to the rotation direction of the first target part.

[0157] Among them, the implementation processes of steps S301 - S302 are similar to those of steps S101 - S102, and will not be elaborated here.

[0158] S303, obtain first data, where the first data includes the rotation direction of the second target part of the user.

[0159] Among them, because the user's eye movement is faster than head movement, after the VR device requests the data of the display screen to be determined from the server according to the rotation direction of the user's eyeball, the data of the display screen to be determined can be first stored in the local cache; then it is determined that the user really wants to change the display screen of the VR device currently, rather than physiological eye movement, such as unconscious micro - movement of the eyeball, etc. The VR device can further judge according to the rotation data of the user's head (i.e., the second target part) to avoid phenomena such as trembling of the display screen of the VR device caused by frequent micro - movement of the eyeball. Optionally, the rotation direction of the user's head can be obtained through the measurement data of the IMU configured in the VR device.

[0160] In a possible implementation manner, the process for the VR device to obtain the first data can be: the VR device first determines whether the user's head has rotated and is currently in a stationary state. For example, when the head is stationary, the acceleration value of the z - axis of the accelerometer is the acceleration due to gravity, and it will change when rotating. Then, the rotation of the head and its current stationary state can be judged through the z - axis acceleration value; if so, the rotation direction of the user's head is determined according to the measurement data.

[0161] S304. If the rotation direction of the first target part is the same as that of the second target part, the to-be-displayed screen is displayed to the user through the display screen according to the data of the to-be-displayed screen.

[0162] Specifically, if the VR device determines that the rotation direction of the user's eyeballs is the same as that of the head, the to-be-displayed screen is displayed through the display screen according to the above process of outputting the display screen; if the rotation direction of the user's eyeballs is different from that of the head, the displayed screen remains unchanged.

[0163] In the above display method, the VR device displays the to-be-displayed screen to the user through the display screen only when the rotation directions of the user's eyeballs and the head are the same, which can reduce the phenomenon of the display screen shaking caused by the frequent micro-movements of the user's eyeballs and further improve the user's visual experience.

[0164] In one embodiment, to further improve the user's visual experience, the VR device can further determine the conditions for displaying the to-be-displayed screen. Figure 9 FIG. is a schematic flowchart of another display method provided by an embodiment of the present application. The method includes:

[0165] S401. If it is recognized that the first target part of the user starts to rotate, a first request is sent to the server, and the first request carries the rotation direction and rotation angle of the first target part.

[0166] In this embodiment, in addition to carrying the above rotation direction of the user's eyeballs, the first request can also carry the rotation angle of the user's eyeballs. Among them, the rotation angle of the eyeballs can be determined through the following process: Generally, the human eyeball is approximately spherical, with an anteroposterior diameter of 24 millimeters (mm), a vertical diameter of 23.5 mm, and an equatorial circumference of 75 mm. In this embodiment, it is assumed that the eyeball is a 360° sphere with a diameter of 24 mm and a circumference of 75 mm. First, the position of the eyeball in adjacent eye images can be determined through the method of the above embodiment. Taking the pupil coordinates of the eyeball position as an example, the rotational displacement of the eyeball can be determined according to the pupil coordinates; for example, Figure 10 FIG. (a) therein is the first image, Figure 10 FIG. (b) therein is the second image. The difference in the horizontal coordinates of the pupils in the two images is denoted as the rotational displacement d, and then the rotational angle of the eyeball can be calculated through the relationship formula.

[0167] S402. Receive the data of the to-be-displayed screen determined by the server according to the rotation direction of the first target part.

[0168] In a possible implementation manner, the implementation process of this step is similar to that of S102 above and will not be elaborated here.

[0169] In another possible implementation, the server can also jointly determine the data of the to-be-displayed image based on the rotation direction and rotation angle of the first target part. Suppose the viewing angle of the images that the currently set VR device can present is 30°, and the rotation angle of the eyeball to the right is 45°. Then the server can select the viewing angle image that is separated from the current viewing angle image by half of the viewing angle as the to-be-displayed image, and send the data of the to-be-displayed image to the VR device.

[0170] In yet another possible implementation, the above first request may carry the rotation direction and rotation displacement of the first target part, and the server can jointly determine the data of the to-be-displayed image based on the rotation direction and rotation displacement of the first target part. Optionally, the server can set a correspondence between the rotation displacement of an eyeball and the displacement of the image movement. For example, when the rotation displacement of the eyeball is 1 mm, the image needs to move 1 centimeter (cm); then it can calculate the displacement that the displayed image needs to move according to the received rotation displacement of the eyeball, and use the image data within the current viewing angle after the movement as the data of the to-be-displayed image. Exemplarily, as Figure 11 shown, the image within the current viewing angle is Q. Suppose the server receives that the eyeball moves 10 mm to the right, then the image needs to move 10 cm to the right, and the image within the current viewing angle after the movement is R. The server can send the image data corresponding to R to the VR device as the data of the to-be-displayed image.

[0171] S403. Obtain first data, where the first data includes the rotation direction and rotation angle of the second target part of the user.

[0172] S404. If the rotation direction of the first target part is the same as the rotation direction of the second target part, and the angle difference between the rotation angle of the first target part and the rotation angle of the second target part is less than or equal to a preset threshold, then display the to-be-displayed image to the user through the display screen according to the data of the to-be-displayed image.

[0173] Among them, the rotation angle of the second target part (head) of the user can be output by the above IMU. Then the VR device determines whether the rotation direction of the eyeball is the same as the rotation direction of the head, and the angle difference between the rotation angle of the eyeball and the rotation angle of the head is less than or equal to a preset threshold (such as 5°). If both of these conditions are met at the same time, the to-be-displayed image is displayed to the user through the display screen.

[0174] For the method in this embodiment, after the VR device recognizes the eye rotation and sends a first request to the server, it usually takes about 100 ms until the IMU recognizes the user's head rotation and then stops. Therefore, by using this time difference to request the frame data from the server, the frame to be displayed can be immediately displayed when the display condition is met, greatly reducing the latency of frame display.

[0175] For the above display method, when the VR device recognizes that the user's eyes start to rotate, it sends a first request to the server, and then stores the data of the frame to be displayed received into the buffer. When the rotation direction of the user's eyes is the same as that of the head, and the angular difference between the rotation angle of the eyes and the rotation angle of the head is less than or equal to a preset threshold, the frame to be displayed is displayed to the user through the display screen, which can further improve the user's visual experience.

[0176] The above text details the examples of the display method provided by the embodiments of this application. It can be understood that in order for the electronic device to implement the above functions, it includes the corresponding hardware and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed herein, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of this application.

[0177] The embodiments of this application can divide the electronic device into functional modules according to the above method examples. For example, each function can be corresponding to each functional module, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of this application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0178] It should be noted that all the relevant content of each step involved in the above method embodiments can be cited in the function description of the corresponding functional module, and will not be repeated here.

[0179] The electronic device provided in this embodiment is used to execute the above display method, so it can achieve the same effect as the above implementation method.

[0180] In the case of adopting an integrated unit, the electronic device may further include a processing module, a storage module, and a communication module. Among them, the processing module may be used to control and manage the operations of the electronic device. The storage module may be used to support the electronic device to execute stored program codes and data, etc. The communication module may be used to support the communication between the electronic device and other devices.

[0181] Among them, the processing module may be a processor or a controller. It may implement or execute various exemplary logic blocks, modules, and circuits described in connection with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device for interacting with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0182] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a device having Figure 2 the structure shown.

[0183] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor is caused to execute the display method described in any one of the above embodiments.

[0184] An embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, the computer is caused to execute the above-related steps to implement the display method in the above embodiments.

[0185] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component, or a module. The device may include a processor and a memory connected thereto; among them, the memory is used to store computer execution instructions. When the device runs, the processor may execute the computer execution instructions stored in the memory so that the chip executes the display method in each of the above method embodiments.

[0186] Among them, the electronic device, the computer-readable storage medium, the computer program product, or the chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved by them may refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0187] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0188] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units, that is, they can be located in one place or distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods in the various embodiments of the present application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks or optical disks and other various media that can store program codes.

[0192] The above content is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A display method, characterized in that, Applied to an electronic device including a display screen, the method includes: If it is recognized that the first target part of the user starts to rotate, send a first request to the server, where the first request carries the rotation direction of the first target part; the first request also carries the rotation angle of the first target part; Receive the data of the to-be-displayed screen determined by the server according to the rotation direction of the first target part; Obtain first data, where the first data includes the rotation direction of the second target part of the user; the first data also includes the rotation angle of the second target part; According to the data of the to-be-displayed screen, display the to-be-displayed screen to the user through the display screen, including: If the rotation direction of the first target part is the same as the rotation direction of the second target part, and the angle difference between the rotation angle of the first target part and the rotation angle of the second target part is less than or equal to a preset threshold, then display the to-be-displayed screen to the user through the display screen according to the data of the to-be-displayed screen; if the rotation direction of the first target part is different from the rotation direction of the second target part, do not change the screen displayed on the display screen.

2. The method according to claim 1, characterized in that, An inertial measurement unit is configured in the electronic device, and the obtaining of the first data includes: According to the measurement data of the inertial measurement unit, judge whether the second target part has rotated and is currently in a stationary state; If so, obtain the first data according to the measurement data.

3. The method according to claim 1, wherein The first target part includes the eyeball or pupil, and the second target part includes the user's head.

4. The method according to claim 1, characterized in that Before the step of if it is recognized that the first target part of the user starts to rotate and send a first request to the server, the method further includes: Obtain a first image and a second image, where the first image and the second image are two adjacent frames of images, and both the first image and the second image include the first target part; Identify the first position of the first target part in the first image and the second position of the first target part in the second image respectively; If the second position is different from the first position, determine that the first target part starts to rotate.

5. The method according to claim 4, characterized in that The method further includes: Determine the rotation direction of the first target part according to the second position and the first position, and generate the first request according to the rotation direction of the first target part.

6. The method according to claim 5, wherein The first request also carries the rotation displacement of the first target part, and the receiving of the data of the to-be-displayed screen determined by the server according to the rotation direction of the first target part includes: Receive the data of the to-be-displayed screen determined by the server according to the rotation direction and rotation displacement of the first target part.

7. The method according to claim 4, wherein Both the first image and the second image are eye images of the user.

8. An electronic device, characterized in that, Includes: A display screen; One or more processors; One or more memories; A module installed with multiple application programs; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device is caused to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images