Electronic equipment, display method thereof, processing equipment and computer readable storage medium

Abstract

The invention discloses electronic equipment and a display method thereof, processing equipment and a computer readable storage medium, the electronic equipment comprises an image acquisition device, and the method comprises the following steps: detecting whether the electronic equipment starts a perspective function or not; when the electronic equipment starts the perspective function, the collection frame rate of the image collection device is controlled to be a second frequency, the second frequency is an integral multiple of the first frequency, and the first frequency is the light emitting frequency of ambient light.

Description

Technical Field

[0001] The present disclosure relates to intelligent display technology, and particularly to an electronic device, a display method thereof, a processing device, and a computer-readable storage medium. Background Art

[0002] Currently, the virtual reality (VR) / mixed reality (MR) industry is developing rapidly. Among them, VR applications are computer-generated 3D environments that allow users to be completely immersed in the virtual world presented by the head-mounted device without seeing the real environment. MR applications mix the real world and the virtual world to create a new visual and interactive environment that contains both physical entities and virtual information. The characters and objects in the real world and the virtual world can cross the real boundary, creating a more complex and exciting experience.

[0003] In some VR / MR products, the camera for see-through and the camera for gesture tracking are multiplexed. When the condition for starting see-through is met, the camera acquires external images. Due to the interference between the emission frequency of ambient light and the frame rate of camera acquisition, especially when the indoor light-emitting diode (LED) lights are on, the emission frequency of indoor LED lights will seriously interfere with the frame rate of camera acquisition, causing an obvious flicker phenomenon, which will have an uncomfortable impact on users and affect the user experience. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail in this document. This overview is not intended to limit the scope of protection of the claims.

[0005] The present disclosure provides a display method for an electronic device. The electronic device includes an image acquisition device. The display method includes: detecting whether the electronic device enables a see-through function; when the electronic device enables the see-through function, controlling the acquisition frame rate of the image acquisition device to be a second frequency, where the second frequency is an integer multiple of the first frequency, and the first frequency is the emission frequency of ambient light.

[0006] An embodiment of the present disclosure also provides a processing device, including: a processor and a memory storing a computer program that can run on the processor. When the processor executes the program, the steps of the display method described above are implemented.

[0007] An embodiment of the present disclosure also provides an electronic device, including: an image acquisition device and the processing device according to any embodiment of the present disclosure.

[0008] Embodiments of the present disclosure also provide a computer-readable storage medium storing executable instructions that, when executed by a processor, can implement the display method described in any of the above.

[0009] For the electronic device, its display method, processing device, and computer-readable storage medium according to embodiments of the present disclosure, when the perspective function of the electronic device is turned on, the acquisition frame rate of the image acquisition device is controlled to be a second frequency, where the second frequency is an integer multiple of the first frequency, and the first frequency is the emission frequency of the ambient light. This solves the flicker problem caused by the interference between the emission frequency of the ambient light and the camera acquisition frame rate, and improves the user experience.

[0010] Other features and advantages of the present disclosure will be described in the following specification, and in part, will be obvious from the specification, or will be understood by implementing the present disclosure. Other advantages of the present disclosure can be realized and obtained through the solutions described in the specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings are used to provide an understanding of the technical solutions of the present disclosure, and constitute a part of the specification. They are used together with the embodiments of the present disclosure to explain the technical solutions of the present disclosure, and do not constitute a limitation to the technical solutions of the present disclosure.

[0012] Figure 1 It is a schematic flowchart of a display method according to an exemplary embodiment of the present disclosure.

[0013] Figure 2 It is a schematic diagram of a usage method of a head-mounted display according to an exemplary embodiment of the present disclosure.

[0014] Figures 3 to 8 It is a schematic flowchart of several display methods according to an exemplary embodiment of the present disclosure.

[0015] Figure 9 It is a schematic diagram of an interface for manually setting the acquisition frame rate of an image acquisition device according to an exemplary embodiment of the present disclosure.

[0016] Figure 10 It is a schematic diagram of the structure of a processing device according to an exemplary embodiment of the present disclosure.

[0017] Figure 11 It is a schematic diagram of the structure of a head-mounted display according to an exemplary embodiment of the present disclosure.

[0018] Figure 12 It is a schematic diagram of the image processing process when an electronic device only displays a virtual view in a non-perspective mode according to an exemplary embodiment of the present disclosure.

[0019] Figure 13Schematic diagram of the image processing process when an electronic device only displays an environmental image in the perspective mode according to an exemplary embodiment of the present disclosure.

[0020] Figure 14 Schematic diagram of the image processing process when an electronic device displays an environmental image and a virtual view in the perspective mode according to an exemplary embodiment of the present disclosure. Detailed implementation manners

[0021] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it will be obvious to those of ordinary skill in the art that there can be more embodiments and implementation solutions within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the detailed implementation manners, many other combination ways of the disclosed features are also possible. Unless specifically restricted, any feature or element of any embodiment can be combined with any other feature or element in any other embodiment, or can replace any other feature or element in any other embodiment.

[0022] The present disclosure includes and contemplates combinations with features and elements known to those of ordinary skill in the art. The embodiments, features, and elements already disclosed in the present disclosure can also be combined with any conventional features or elements to form a unique inventive solution defined by the claims. Any feature or element of any embodiment can also be combined with features or elements from other inventive solutions to form another unique inventive solution defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present disclosure can be implemented alone or in any suitable combination. Therefore, the embodiments are not subject to other restrictions except those made according to the appended claims and their equivalent replacements. In addition, various modifications and changes can be made within the protection scope of the appended claims.

[0023] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not depend on the specific sequence of the steps described herein, the method or process should not be limited to the specific sequence of steps described. As will be understood by those of ordinary skill in the art, other step sequences are also possible. Therefore, the specific sequence of steps set forth in the specification should not be construed as a limitation on the claims. In addition, the claims directed to the method and / or process should not be limited to performing their steps in the order written, and those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the embodiments of the present disclosure.

[0024] As Figure 1As shown, an embodiment of the present disclosure provides a display method for an electronic device. The electronic device includes an image acquisition device. The method includes:

[0025] Step 101, detecting whether the electronic device enables the perspective function;

[0026] Step 102, when the electronic device enables the perspective function, controlling the acquisition frame rate of the image acquisition device to be a second frequency f2, where the second frequency f2 is an integer multiple of the first frequency f1, and the first frequency f1 is the emission frequency of the ambient light.

[0027] In the embodiment of the present disclosure, the electronic device may be a head-mounted display or other electronic devices, and the embodiment of the present disclosure does not limit this.

[0028] As Figure 2 shown, in the embodiment of the present disclosure, a head-mounted display (HMD), that is, a head-mounted display, is worn on the user's head and displays an image in front of the user's eyes. The head-mounted display generally forms the shape of goggles or the frame of large glasses. The head-mounted display usually includes a display device and a headband. The display device has a high-resolution liquid crystal display screen or an organic light-emitting diode (OLED) display screen, and is usually called glasses because it is made in the shape of eyes, and is used to display the same (or different) video images to the left and right eyes; the headband is used to connect the display device and wear the display device on the user's head and face.

[0029] In the embodiment of the present disclosure, the playback mode of the head-mounted display may be a non-perspective mode (that is, the perspective function is turned off). At this time, the head-mounted display presents a 3D or 2D virtual view; the playback mode of the head-mounted display may also be a perspective mode (that is, the perspective function is turned on). At this time, the head-mounted display presents an environmental image, or presents a fusion image of the virtual view and the environmental image (specifically presenting the environmental image or presenting the fusion image of the virtual view and the environmental image can be set according to the user's needs).

[0030] In the embodiment of the present disclosure, the head-mounted display may include a set of optical lenses, and the display screen image in front of the eyes is collimated, magnified, and pulled away through the optical lenses.

[0031] In some exemplary embodiments, the parallax images may be provided to the left and right eyes in sequence according to the time series. Since the images obtained by our left and right eyes are different, a sense of space is generated in the brain. By the head-mounted display, the left and right eyes observe images with slight differences, so as to observe a three-dimensional virtual view. At this time, assuming that the refresh rate of the head-mounted display is 120 Hz, then when playing frames, the playback frame rates of the left and right eyes are each 60 Hz.

[0032] In some exemplary embodiments, the head-mounted display may include a head movement tracking sensor ( Figure 2 , not shown in

[0033] ), and X, Y, Z-axis and front and back side tracking may be achieved through the head movement tracking sensor. In some exemplary embodiments, the head movement tracking sensor may include a gyroscope and an accelerometer. The gyroscope is used to measure the rotation angle of the head, and the accelerometer is used to measure the acceleration of the head. The gyroscope and the accelerometer may be integrated into a six-axis inertial sensor. In some other exemplary embodiments, the head movement tracking sensor may further include a magnetometer, which can be used to measure the direction of the head relative to the earth's magnetic field. The gyroscope, the accelerometer, and the magnetometer may be integrated into a nine-axis sensor.

[0034] In the embodiments of the present disclosure, the head-mounted display includes an image acquisition device ( Figure 2 , not shown in

[0035] ). The image acquisition device can be multiplexed in the perspective mode and the non-perspective mode. In the perspective mode, the image acquisition device can acquire ambient images and transmit them to the processing device. The processing device processes the acquired images, converts them into images that can be perceived by humans, and presents them on the screen. In the non-perspective mode (and / or the perspective mode), the image acquisition device can acquire hand images and transmit them to the processing device. The processing device detects the operations performed by the user in the three-dimensional environment through a gesture recognition algorithm, including but not limited to playing video games, navigating menus, controlling media playback, etc.

[0036] The display method of the electronic device according to the embodiments of the present disclosure controls the acquisition frame rate of the image acquisition device to be the second frequency f2 when the electronic device such as a head-mounted display turns on the perspective function. Here, the second frequency f2 is an integer multiple of the first frequency f1, so that when all pixel points receive exposure, the light energy received is the same, solving the flicker problem caused by the mismatch between the emission frequency of the ambient light and the camera acquisition frame rate, and improving the user experience.

[0037] In the embodiments of the present disclosure, the acquisition frame rate of the image acquisition device refers to the number of frames acquired by the image acquisition device per unit time. Exemplarily, assuming the acquisition frame rate is 50 Hz, the number of image frames acquired by the image acquisition device per second is 50 frames. And the exposure time of the image acquisition device refers to the time required for the photosensitive element of the image acquisition device to sense light during the shooting process. The higher the acquisition frame rate of the image acquisition device, the shorter its exposure time; the lower the acquisition frame rate of the image acquisition device, the longer its exposure time. The length of the exposure time has a certain impact on the imaging quality. For example, if the exposure time is set too long or too short, the acquired image may become blurred (if the exposure time is set too long, the instability of the shooting subject will cause the acquired image to become blurred; if the exposure time is set too short, insufficient received light energy will also cause the acquired image to become blurred). Therefore, the acquisition frame rate of the image acquisition device cannot be set too high or too low.

[0038] Exemplarily, the second frequency f2 = N * the first frequency f1, where N is 1 or 2.

[0039] In the embodiments of the present disclosure, the ambient light can be natural light or artificial light.

[0040] Among them, natural light is one of the most common light sources in our daily life. It is generated by the sun. The sun is a huge star with a core temperature as high as millions of degrees. Hydrogen atoms in its core undergo nuclear fusion under the action of high temperature and high pressure, releasing huge energy. This energy radiates outward in the form of light, forming natural light. Since the sun has been emitting light continuously, it can be considered that its emission frequency is positive infinity (infinite).

[0041] All light sources manufactured by humans are artificial light relative to natural light, such as lights, light emitters, etc. Lights can include the light emitted by incandescent lamps, fluorescent lamps, or LED lamps, etc. Artificial light sources such as incandescent lamps, fluorescent lamps, or LED lamps usually operate directly at the power supply frequency. Therefore, their light emission frequency is the same as the power supply frequency. In some exemplary embodiments, the first frequency f1 is preset by the electronic device according to the power supply frequency of its own location. For example, the first frequency f1 can be the power supply frequency. The power supply frequency refers to the rated frequency adopted by power generation, transmission, transformation, and distribution equipment of the power system, as well as industrial and civil electrical equipment, with the unit of Hertz (Hz). Some countries (such as China, India, etc.) adopt 50Hz, and some countries (such as the United States, Japan, etc.) adopt 60Hz. For example, assuming the location of the electronic device is in China, the first frequency f1 pre-stored in the electronic device is 50Hz.

[0042] In some other exemplary embodiments, the first frequency f1 is set by the electronic device according to the light emission frequency of the ambient light detected by a pre-set ambient light sensor. For example, if the light emission frequency of the ambient light detected by the ambient light sensor is 60Hz, then the electronic device sets the first frequency f1 to 60Hz.

[0043] In some exemplary embodiments, before the step of controlling the frame rate of the image acquisition device to be the second frequency f2, the method further includes:

[0044] Detect whether the light emission frequency of the ambient light is the first frequency f1. When the light emission frequency of the ambient light is the first frequency f1, trigger the step of controlling the frame rate of the image acquisition device to be the second frequency f2.

[0045] In some exemplary embodiments, as Figure 1 shown, the method further includes:

[0046] Step 103: When the electronic device does not turn on the perspective function, control the frame rate of the image acquisition device to be the third frequency f3, and the third frequency f3 is greater than the second frequency f2.

[0047] Still taking the electronic device as a head-mounted display as an example, when it is detected that the head-mounted display does not turn on (or turns off) the perspective function, control the frame rate of the image acquisition device to be the third frequency f3, and the third frequency f3 is greater than the second frequency f2. This can improve the accuracy of gesture tracking of the head-mounted display, reduce the latency of gesture interaction, so as to be able to respond to the control actions corresponding to gestures in a timely manner and ensure the smoothness of the display. The present disclosure controls the image acquisition device to adopt different frame rates in the perspective mode and the non-perspective mode to adapt to the different functional requirements of different modes.

[0048] In some exemplary embodiments, the method further includes:

[0049] When the electronic device enables the perspective function and the emission frequency of the ambient light is not the first frequency f1, control the acquisition frame rate of the image acquisition device to be the fourth frequency f4, where the fourth frequency f4 is greater than the second frequency f2.

[0050] When the emission frequency of the ambient light is not the first frequency f1, the user may be outdoors or indoors without lighting. At this time, the acquisition frame rate of the image acquisition device can be controlled to be a relatively high frame rate (or the highest frame rate), and there is no need to worry about the problem of flicker. In the embodiments of the present disclosure, the fourth frequency f4 may be equal to the third frequency f3 or may not be equal to the third frequency f3, and the embodiments of the present disclosure do not limit this.

[0051] In some exemplary embodiments, detecting whether the electronic device enables the perspective function includes any one or more of the following:

[0052] Detecting whether an instruction to enable the perspective function is received, detecting whether the user reaches the safety boundary, and detecting whether the distance between an external object and the user is less than or equal to a preset distance threshold.

[0053] Still taking the electronic device as a head-mounted display as an example, in the embodiments of the present disclosure, the head-mounted display enables the perspective function, which can be manually enabled by the user or automatically triggered when some automatic trigger conditions are met. These automatic trigger conditions include but are not limited to the user reaching the safety boundary, the distance between an external object and the user being less than or equal to a preset distance threshold, and so on.

[0054] Correspondingly, the head-mounted display turns off the perspective function, which can be manually turned off by the user or automatically triggered when some automatic trigger conditions are met. These automatic trigger conditions include but are not limited to the user moving away from the safety boundary, the distance between an external object and the user being greater than a preset distance threshold, and so on.

[0055] In some exemplary embodiments, the method further includes:

[0056] Obtain the screen display frame rate of the electronic device;

[0057] Detect whether the ratio of the screen display frame rate of the electronic device to the acquisition frame rate of the image acquisition device is within a preset ratio range;

[0058] When the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is not within the preset ratio range, adjust the screen display frame rate so that the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within the preset ratio range.

[0059] Taking the electronic device as a head-mounted display as an example, in the embodiments of the present disclosure, the screen display frame rate is the number of image frames displayed on the screen of the head-mounted display per second. After the image acquisition device acquires an image, it transmits the acquired image to the processing device in the head-mounted display. The processing device processes the acquired image, converts it into an image that can be perceived by humans, and presents it on the screen. When the acquisition frame rate of the image acquisition device is less than the screen display frame rate of the head-mounted display, there will be a problem that the environmental image has not been transmitted when the screen is refreshed. Therefore, in the embodiments of the present disclosure, the acquisition frame rate of the image acquisition device should be greater than or equal to the screen display frame rate, that is, the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device should be within a preset ratio range, and the preset ratio range is less than or equal to 1. Exemplarily, the preset ratio range can be [1 / 2, 1]. However, the present disclosure does not limit this.

[0060] In some exemplary embodiments, such as Figure 3 shown, the embodiments of the present disclosure further provide a display method, including:

[0061] Detecting whether the perspective mode is enabled;

[0062] When the perspective mode is enabled, determining the corresponding power supply power frequency according to the current location (for example, selecting 50 Hz in China), and configuring the acquisition frame rate of the image acquisition device (such as a camera) to operate according to the determined power supply power frequency (or an integer multiple of the power supply power frequency).

[0063] In the embodiments of the present disclosure, after the electronic device is powered on, it detects whether the perspective mode is enabled. When the perspective mode is enabled, the acquisition frame rate of the image acquisition device can be directly controlled according to the power supply power frequency of the current location (in the foregoing embodiments, the acquisition frame rate of the image acquisition device is controlled according to the emission frequency of the ambient light). Since the head-mounted display is generally used under indoor lighting conditions, directly controlling the acquisition frame rate of the image acquisition device according to the power supply power frequency of the current location, such a setting mode is simpler and more efficient, thus simplifying the device requirements. In the perspective mode, it is further determined whether the electronic device turns off the perspective mode.

[0064] In some exemplary embodiments, such as Figure 3 shown, the method further includes:

[0065] When the perspective mode is not enabled (or turned off), determining whether the gesture recognition function is enabled;

[0066] When the gesture recognition function is enabled, controlling the acquisition frame rate of the image acquisition device to be the third frequency f3, and the third frequency f3 is greater than the second frequency f2.

[0067] In an embodiment of the present disclosure, in the non-perspective mode, when the gesture recognition function is enabled, the acquisition frame rate of the image acquisition device is configured to work at a higher frame rate or the highest frame rate suitable for the screen display frame rate, so as to improve the accuracy of gesture tracking of the head-mounted display and reduce the latency of gesture interaction, thereby enabling timely response to control actions corresponding to gestures.

[0068] In some exemplary embodiments, as Figure 4 shown, an embodiment of the present disclosure further provides a display method, including:

[0069] Detecting whether the perspective function is enabled;

[0070] When the perspective function is enabled, detecting whether there is interference between the emission frequency of the ambient light and the acquisition frame rate of the image acquisition device; when there is interference between the emission frequency of the ambient light and the acquisition frame rate of the image acquisition device, controlling the acquisition frame rate of the image acquisition device to switch to a second frequency f2, and the second frequency f2 is an integer multiple of the emission frequency of the ambient light.

[0071] In an embodiment of the present disclosure, detecting whether there is interference between the emission frequency of the ambient light and the acquisition frame rate of the image acquisition device means detecting whether the ambient light is non-natural light and whether the acquisition frame rate of the image acquisition device is not an integer multiple of the emission frequency of the ambient light. When the ambient light is non-natural light and the acquisition frame rate of the image acquisition device is not an integer multiple of the emission frequency of the ambient light, there is interference between the emission frequency of the ambient light and the acquisition frame rate of the image acquisition device.

[0072] In an embodiment of the present disclosure, when it is detected that the condition for enabling the perspective mode is met (such as manually enabling the perspective function or the user reaches the safety boundary, an external object is within a predetermined distance range from the wearer, etc.), the ambient light sensor sends the detected emission frequency of the ambient light to the processing device. At the same time, the processing device receives the acquisition frame rate of the image acquisition device. The processing device determines the matching degree of the two according to the received emission frequency of the ambient light and the acquisition frame rate of the image acquisition device (that is, determines whether there is interference between the two). When the emission frequency of the ambient light and the acquisition frame rate of the image acquisition device do not meet the preset matching relationship (such as they are not equal or the acquisition frame rate is not an integer multiple of the emission frequency), controlling the acquisition frame rate of the image acquisition device to switch to the second frequency f2, so that the second frequency f2 and the emission frequency of the ambient light meet the preset matching relationship. For example, when it is detected that the user wearing the terminal device is indoors and uses a lamp for lighting, and the acquisition frame rate of the image acquisition device is 70Hz, a flickering phenomenon will occur if they do not match. The acquisition frame rate of the image acquisition device is switched to the second frequency. For example, assuming that the power supply power frequency is 50Hz, the acquisition frame rate of the image acquisition device can be switched to 50Hz.

[0073] The processing device receives the current screen display frame rate and determines whether the ratio of the second frequency f2 to the screen display frame rate is within a preset range (alternatively, determines whether the second frequency f2 is greater than or equal to the current screen display frame rate). If it is within the preset range (alternatively, the second frequency f2 is greater than or equal to the current screen display frame rate), it controls the image acquisition device to operate at the second frequency f2. When the ratio of the second frequency f2 to the screen display frame rate is not within the preset range (alternatively, the second frequency f2 is less than the current screen display frame rate), it adjusts the screen display frame rate so that the ratio of the second frequency f2 to the screen display frame rate is within the preset range (alternatively, the second frequency f2 is greater than or equal to the screen display frame rate). And it detects whether the perspective function is turned off.

[0074] When it is detected that the user wearing the terminal device is outdoors or indoors without lighting, the processing device learns from the detected luminous frequency of the ambient light that the luminous frequency of the current ambient light is not the power frequency. At this time, it can control the image acquisition device to acquire images at a frame rate higher than the second frequency. For example, it can control the acquisition frame rate of the image acquisition device to be 70Hz or 80Hz, etc.

[0075] When it is detected that the perspective function is not turned on, it controls the image acquisition device to operate at the third frequency f3, where the third frequency f3 is greater than the second frequency f2.

[0076] When it is detected that the perspective function is turned off (from being turned on to being turned off, the condition for turning off the perspective function is determined according to the condition for turning on the perspective function. When it is detected that the wearer retreats within the safe range, the perspective function is turned off; it can also be manually turned off, etc.), it controls the image acquisition device to operate at the third frequency f3, where the third frequency f3 is greater than the second frequency f2, and it detects the current screen display frame rate. If the ratio of the third frequency f3 to the screen display frame rate is not within the preset range, then it adjusts the screen display frame rate so that the ratio of the adjusted screen display frame rate to the current acquisition frame rate is within the preset ratio range, thereby being able to improve the accuracy of gesture tracking of the terminal device, reduce the latency of gesture interaction, and be able to meet the user's requirements for the display screen.

[0077] In some other exemplary embodiments, as Figure 5 shown, the method further includes:

[0078] In the perspective mode, it detects whether the frequency of the ambient light changes;

[0079] When the frequency of the ambient light changes, it controls the acquisition frame rate of the image acquisition device to switch to the fourth frequency f4, where the fourth frequency f4 is greater than the second frequency f2.

[0080] In an embodiment of the present disclosure, when it is detected that the emission frequency of the ambient light changes, the acquisition frame rate of the image acquisition device is switched to the fourth frequency f4, and the fourth frequency f4 is greater than the second frequency f2. Corresponding scenarios include, for example, detecting that the wearer walks from indoors to outdoors, or turning off the lights indoors, so as to improve the update rate of the external environment and obtain the external scene in a timely manner.

[0081] In an embodiment of the present disclosure, after the acquisition frame rate of the image acquisition device is switched to the fourth frequency f4, it can continue to be determined whether the fourth frequency f4 matches the current screen display frame rate (for example, determining whether the ratio of the current screen display frame rate to the fourth frequency f4 is within a preset ratio range, or determining whether the fourth frequency f4 is greater than or equal to the current screen display frame rate). When the fourth frequency f4 does not match the current screen display frame rate, the current screen display frame rate is adjusted so that the fourth frequency f4 matches the adjusted screen display frame rate; or, the acquisition frame rate of the image acquisition device is adjusted so that the adjusted acquisition frame rate matches the current screen display frame rate.

[0082] Exemplarily, assume that the current screen display frame rate is the seventh frequency f5, the acquisition frame rate of the image acquisition device is the fourth frequency f4, the optional screen display frame rates further include the eighth frequency f6, the optional acquisition frame rates of the image acquisition device further include the fifth frequency f7, etc. When the fourth frequency f4 is greater than or equal to the current screen display frame rate f5, control the image acquisition device to work at the fourth frequency f4, and control the screen to work at the current screen display frame rate f5; when the fourth frequency f4 is less than the current screen display frame rate f5, control the image acquisition device to work at the fifth frequency f7, where f7 is greater than f4 and f7 is less than f5. Or, control the screen display frame rate to work at the eighth frequency f6, and the acquisition frame rate of the image acquisition device is the fourth frequency f4, where f6 is less than f5 and f4 is greater than f6.

[0083] In some other exemplary embodiments, as Figure 6 shown, the method further includes:

[0084] When the electronic device has enabled the perspective function, detect whether the head movement speed is greater than or equal to a preset movement threshold;

[0085] When the head movement speed is greater than or equal to the preset movement threshold, control the acquisition frame rate of the image acquisition device to be the fifth frequency f7, and the fifth frequency f7 is greater than the second frequency f2;

[0086] When the head movement speed is less than the preset movement threshold, trigger the step of detecting the emission frequency of the ambient light.

[0087] In the 3D playback mode, by turning the head, the user can freely observe various corners of the virtual view and / or the surrounding environment. This characteristic of free observation is closer to the observation experience in the real world. When it is detected that the head rotation speed exceeds a certain speed (i.e., it is detected that the rotation speed exceeds the preset motion threshold), the image acquisition device is controlled to operate at the fifth frequency f7, and the fifth frequency f7 is greater than the second frequency f2 (since in the process of motion, from the acquisition of the picture by the camera system to the final display of the picture on the screen, it needs to go through the steps of acquisition → processing → rendering. Therefore, in order to prevent the picture from freezing, the acquisition frame rate of the camera, that is, the fifth frequency f7, should be greater than the second frequency f2. In addition, the fifth frequency f7 is greater than or equal to the current screen display frame rate), so as to increase the frequency of obtaining the external scene, meet the matching with the head rotation speed, prevent the picture from freezing due to too fast head rotation, and monitor the head rotation speed in real time. Once it is detected that the head rotation speed is lower than the preset motion threshold within the preset time, the step of detecting whether there is interference between the emission frequency of the ambient light and the camera acquisition frame rate will be switched to.

[0088] In some exemplary embodiments, the method further includes:

[0089] When the electronic device does not turn on the perspective function, obtain the frame rate recommended by the application;

[0090] When the frame rate recommended by the application is less than or equal to the third frequency f3, control the acquisition frame rate of the image acquisition device to be the frame rate recommended by the application;

[0091] When the frame rate recommended by the application is greater than the third frequency f3, control the acquisition frame rate of the image acquisition device to be the sixth frequency, and the sixth frequency is less than the frame rate recommended by the application.

[0092] In the non-perspective mode, the process of gesture interaction between different applications is generally different. In the non-perspective mode, the acquisition frame rate of the image acquisition device determined by the processing device is the third frequency f3, and the third frequency f3 may be different from the frame rate recommended by the application. The frame rate recommended by the application may be greater than, equal to, or less than this third frequency.

[0093] As Figure 7 shown, when the frame rate recommended by the application is less than the third frequency f3, control the image acquisition device to operate at the frame rate recommended by the application, reduce the processing load of the processing device, and be able to determine the corresponding frame rate of the camera specifically required for the gesture interaction process according to the specific application; for example, when the frame rates supported by the camera are 50, 60, 70Hz (the highest frame rate supported by the camera is 70Hz), and the frame rate recommended by the application is 60Hz, then control the camera to operate at 60Hz at this time; when the frame rate recommended by the application is greater than the third frequency f3, the processing device combines the application scenario complexity, CPU processing ability, and screen refresh rate to determine the camera frame rate corresponding to the application.

[0094] For example, assume that the acquisition frame rates supported by an image acquisition device (such as a camera) are 50, 60, 70, 80, 90, 100 Hz. In an application, the recommended acquisition frame rate by the application is 120 Hz. At this time, the processing device combines the application scenario complexity, CPU processing power, and screen refresh rate to determine the camera frame rate corresponding to this application. Assume that the processing device combines the application scenario complexity, CPU processing power, and screen refresh rate to determine that the suitable frame rate for the camera is 90 Hz. Then, control the camera to work at a frame rate of 90 Hz at this time. In another application, the recommended frame rate by the application is 110 Hz. The processing device combines the application scenario complexity, CPU processing power, and screen refresh rate to determine that the suitable frame rate for the camera is 100 Hz. Then, control the camera to work at a frame rate of 100 Hz at this time.

[0095] In some exemplary embodiments, the method further includes:

[0096] Obtain the maximum screen display frame rate. When the maximum screen display frame rate is greater than the recommended frame rate by the application, adjust the acquisition frame rate of the image acquisition device to the recommended frame rate by the application;

[0097] When the maximum screen display frame rate is less than the recommended frame rate by the application, perform data statistics based on the amount of data collected by the image acquisition device and the amount of data to be displayed on the display screen, and determine the acquisition frame rate and screen display frame rate of the image acquisition device according to the statistical results.

[0098] Exemplarily, as Figure 8 shown, assume that the obtained maximum screen display frame rate is N. When the value of N is greater than the recommended frame rate M by the application, it means that the system frame rate cannot render the amount of data exceeding the current display total. At this time, adjust the acquisition frame rate of the image acquisition device to the recommended frame rate M by the application (the screen display frame rate also uses the recommended frame rate M by the application).

[0099] When the value of N is less than the recommended frame rate M by the application, input the camera and display parameters into the data statistics module for data statistics, and output the best frame rate x that the current system can execute based on the total data amount of the display and the camera. The design principle of the data statistics module is to calculate the best operating frame rate by combining the current CPU / GPU gear and the total amount of data that can be rendered. As above, after obtaining the best operating frame rate, the processing device will send a frame rate switching command to the display screen, and finally the image acquisition device and the display screen will operate at the same frame rate.

[0100] In some exemplary embodiments, the second frequency is generated by the user selecting according to the first selectable frame rate of the electronic device, or the second frequency is generated by the processing device based on the power supply power frequency of the region where the electronic device is located, where the first selectable frame rate of the electronic device includes at least one second frequency recommended frame rate, and the second frequency recommended frame rate is consistent with the power supply power frequency of the region where the electronic device is located.

[0101] In some exemplary embodiments, the third frequency is generated by the user according to the second selectable frame rate of the electronic device, or the third frequency is generated by the processing device based on the load condition of the electronic device.

[0102] In the embodiments of the present disclosure, the user can manually set the third frequency and the second frequency of the image acquisition device. For example, the user can select 50Hz / 90Hz, 50Hz / 72Hz, 50 / 120Hz, 60Hz / 90Hz, 60Hz / 72Hz, 60 / 120Hz, etc. Among them, the number before the " / " represents the acquisition frame rate of the image acquisition device in the perspective mode, and the number after the " / " represents the acquisition frame rate of the image acquisition device in the non-perspective mode. For example, 50Hz / 90Hz means that the image acquisition device works at an acquisition frame rate of 50Hz in the perspective mode and 90Hz in the non-perspective mode; 60Hz / 90Hz means that the camera works at an acquisition frame rate of 60Hz in the perspective mode and 90Hz in the non-perspective mode; and so on for others.

[0103] In the embodiments of the present disclosure, the electronic device can recommend the acquisition frame rate in the perspective mode according to the power frequency of the current region. For example, in China, when the user makes a selection, the three options of 50Hz / 90Hz, 50Hz / 72Hz, and 50 / 120Hz are set to gray to recommend that the user preferably select a suitable acquisition frame rate from these 3 options. When the electronic device moves to other regions with different power frequencies, the recommended acquisition frame rate of the electronic device can change accordingly. For example, in the United States, the three options of 60Hz / 90Hz, 60Hz / 72Hz, and 60 / 120Hz are set to gray to recommend that the user preferably select a suitable acquisition frame rate from these 3 options. The above second selectable frame rates include values such as 90Hz, 72Hz, and 120Hz. Of course, the present disclosure only uses these values as examples for illustrative purposes. The electronic device can recommend other frequency values according to needs, or can also generate the value of the third frequency according to its own load condition (the load condition can include the screen display ability, image acquisition ability, frame rate recommended by applications, etc.) of the electronic device.

[0104] The embodiments of the present disclosure further provide a processing device, which may include a processor and a memory storing a computer program that can run on the processor. When the processor executes the computer program, the steps of the display method described in any one of the previous items in the present disclosure are implemented.

[0105] Such as Figure 10As shown, in one example, the processing device may include: a processor 1010, a memory 1020, a bus system 1030, and a transceiver 1040. Among them, the processor 1010, the memory 1020, and the transceiver 1040 are connected through the bus system 1030. The memory 1020 is used to store instructions, and the processor 1010 is used to execute the instructions stored in the memory 1020 to control the transceiver 1040 to send signals. Specifically, the transceiver 1040 may receive the emission frequency of the ambient light detected by the ambient light sensor under the control of the processor 1010. The processor 1010 detects whether the electronic device enables the perspective function. When the electronic device enables the perspective function, the acquisition frame rate of the image acquisition device is controlled to be a second frequency, and the second frequency is an integer multiple of the first frequency, and the first frequency is the emission frequency of the ambient light.

[0106] It should be understood that the processor 1010 may be a central processing unit (CPU), and the processor 1010 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0107] The memory 1020 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1010. A part of the memory 1020 may also include a non-volatile random access memory. For example, the memory 1020 may also store information about the device type.

[0108] In addition to including a data bus, the bus system 1030 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clear illustration, in Figure 10 all kinds of buses are labeled as the bus system 1030.

[0109] In the implementation process, the processing performed by the processing device may be completed by the integrated logic circuit in the hardware of the processor 1010 or the instructions in the form of software. That is, the method steps of the embodiments of the present disclosure may be embodied as being executed by the hardware processor, or completed by a combination of the hardware and software modules in the processor. The software module may be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 1020, and the processor 1010 reads the information in the memory 1020 and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0110] Embodiments of the present disclosure also provide an electronic device, including an ambient light sensor, a display screen, an image acquisition device, and a processing device. The processing device may be the processing device described in any embodiment of the present disclosure. The ambient light sensor is configured to detect the emission frequency of ambient light; the image acquisition device is configured to acquire an ambient image and / or a hand image in a perspective mode; and acquire a hand image in a non-perspective mode and transmit the acquired image to the processing device.

[0111] In embodiments of the present disclosure, the electronic device may be a head-mounted display or other electronic devices, and the embodiments of the present disclosure do not limit this.

[0112] In an exemplary embodiment, the head-mounted display further includes a head movement tracking sensor, and X, Y, Z axis and front and rear side tracking are implemented through the head movement tracking sensor.

[0113] In existing VR display devices, the graphics card is responsible for rendering VR images, and the display is responsible for displaying the VR images rendered by the graphics card; when the rendering speed of the graphics card is faster, the number of frames of the VR images is higher, and the more VR images the graphics card transmits to the display. When the refresh rate of the display is relatively high, the more VR images the display can display. Specifically, when the number of frames of the image is higher than the refresh rate, the frame output speed of the graphics card is higher than the refresh speed of the display, resulting in the display being unable to process the output image frames in time, causing tearing of each frame image displayed by the display; at the same time, when the display is unable to process the output image frames in time, the VR images output by the graphics card cannot be fully displayed by the display, resulting in frame loss problems. For example, when the graphics card renders 100 images per second (100 frames of images per second), and the display can only display 60 images per second (the refresh rate of the display is 60 Hz), then the user can only see 60 images rendered by the graphics card, and the remaining 40 images cannot be displayed by the display, causing the graphics card to do useless work to a certain extent and resulting in frame loss problems in the images displayed by the display. When the number of frames of the image is lower than the refresh rate, the display can not only fully display the images rendered by the graphics card, but also at least two consecutive frames of an image are displayed by the display, and this phenomenon is called the "image jitter phenomenon". Therefore, when the refresh rate of the display matches the number of frames of the VR images, the screen can display smooth VR images, bringing a high visual experience to the user.

[0114] To match certain scenes presented in the immersive environment provided by a VR display device, users will perform human-computer interaction with the VR display device through head movements, so that users can enjoy a real usage experience. Due to the limitation of the rendering ability of the graphics processor of the VR display device, when certain scenes presented in the immersive environment are relatively complex, if the user's head movement speed is relatively fast and the number of image frames is lower than the refresh rate, if there is an image jitter phenomenon, there will be a problem of image delay in the image displayed on the monitor. For example: when the user keeps the head movement stationary, the image jitter phenomenon will not cause discomfort to the user; when the user's head rotates from the old position to the new position, the image jitter phenomenon makes the current frame image displayed on the monitor still be the image of the user's head at the old position. At this time, the user's brain reaction has switched to the new position, but the image information received by the eyes is still the image information of the old position. At this time, the image information received by the eyes does not match the image information recognized by the brain, which will make the user feel dizzy; moreover, the larger the head rotation angle, the stronger this sense of dizziness will be.

[0115] Time warp (TW) is a technology for image frame correction. It obtains the current frame image by performing warping processing on the previous frame image and transmits the obtained current frame image to the monitor to reduce the head dizziness caused by the image delay problem. Generally speaking, the method adopted for warping processing is the direction-based warping processing technology, which can correct the head rotation change posture. This warping processing technology has obvious warping effects on two-dimensional images and does not require too much system resources. For example, for complex scenes, the direction-based warping processing technology can generate the current frame image based on a relatively small amount of data processing. Specifically, before receiving the current frame vertical synchronization signal, the graphics card needs to complete the TW processing of the previous frame image to obtain the current frame image, so as to ensure that after receiving the current frame synchronization signal, the monitor can display the current frame image and avoid the image delay problem during head rotation. That is to say, within the rendering time of one frame of image, it is necessary to sequentially complete the current frame image rendering operation (rendering operation) and the TW processing operation (TW operation) of the current frame image to ensure that the monitor can display the current frame image processed by TW in the next frame.

[0116] In the existing TW technology, the image rendering process and the time warping process are carried out in the same thread, making it impossible for the image rendering process and the time warping process to be carried out simultaneously. If the graphics card performance is relatively low, it cannot sequentially complete the current frame image rendering operation and the current frame image TW processing operation within one frame rendering time. People have improved the TW technology. Asynchronous TimeWarp (ATW) technology is an intermediate frame generation technology, which can not only reduce image latency and solve the problem of head dizziness, but also complete the current frame image rendering operation and the current frame image TW processing operation within one frame rendering time when the graphics card performance is relatively low. Specifically, the ATW technology arranges the rendering operation and the TW operation in two threads to adapt to the graphics card performance, avoid image delay, and reduce image jitter. The thread where the rendering operation is located is defined as the rendering thread (MRT thread), and the thread where the ATW operation is located is defined as the asynchronous time warp thread (ATW thread).

[0117] In some exemplary embodiments, when the electronic device enables the perspective function and simultaneously displays the environmental image and the virtual view, the electronic device starts the camera thread, the rendering thread, and the asynchronous time warp thread, where:

[0118] At the first starting moment, the camera thread acquires the image data collected by the image acquisition device and sends it to the asynchronous time warp thread;

[0119] At the first starting moment, the rendering thread renders the first screen scene data based on the first predicted user pose data; sends the rendered first screen scene data to the asynchronous time warp thread at the first intermediate moment, and renders the second screen scene data based on the second predicted user pose data; sends the rendered second screen scene data to the asynchronous time warp thread at the first display moment;

[0120] At the first intermediate moment, the asynchronous time warp thread performs a fusion operation on the first screen scene data and the image data and corrects it based on the third predicted user pose data to obtain the first screen output data; outputs the first screen output data to the display screen for display at the first display moment, and performs a fusion operation on the second screen scene data and the image data and corrects it based on the third predicted user pose data to obtain the second screen output data; outputs the second screen output data to the display screen for display at the second display moment. The first intermediate moment is between the first starting moment and the first display moment, and the first display moment is between the first intermediate moment and the second display moment.

[0121] Exemplarily, the first starting moment can be the starting moment of the current frame, the first intermediate moment can be the intermediate moment between the current frame and the next frame, the first display moment can be the starting moment of the next frame, and the second display moment can be the intermediate moment between the next frame and the frame after the next frame.

[0122] In the embodiments of the present disclosure, the first predicted user posture data is the user posture data at the first display moment predicted based on the user posture data at the first starting moment, the second predicted user posture data is the user posture data at the second display moment predicted based on the user posture data at the first intermediate moment, and the third predicted user posture data is the user posture data at the first display moment predicted based on the user posture data at the first intermediate moment.

[0123] In some exemplary embodiments, when the electronic device turns on the perspective function and only displays the environmental image, the electronic device starts a rendering thread and an asynchronous time warp thread, where:

[0124] At the second starting moment, the rendering thread acquires the image data collected by the image acquisition device and sends it to the asynchronous time warp thread;

[0125] At the second intermediate moment, the asynchronous time warp thread corrects the image data based on the fourth predicted user posture data; and outputs the corrected data to the display screen for display at the third display moment, where the second intermediate moment is between the second starting moment and the third display moment.

[0126] Exemplarily, the second starting moment may be the starting moment of the current frame, the second intermediate moment may be the intermediate moment between the current frame and the next frame, and the third display moment may be the starting moment of the next frame.

[0127] In the embodiments of the present disclosure, the fourth predicted user posture data is the user posture data at the third display moment predicted based on the user posture data at the second intermediate moment.

[0128] In some exemplary embodiments, when the electronic device turns off the perspective function and only displays the virtual view, the electronic device starts a rendering thread and an asynchronous time warp thread, where:

[0129] At the third starting moment, the rendering thread renders the first screen scene data based on the fifth predicted user posture data; sends the rendered first screen scene data to the asynchronous time warp thread at the third intermediate moment, and renders the second screen scene data based on the sixth predicted user posture data; sends the rendered second screen scene data to the asynchronous time warp thread at the fourth display moment;

[0130] At a third intermediate moment, the asynchronous time warp thread corrects the rendered first-screen scene data based on the seventh predicted user pose data to obtain first-screen output data; outputs the first-screen output data to the display screen for display at a fourth display moment, and corrects the rendered second-screen scene data based on the seventh predicted user pose data to obtain second-screen output data; outputs the second-screen output data to the display screen for display at a fifth display moment. The third intermediate moment is between the third starting moment and the fourth display moment, and the fourth display moment is between the third intermediate moment and the fifth display moment.

[0131] Exemplarily, the third starting moment may be the starting moment of the current frame, the third intermediate moment may be the intermediate moment between the current frame and the next frame, the fourth display moment may be the starting moment of the next frame, and the fifth display moment may be the intermediate moment between the next frame and the frame after the next frame.

[0132] In the embodiments of the present disclosure, the fifth predicted user pose data is the user pose data at the fourth display moment predicted based on the user pose data at the third starting moment, the sixth predicted user pose data is the user pose data at the fifth display moment predicted based on the user pose data at the third intermediate moment, and the seventh predicted user pose data is the user pose data at the fourth display moment predicted based on the user pose data at the third intermediate moment.

[0133] In the embodiments of the present disclosure, the "first", "second", and "third" in the first starting moment, the second starting moment, and the third starting moment are only used to distinguish different embodiments, and there is no temporal order relationship among the three. Similarly, the "first", "second", and "third" in the first intermediate moment, the second intermediate moment, and the third intermediate moment are only used to distinguish different embodiments, and there is no temporal order relationship among the three. In addition, although the second display moment is after the first display moment and the fifth display moment is after the fourth display moment, the "first", "third", and "fourth" in the first display moment, the third display moment, and the fourth display moment are only used to distinguish different embodiments, and there is no temporal order relationship among the three.

[0134] Figures 12 to 14The figure shows a schematic diagram of the image processing process based on ATW in an embodiment of the present disclosure. Among them, VsyncN-1, VsyncN, VsyncN+1, and VsyncN+2 respectively represent the vertical synchronization signals of the (N-1)th frame, the Nth frame, the (N + 1)th frame, and the (N + 2)th frame. The moment of receiving the vertical synchronization signal of the (N-1)th frame is defined as the start moment of the (N-1)th frame, the moment of receiving the vertical synchronization signal of the Nth frame is defined as the end moment of the (N-1)th frame or the start moment of the Nth frame, the moment of receiving the vertical synchronization signal of the (N + 1)th frame is defined as the end moment of the Nth frame or the start moment of the (N + 1)th frame, and the moment of receiving the vertical synchronization signal of the (N + 2)th frame is defined as the end moment of the (N + 1)th frame or the start moment of the (N + 2)th frame.

[0135] The intermediate moment between the start moment of the (N-1)th frame and the end moment of the (N-1)th frame is defined as the intermediate moment of the (N-1)th frame; the intermediate moment between the start moment of the Nth frame and the end moment of the Nth frame is defined as the intermediate moment of the Nth frame; the intermediate moment between the start moment of the (N + 1)th frame and the end moment of the (N + 1)th frame is defined as the intermediate moment of the (N + 1)th frame.

[0136] Figure 12 It is a schematic diagram of the image processing process when only virtual views are displayed in the non-perspective mode. As Figure 12 shown, at the start moment of the (N-1)th frame (or the start moment of the Nth frame), the rendering thread renders the first-screen scene data based on the user pose data at the display moment (i.e., after 1.5 frames). At the intermediate moment of the (N-1)th frame (or the intermediate moment of the Nth frame), the rendered first-screen scene data is sent to the asynchronous time warping thread. At the same time, the second-screen scene data is rendered based on the user pose data at the display moment (i.e., after 1.5 frames). At the start moment of the Nth frame (or the start moment of the (N + 1)th frame), the rendered second-screen scene data is sent to the asynchronous time warping thread.

[0137] After receiving the rendered first-screen scene data at the intermediate moment of the (N-1)th frame (or the intermediate moment of the Nth frame), the asynchronous time warping thread corrects the rendered first-screen scene data based on the user pose data at the display moment (i.e., after 1 frame) to obtain the first-screen scene output data. After the correction is completed, the first-screen scene output data is output to the display screen at the end moment of the (N-1)th frame (or the end moment of the Nth frame); after receiving the rendered second-screen scene data at the end moment of the (N-1)th frame (or the end moment of the Nth frame), the asynchronous time warping thread corrects the rendered second-screen scene data based on the user pose data at the display moment (i.e., after 1 frame) to obtain the second-screen scene output data. After the correction is completed, the second-screen scene output data is output to the display screen at the intermediate moment of the Nth frame (or the intermediate moment of the (N + 1)th frame).

[0138] Optionally, the first screen may be a screen for displaying the image seen by the user's left eye, and the second screen may be a screen for displaying the image seen by the user's right eye. The first screen scene data may include various parameters of the objects that should be presented in the field of view of the user's left eye, while the second screen scene data may include various parameters of the objects that should be presented in the field of view of the user's right eye. As described above, the meanings of the first screen and the second screen may be interchanged, and the first screen and the second screen may be various types of display screens, such as a liquid crystal display (LCD), an organic light emitting diode display (OLED), a micro light emitting diode (Micro-LED) display, etc. These display devices achieve the refresh of a frame of display image, for example, by progressive scanning and other means.

[0139] To enable the image presented by the virtual reality device to be all the objects that the user can observe in the field of view when using the virtual reality device, usually all the objects within the user's field of view need to be rendered. At the same time, different images need to be presented to the user's left eye and right eye (that is, a parallax is formed), and then the user's brain synthesizes the different images of the left and right eyes, so as to present a stereoscopic image visually to the user. Thus, in the first screen scene data and the second screen scene data, the parameters of the same object may also be different. For example, for the same object, the position of the object observed by the left eye and the right eye, and the situation of light refracting through the object to the left and right eyes may also be different. Therefore, in the present disclosure, the MRT thread renders both the first screen scene data and the second screen scene data. By adopting such a method, compared with first rendering the object as a whole and then separately outputting the rendered output result to the first screen and the second screen through affine transformation, more accurate and more stereoscopic images can be obtained.

[0140] Optionally, the rendering performed on the first screen scene data and the second screen scene data is multi-render target rendering.

[0141] The multi-render target rendering technique can save the data of multiple pixels into different buffers (e.g., the buffer for color map, the buffer for normal map, and the buffer for depth map) so that this data becomes the parameters of subsequent lighting effect shaders, improving the fineness of the output image. Through the multi-render target rendering technique, the rendering of lighting information can be delayed, improving the rendering processing speed. At the same time, the MRT thread renders the first screen scene data and the second screen scene data respectively. Therefore, the pixel data of each object in the first screen scene data and the second screen scene data needs to be saved into the above-mentioned buffer respectively, and then the data is parsed in the lighting effect shader, thereby enhancing the lighting effect of the images displayed in the first screen and the second screen. Compared with the usual method of performing lighting shading calculations for each pixel in the entire scene using a lighting shader, in the embodiments of the present disclosure, using the MRT thread to perform multi-render target rendering on the first screen scene data and the second screen scene data respectively can reduce the number of arithmetic operations for the processor to perform lighting shading calculations and can obtain an image with an enhanced lighting effect.

[0142] After completing the rendering of the first screen scene data and the second screen scene data, the rendering thread triggers an asynchronous time warp thread (ATW thread) and sends the rendered first screen scene data and second screen scene data to the asynchronous time warp thread through thread communication with the asynchronous time warp thread. Then, the rendering thread waits for the next vertical synchronization signal (the second vertical synchronization signal) to perform the next cycle of rendering scene data.

[0143] It can be understood that when rendering a VR image, the VR image needs to be rendered according to the VR image rendering parameters. The VR image rendering parameters include rendering resolution, field of view angle, and user pose information. Among them, the field of view angle and user pose information can achieve texture rendering of the VR image, especially for a VR image with a complex scene, and the scene complexity and rendering resolution play a decisive role in the VR image rendering time. When performing TW processing, it is necessary to process the image information according to the user pose information to achieve time warping processing of the image information.

[0144] Figure 13 It is a schematic diagram of the image processing process when only the environment image is displayed in the perspective mode. As Figure 13 shown, at the start time of the (N - 1)th frame (or the start time of the Nth frame), the rendering thread acquires the image data collected by the image acquisition device and sends it to the asynchronous time warp thread.

[0145] As Figure 13As shown, at the midpoint of the (N-1)th frame (or the midpoint of the Nth frame), the asynchronous time warp thread corrects the received camera image data based on the user pose data at the display time (i.e., after 1 frame) to obtain the corrected camera image data. At the start of the Nth frame (or the start of the (N+1)th frame), after receiving the vertical synchronization signal of the Nth frame (or the (N+1)th frame), the asynchronous time warp thread outputs the corrected camera image data to the display screen for display.

[0146] Figure 14 It is a schematic diagram of the image processing process when both the environmental image and the virtual view are displayed in the perspective mode. As Figure 14 shown, at the start of the (N-1)th frame (or the start of the Nth frame), the camera thread acquires the image data collected by the image acquisition device and sends it to the asynchronous time warp thread. The rendering thread renders the first screen scene data based on the user pose data at the display time (i.e., after 1.5 frames), and sends the rendered first screen scene data to the asynchronous time warp thread at the midpoint of the (N-1)th frame (or the midpoint of the Nth frame). At the same time, it renders the second screen scene data based on the user pose data at the display time (i.e., after 1.5 frames), and sends the rendered second screen scene data to the asynchronous time warp thread at the start of the Nth frame (or the start of the (N+1)th frame).

[0147] After receiving the rendered first screen scene data and the camera image data at the midpoint of the (N-1)th frame (or the midpoint of the Nth frame), the asynchronous time warp thread performs a dual-image overlay and fusion operation on the first screen scene data and the camera image data to obtain the fused first screen scene data, and corrects the fused first screen scene data based on the user pose data at the display time (i.e., after 1 frame) to obtain the first screen scene output data. After the correction is completed, the first screen scene output data is output to the display screen for display at the end of the (N-1)th frame (or the end of the Nth frame); after receiving the rendered second screen scene data at the end of the (N-1)th frame (or the end of the Nth frame), it performs a dual-image overlay and fusion operation on the second screen scene data and the camera image data to obtain the fused second screen scene data, and corrects the fused second screen scene data based on the user pose data at the display time (i.e., after 1 frame) to obtain the second screen scene output data. After the correction is completed, the second screen scene output data is output to the display screen for display at the midpoint of the Nth frame (or the midpoint of the (N+1)th frame).

[0148] Embodiments of the present disclosure also provide a computer-readable storage medium storing executable instructions, which, when executed by a processor, can implement the display method provided in any of the above embodiments of the present disclosure. The display method can be used to control the display of the electronic device provided in the above embodiments of the present disclosure, solving the flicker problem caused by the interference between the emission frequency of the ambient light and the camera capture frame rate, and improving the user experience. The method of driving the electronic device to display by executing the executable instructions is basically the same as the display method provided in the above embodiments of the present disclosure, and will not be elaborated herein.

[0149] In the description of the embodiments of the present disclosure, it should be understood that the orientation or positional relationship indicated by the terms "middle", "upper", "lower", "front", "rear", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present disclosure.

[0150] In the description of the embodiments of the present disclosure, unless otherwise clearly specified and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the meanings of the above terms in the present disclosure can be understood accordingly.

[0151] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof. In the hardware implementation, the division between the functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be executed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0152] Although the embodiments disclosed in this disclosure are as above, the content described above is only an embodiment adopted for the convenience of understanding this disclosure and is not intended to limit this disclosure. Any person skilled in the art within the scope of this disclosure may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this disclosure. However, the protection scope of this disclosure shall still be subject to the scope defined by the appended claims.

Claims

1. A display method for an electronic device, characterized in that, The electronic device includes an image acquisition device, and the method includes: Detecting whether the electronic device enables the perspective function; When the electronic device enables the perspective function, controlling the acquisition frame rate of the image acquisition device to be a second frequency, where the second frequency is an integer multiple of a first frequency, and the first frequency is the emission frequency of ambient light.

2. The display method according to claim 1, wherein Before the step of controlling the acquisition frame rate of the image acquisition device to be the second frequency, the method further includes: Detecting the emission frequency of ambient light; When the emission frequency of the ambient light is the first frequency, triggering the step of controlling the acquisition frame rate of the image acquisition device to be the second frequency.

3. The display method according to claim 1 or 2, characterized in that, The detecting whether the electronic device enables the perspective function includes any one or more of the following: Detecting whether an instruction to enable the perspective function is received, detecting whether the user reaches a safety boundary, and detecting whether the distance between an external object and the user is less than or equal to a preset distance threshold.

4. The display method according to claim 1 or 2, characterized in that, The method further includes: Obtaining the screen display frame rate of the electronic device; Detecting whether the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within a preset ratio range; When the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is not within the preset ratio range, adjusting the screen display frame rate so that the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within the preset ratio range.

5. The display method according to claim 1 or 2, characterized in that The method further includes: Detecting whether the electronic device disables the perspective function; When the electronic device disables the perspective function, detecting whether an instruction to enable the gesture recognition function is received; When the instruction to enable the gesture recognition function is received, controlling the acquisition frame rate of the image acquisition device to be a third frequency, and the third frequency is greater than the second frequency.

6. The display method according to claim 1 or 2, characterized in that, The method further includes: Detecting whether the emission frequency of ambient light changes; When the emission frequency of the ambient light changes, controlling the acquisition frame rate of the image acquisition device to be a fourth frequency, the fourth frequency is greater than the second frequency, and detecting whether the ratio of the screen display frame rate of the electronic device to the acquisition frame rate of the image acquisition device is within a preset ratio range; When the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is not within the preset ratio range, adjusting the screen display frame rate so that the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within the preset ratio range.

7. The display method according to claim 1 or 2, characterized in that, The method further includes: When the electronic device enables the perspective function and the emission frequency of the ambient light is not the first frequency, controlling the acquisition frame rate of the image acquisition device to be a fourth frequency, and the fourth frequency is greater than the second frequency.

8. The display method according to claim 1 or 2, characterized in that The method further includes: When the electronic device has enabled the perspective function, detecting whether the head movement speed is greater than or equal to a preset movement threshold; When the head movement speed is greater than or equal to the preset movement threshold, controlling the acquisition frame rate of the image acquisition device to be a fifth frequency, and the fifth frequency is greater than the second frequency; When the head movement speed is less than the preset movement threshold, triggering the step of detecting the emission frequency of ambient light.

9. The display method according to claim 1 or 2, characterized in that The method further includes: When the electronic device does not turn on the perspective function, control the acquisition frame rate of the image acquisition device to be a third frequency, and the third frequency is greater than the second frequency.

10. The display method according to claim 9, wherein The method further includes: When the electronic device does not turn on the perspective function, obtain the frame rate recommended by the application; When the frame rate recommended by the application is less than or equal to the third frequency, control the acquisition frame rate of the image acquisition device to be the frame rate recommended by the application; When the frame rate recommended by the application is greater than the third frequency, control the acquisition frame rate of the image acquisition device to be a sixth frequency, and the sixth frequency is less than the frame rate recommended by the application.

11. The display method according to claim 10, wherein The method further includes: Obtain the maximum screen display frame rate of the electronic device, and detect the magnitude relationship between the maximum screen display frame rate and the frame rate recommended by the application; When the maximum screen display frame rate is greater than the frame rate recommended by the application, adjust the acquisition frame rate of the image acquisition device to be consistent with the frame rate recommended by the application; When the maximum screen display frame rate is less than the frame rate recommended by the application, perform data statistics based on the data volume collected by the image acquisition device and the data volume to be displayed on the display screen of the electronic device, and determine the acquisition frame rate of the image acquisition device and the screen display frame rate according to the statistical results.

12. The display method according to claim 9, wherein The second frequency is generated by the user selecting based on the first selectable frame rate of the electronic device, or the second frequency is generated by the electronic device based on the power supply power frequency in the area where the electronic device is located, where the first selectable frame rate includes at least one second frequency recommended frame rate, and the second frequency recommended frame rate is consistent with the power supply power frequency in the area where the electronic device is located; The third frequency is generated by the user selecting based on the second selectable frame rate of the electronic device, or the third frequency is generated by the electronic device based on the load condition of the electronic device.

13. The display method according to claim 1 or 2, characterized in that, When the electronic device turns on the perspective function and simultaneously displays the environmental image and the virtual view, the electronic device starts the camera thread, the rendering thread, and the asynchronous time warp thread, where: At the first start time, the camera thread obtains the image data collected by the image acquisition device and sends it to the asynchronous time warp thread; At the first start time, the rendering thread renders the first screen scene data based on the first predicted user pose data; sends the rendered first screen scene data to the asynchronous time warp thread at the first intermediate time, and renders the second screen scene data based on the second predicted user pose data; sends the rendered second screen scene data to the asynchronous time warp thread at the first display time; At the first intermediate moment, the asynchronous time warp thread performs a fusion operation on the first screen scene data and the image data and performs correction based on the third predicted user pose data to obtain first screen output data; at the first display moment, the first screen output data is output to the display screen for display, and a fusion operation on the second screen scene data and the image data is performed and correction is performed based on the third predicted user pose data to obtain second screen output data; at the second display moment, the second screen output data is output to the display screen for display, the first intermediate moment is between the first start moment and the first display moment, and the first display moment is between the first intermediate moment and the second display moment.

14. The display method according to claim 1 or 2, characterized in that, When the electronic device turns on the perspective function and only displays the environmental image, the electronic device starts a rendering thread and an asynchronous time warp thread, where: At the second start moment, the rendering thread acquires the image data collected by the image acquisition device and sends it to the asynchronous time warp thread; At the second intermediate moment, the asynchronous time warp thread corrects the image data based on the fourth predicted user pose data; at the third display moment, the corrected data is output to the display screen for display, and the second intermediate moment is between the second start moment and the third display moment.

15. The display method according to claim 1 or 2, characterized in that, When the electronic device turns off the perspective function and only displays the virtual view, the electronic device starts a rendering thread and an asynchronous time warp thread, where: At the third start moment, the rendering thread renders the first screen scene data based on the fifth predicted user pose data; at the third intermediate moment, the rendered first screen scene data is sent to the asynchronous time warp thread, and the second screen scene data is rendered based on the sixth predicted user pose data; at the fourth display moment, the rendered second screen scene data is sent to the asynchronous time warp thread; At the third intermediate moment, the asynchronous time warp thread corrects the rendered first screen scene data based on the seventh predicted user pose data to obtain first screen output data; at the fourth display moment, the first screen output data is output to the display screen for display, and the rendered second screen scene data is corrected based on the seventh predicted user pose data to obtain second screen output data; at the fifth display moment, the second screen output data is output to the display screen for display, the third intermediate moment is between the third start moment and the fourth display moment, and the fourth display moment is between the third intermediate moment and the fifth display moment.

16. A processing device, characterized in that, Including: A processor and a memory storing a computer program that can run on the processor, where the processor implements the steps of the display method according to any one of claims 1 to 15 when executing the computer program.

17. An electronic device, characterized in that, Including: An image acquisition device and a processing device according to claim 16.

18. A computer-readable storage medium, characterized in that, Storing computer-executable instructions for executing the display method according to any one of claims 1 to 15.

Images