Electronic device and display method therefor, processing device, and computer-readable storage medium

Abstract

An electronic device and a display method therefor, a processing device, and a computer-readable storage medium. The electronic device comprises an image collection apparatus. The method comprises: detecting whether an electronic device enables a see-through function; and when the electronic device enables the see-through function, controlling the collection frame rate of an image collection apparatus to be a second frequency, wherein the second frequency is an integer multiple of a first frequency, and the first frequency is the light-emission frequency of ambient light.

Description

Electronic device and display method thereof, processing device, and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 29, 2023, with application number 202311861641.1 and invention name “Electronic device and its display method, processing device, computer-readable storage medium”, the content of which should be understood as incorporated into this application by reference. Technical Field

[0002] The present disclosure relates to intelligent display technology, and in particular to an electronic device and a display method thereof, a processing device, and a computer-readable storage medium. Background Art

[0003] The Virtual Reality (VR) / Mixed Reality (MR) industry is rapidly developing. VR applications are computer-generated 3D environments that allow users to completely immerse themselves in a virtual world presented by a headset, oblivious to the real world. MR applications blend the real and virtual worlds to create new visual and interactive environments that incorporate both physical entities and virtual information. Real and virtual characters and objects can transcend the boundaries of reality, creating more complex and exciting experiences.

[0004] Some VR / MR products reuse the see-through camera and the gesture tracking camera. When the see-through conditions are met, the camera captures external images. However, the frequency of ambient light interferes with the camera's acquisition frame rate, especially when indoor light-emitting diode (LED) lights are on. This interferes severely with the camera's acquisition frame rate, causing noticeable flicker. This can cause discomfort to users and affect their user experience. Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The present disclosure provides a display method for an electronic device, wherein the electronic device includes an image acquisition device, and the display method includes: detecting whether a perspective function of the electronic device is turned on; when the perspective function is turned on, controlling the acquisition frame rate of the image acquisition device to be a second frequency, wherein the second frequency is an integer multiple of the first frequency, and the first frequency is the luminous frequency of ambient light.

[0007] An embodiment of the present disclosure further provides a processing device, comprising: a processor and a memory storing a computer program that can be run on the processor, wherein the processor implements the steps of the display method as described above when executing the program.

[0008] An embodiment of the present disclosure further provides an electronic device, comprising: an image acquisition device and a processing device as described in any embodiment of the present disclosure.

[0009] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the display method as described in any one of the above items can be implemented.

[0010] The electronic device and its display method, processing device, and computer-readable storage medium of the embodiments of the present disclosure solve the flicker problem caused by interference between the luminous frequency of the ambient light and the camera acquisition frame rate by controlling the acquisition frame rate of the image acquisition device to a second frequency when the perspective function of the electronic device is turned on, wherein the second frequency is an integer multiple of the first frequency, and the first frequency is the luminous frequency of the ambient light, thereby improving the user experience.

[0011] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. Other advantages of the present disclosure can be realized and obtained through the solutions described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide an understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation to the technical solution of the present disclosure.

[0013] FIG1 is a flow chart of a display method according to an exemplary embodiment of the present disclosure.

[0014] FIG2 is a schematic diagram of a method for using a head-mounted display according to an exemplary embodiment of the present disclosure.

[0015] 3 to 8 are flowcharts of several display methods according to exemplary embodiments of the present disclosure.

[0016] FIG9 is a schematic diagram of an interface for manually setting a capture frame rate of an image capture device according to an exemplary embodiment of the present disclosure.

[0017] FIG10 is a schematic structural diagram of a processing device according to an exemplary embodiment of the present disclosure.

[0018] FIG11 is a schematic structural diagram of a head-mounted display according to an exemplary embodiment of the present disclosure.

[0019] FIG12 is a schematic diagram of an image processing process when an electronic device displays only a virtual view in a non-perspective mode according to an exemplary embodiment of the present disclosure.

[0020] FIG13 is a schematic diagram of an image processing process when an electronic device displays only an environment image in a perspective mode according to an exemplary embodiment of the present disclosure.

[0021] FIG14 is a schematic diagram of an image processing process when an electronic device displays an environment image and a virtual view in a perspective mode according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The present disclosure describes a plurality of embodiments, but this description is exemplary rather than restrictive, and it will be apparent to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0023] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The disclosed embodiments, features, and elements of the present disclosure may also be combined with any conventional features or elements to form a unique inventive solution defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other inventive solutions to form another unique inventive solution defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this disclosure may be implemented individually or in any appropriate combination. Therefore, the embodiments are not subject to other limitations except for the limitations set forth in the appended claims and their equivalents. In addition, various modifications and changes may be made within the scope of protection of the appended claims.

[0024] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation on the claims. In addition, the claims to the method and / or process should not be limited to performing their steps in the order written, and those skilled in the art can readily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.

[0025] As shown in FIG1 , an embodiment of the present disclosure provides a display method for an electronic device, wherein the electronic device includes an image acquisition device, and the method includes:

[0026] Step 101: Detect whether the electronic device has a perspective function enabled;

[0027] Step 102: When the electronic device turns on the perspective function, the image acquisition device is controlled to have a capture frame rate of a second frequency f2, wherein the second frequency f2 is an integer multiple of the first frequency f1, and the first frequency f1 is the luminous frequency of the ambient light.

[0028] In the embodiment of the present disclosure, the electronic device may be a head-mounted display or other electronic device, and the embodiment of the present disclosure does not limit this.

[0029] As shown in FIG2 , in an embodiment of the present disclosure, a head-mounted display (HMD) is worn on the user's head and displays images in front of the user's eyes. The head-mounted display is generally formed in the shape of goggles or the frame shape of large glasses. The head-mounted display usually includes a display device and a headband. The display device has a high-resolution liquid crystal display or an organic light-emitting diode (OLED) display. It is usually called glasses because it is made in the shape of eyes. It is used to display the same (or different) video images to the left eye and the right eye; the headband is used to connect the display device and wear the display device on the user's head and face.

[0030] In the disclosed embodiment, the playback mode of the head-mounted display can be a non-perspective mode (i.e., the perspective function is turned off), in which case the head-mounted display presents a 3D or 2D virtual view; the playback mode of the head-mounted display can also be a perspective mode (i.e., the perspective function is turned on), in which case the head-mounted display presents an environmental image, or a fused image of the virtual view and the environmental image (whether to present the environmental image or the fused image of the virtual view and the environmental image can be set according to user needs).

[0031] In the embodiment of the present disclosure, the head-mounted display may include a set of optical lenses, which collimate, magnify and zoom out the image on the display screen in front of the eyes.

[0032] In some exemplary embodiments, parallax images can be provided to the left and right eyes sequentially in time. The difference in images between the left and right eyes creates a sense of space in the brain, allowing the left and right eyes to perceive slightly different images through the head-mounted display, thereby allowing them to experience a three-dimensional virtual view. Assuming the refresh rate of the head-mounted display is 120Hz, the frame rate for each eye is 60Hz when playing frames.

[0033] In some exemplary embodiments, the head-mounted display may include a head motion tracking sensor (not shown in FIG. 2 ), and the head motion tracking sensor is used to implement X, Y, Z axis and front and back side tracking.

[0034] In some exemplary embodiments, the head motion tracking sensor may include a gyroscope and an accelerometer. The gyroscope is used to measure the rotation angle of the head, and the accelerometer is used to measure the acceleration of the head. The gyroscope and accelerometer can be integrated into a six-axis inertial sensor. In other exemplary embodiments, the head motion tracking sensor may also include a magnetometer. The magnetometer can be used to measure the orientation of the head relative to the Earth's magnetic field. The gyroscope, accelerometer, and magnetometer can be integrated into a nine-axis sensor.

[0035] In an embodiment of the present disclosure, the head-mounted display includes an image acquisition device (not shown in Figure 2), which can be reused in perspective mode and non-perspective mode. In perspective mode, the image acquisition device can acquire environmental images and transmit them to a processing device. The processing device processes the acquired images, converts them into images that can be perceived by humans, and presents them on a screen. In non-perspective mode (and / or perspective mode), the image acquisition device can acquire hand images and transmit them to the processing device. The processing device detects operations performed by the user in a three-dimensional environment through a gesture recognition algorithm, including but not limited to playing video games, navigating menus, controlling media playback, etc.

[0036] In some exemplary embodiments, the image acquisition device may be a complementary metal oxide semiconductor (CMOS) image sensor. Currently, mainstream cameras and mobile phones all use CMOS image sensors. Since the exposure method of CMOS image sensors is performed row by row, the exposure start point and exposure time of each pixel on the same row are the same. Therefore, all pixels in the same row receive the same amount of light energy. However, although the exposure time is the same for pixels on different rows, the exposure start points are different. Therefore, the light energy received by pixels on different rows is not necessarily the same. When the luminous frequency of the ambient light does not match the acquisition frame rate of the image acquisition device, the light energy received by pixels on different rows is different, which produces a noticeable flicker phenomenon, affecting the user experience.

[0037] The display method of the electronic device of the embodiment of the present disclosure controls the acquisition frame rate of the image acquisition device to be a second frequency f2 when the perspective function of the electronic device, such as a head-mounted display, is turned on. The second frequency f2 is an integer multiple of the first frequency f1, so that all pixels receive the same light energy when receiving exposure, thereby solving the flicker problem caused by the mismatch between the luminous frequency of the ambient light and the camera acquisition frame rate, and improving the user experience.

[0038] In the embodiments of the present disclosure, the acquisition frame rate of the image acquisition device refers to the number of frames acquired by the image acquisition device per unit time. For example, assuming that the acquisition frame rate is 50 Hz, the number of image frames acquired by the image acquisition device per second is 50 frames. The exposure time of the image acquisition device refers to the time required for the photosensitive element of the image acquisition device to receive light during the shooting process. The higher the acquisition frame rate of the image acquisition device, the shorter its exposure time; the lower the acquisition frame rate of the image acquisition device, the longer its exposure time. The length of the exposure time has a certain impact on the image quality. For example, if the exposure time is set too long or too short, the acquired image may become blurred (if the exposure time is set too long, the instability of the subject will cause the acquired image to become blurred; if the exposure time is set too short, insufficient light energy received will also cause the acquired image to become blurred). Therefore, the acquisition frame rate of the image acquisition device cannot be set too high or too low.

[0039] Exemplarily, the second frequency f2=N*first frequency f1, where N is 1 or 2.

[0040] In the embodiment of the present disclosure, the ambient light may be natural light or artificial light.

[0041] Natural light is one of the most common light sources in our daily lives. It's produced by sunlight. The sun is a massive star with a core temperature of millions of degrees. Under the influence of high temperature and high pressure, hydrogen atoms in its core undergo nuclear fusion, releasing enormous amounts of energy. This energy radiates outward in the form of light, forming natural light. Because the sun shines continuously, its luminous frequency can be considered positive infinity (infinite).

[0042] In contrast to natural light, all luminous light sources manufactured by humans are artificial light, such as lamps, light emitters, etc. Light can include light emitted by incandescent lamps, fluorescent lamps, or LED lamps. Artificial light sources such as incandescent lamps, fluorescent lamps, or LED lamps usually operate directly at the power supply frequency, so their luminous frequency is the same as the power supply frequency. In some exemplary embodiments, the first frequency f1 is pre-set by the electronic device based on the power supply frequency of the region where the electronic device is located. For example, the first frequency f1 can be the power supply frequency. The power supply frequency refers to the rated frequency used by the power generation, transmission, transformation and distribution equipment of the power system, as well as industrial and civil electrical equipment, and the unit is Hertz (Hz). Some countries (such as China, India, etc.) use 50Hz, and some countries (such as the United States, Japan, etc.) use 60Hz. For example, assuming that the region where the electronic device is located is China, the first frequency f1 pre-stored in the electronic device is 50Hz.

[0043] In other exemplary embodiments, the first frequency f1 is set by the electronic device according to the luminous frequency of the ambient light detected by a preset ambient light sensor. For example, if the luminous frequency of the ambient light detected by the ambient light sensor is 60 Hz, the electronic device sets the first frequency f1 to 60 Hz.

[0044] In some exemplary embodiments, before the step of controlling the acquisition frame rate of the image acquisition device to be the second frequency f2, the method further includes:

[0045] Detecting whether the luminous frequency of the ambient light is the first frequency f1, and when the luminous frequency of the ambient light is the first frequency f1, triggering the step of controlling the acquisition frame rate of the image acquisition device to be the second frequency f2.

[0046] In some exemplary embodiments, as shown in FIG1 , the method further includes:

[0047] Step 103: When the perspective function of the electronic device is not enabled, the image acquisition device is controlled to have an acquisition frame rate of a third frequency f3, where the third frequency f3 is greater than the second frequency f2.

[0048] Taking a head-mounted display as an example, when it is detected that the head-mounted display's see-through function is not enabled (or disabled), the image acquisition device's acquisition frame rate is controlled to a third frequency f3, which is greater than the second frequency f2. This improves the head-mounted display's gesture tracking accuracy and reduces gesture interaction latency, enabling timely responses to control actions corresponding to gestures and ensuring smooth display. The present disclosure controls the image acquisition device to use different acquisition frame rates in see-through and non-see-through modes to accommodate the varying functional requirements of each mode.

[0049] In some exemplary embodiments, the method further comprises:

[0050] When the electronic device turns on the perspective function and the luminous frequency of the ambient light is not the first frequency f1, the acquisition frame rate of the image acquisition device is controlled to be the fourth frequency f4, wherein the fourth frequency f4 is greater than the second frequency f2.

[0051] When the ambient light frequency is not the first frequency f1, the user may be outdoors or indoors without lighting. In this case, the image acquisition device can be controlled to capture a higher frame rate (or the highest frame rate) without worrying about flicker. In the embodiment of the present disclosure, the fourth frequency f4 may be equal to the third frequency f3 or may not be equal to the third frequency f3, and the embodiment of the present disclosure does not limit this.

[0052] In some exemplary embodiments, detecting whether the electronic device has a perspective function enabled includes any one or more of the following:

[0053] Detect whether the instruction to turn on the perspective function is received, detect whether the user has reached the safety boundary, and detect whether the distance between the external object and the user is less than or equal to the preset distance threshold.

[0054] Still taking the electronic device as a head-mounted display as an example, in the embodiment of the present disclosure, the head-mounted display can activate the perspective function manually by the user, or it can be automatically triggered due to the satisfaction of some automatic triggering conditions. These automatic triggering conditions include but are not limited to the user reaching a safe boundary, the distance between the external object and the user being less than or equal to a preset distance threshold, etc.

[0055] Accordingly, the head-mounted display can turn off the perspective function manually by the user, or it can be automatically triggered due to the satisfaction of some automatic triggering conditions. These automatic triggering conditions include but are not limited to the user being away from the safety boundary, the distance between the external object and the user being greater than a preset distance threshold, etc.

[0056] In some exemplary embodiments, the method further comprises:

[0057] Obtaining the screen display frame rate of the electronic device;

[0058] Detecting whether a ratio of a screen display frame rate of the electronic device to a capture frame rate of the image capture device is within a preset ratio range;

[0059] When the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is not within the preset ratio range, the screen display frame rate is adjusted so that the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within the preset ratio range.

[0060] Continuing with the example of a head-mounted display (HMD) as an electronic device, in the embodiments of the present disclosure, the screen display frame rate refers to the number of image frames displayed per second on the HMD screen. After the image acquisition device captures the image, it transmits the captured image to a processing device in the HMD. The processing device processes the captured image, converts it into a human-perceivable image, and displays it on the screen. When the acquisition frame rate of the image acquisition device is lower than the screen display frame rate of the HMD, the problem of the ambient image not being transmitted when the screen refreshes may occur. Therefore, in the embodiments of the present disclosure, the acquisition frame rate of the image acquisition device should be greater than or equal to the screen display frame rate. That is, the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device should be within a preset ratio range, which is less than or equal to 1. For example, the preset ratio range can be [1 / 2, 1], however, the present disclosure does not impose any limitation on this.

[0061] In some exemplary embodiments, as shown in FIG3 , the present disclosure further provides a display method, including:

[0062] Check whether perspective mode is turned on;

[0063] When the perspective mode is turned on, the corresponding power frequency is determined according to the current region (such as 50Hz in China), and the acquisition frame rate of the image acquisition device (such as a camera) is configured to run according to the determined power frequency (or an integer multiple of the power frequency).

[0064] In the embodiment of the present disclosure, after the electronic device is turned on, it detects whether the perspective mode is turned on. When the perspective mode is turned on, the acquisition frame rate of the image acquisition device can be directly controlled according to the power frequency of the current area (the aforementioned embodiment controls the acquisition frame rate of the image acquisition device according to the luminous frequency of the ambient light). Since head-mounted displays are generally used under indoor lighting conditions, the acquisition frame rate of the image acquisition device is directly controlled according to the power frequency of the current area. This setting mode is simpler and more efficient, thereby simplifying the equipment requirements. In the perspective mode, it is further determined whether the electronic device has turned off the perspective mode.

[0065] In some exemplary embodiments, as shown in FIG3 , the method further includes:

[0066] When the perspective mode is not turned on (or off), determine whether the gesture recognition function is turned on;

[0067] When the gesture recognition function is turned on, the acquisition frame rate of the image acquisition device is controlled to be a third frequency f3, and the third frequency f3 is greater than the second frequency f2.

[0068] In the embodiment of the present disclosure, in the non-perspective mode, when the gesture recognition function is turned on, the acquisition frame rate of the image acquisition device is configured to operate at a higher frame rate or the highest frame rate suitable for the screen display frame rate, so as to improve the accuracy of the head-mounted display for gesture tracking and reduce the delay of gesture interaction, so as to be able to respond to the control action corresponding to the gesture in a timely manner.

[0069] In some exemplary embodiments, as shown in FIG4 , the present disclosure further provides a display method, including:

[0070] Detect whether the perspective function is turned on;

[0071] When the perspective function is turned on, it is detected whether there is interference between the luminous frequency of the ambient light and the acquisition frame rate of the image acquisition device; when there is interference between the luminous frequency of the ambient light and the acquisition frame rate of the image acquisition device, the acquisition frame rate of the image acquisition device is controlled to switch to the second frequency f2, and the second frequency f2 is an integer multiple of the luminous frequency of the ambient light.

[0072] In the disclosed embodiments, detecting whether there is interference between the luminous frequency of ambient light and the capture frame rate of an image capture device refers to detecting whether the ambient light is non-natural light and whether the capture frame rate of the image capture device is not an integer multiple of the luminous frequency of the ambient light. When the ambient light is non-natural light and the capture frame rate of the image capture device is not an integer multiple of the luminous frequency of the ambient light, interference exists between the luminous frequency of the ambient light and the capture frame rate of the image capture device.

[0073] In an embodiment of the present disclosure, when it is detected that a condition for enabling the see-through mode is met (such as manually enabling the see-through function, or the user reaching a safety boundary, or an external object being within a predetermined distance from the wearer), the ambient light sensor transmits the detected luminous frequency of the ambient light to the processing device. Simultaneously, the processing device receives the acquisition frame rate of the image acquisition device. The processing device determines a matching degree between the received luminous frequency of the ambient light and the acquisition frame rate of the image acquisition device (i.e., determines whether there is interference between the two). When the luminous frequency of the ambient light and the acquisition frame rate of the image acquisition device do not satisfy a preset matching relationship (e.g., the two are not equal or the acquisition frame rate is not an integer multiple of the luminous frequency), the acquisition frame rate of the image acquisition device is controlled to switch to a second frequency f2 so that the second frequency f2 satisfies a preset matching relationship with the luminous frequency of the ambient light. For example, when it is detected that the user wearing the terminal device is indoors and using a lamp for lighting, and the acquisition frame rate of the image acquisition device is 70 Hz, the mismatch between the two will cause flickering, and the acquisition frame rate of the image acquisition device is switched to the second frequency. For example, assuming the power supply frequency is 50 Hz, the acquisition frame rate of the image acquisition device can be switched to 50 Hz.

[0074] The processing device receives the current screen display frame rate and determines whether the ratio of the second frequency f2 to the screen display frame rate is within a preset range (or, determines whether the second frequency f2 is greater than or equal to the current screen display frame rate). If so, the image acquisition device is controlled to operate at the second frequency f2. When the ratio of the second frequency f2 to the screen display frame rate is not within the preset range (or, the second frequency f2 is less than the current screen display frame rate), the screen display frame rate is adjusted so that the ratio of the second frequency f2 to the screen display frame rate is within the preset range (or, the second frequency f2 is greater than or equal to the screen display frame rate). The processing device also detects whether the perspective function is turned off.

[0075] When it is detected that the user wearing the terminal device is outdoors or indoors without lighting, the processing device knows that the current luminous frequency of the ambient light is not the power frequency based on the detected luminous frequency of the ambient light. At this time, the image acquisition device can be controlled to capture images at a frame rate higher than the second frequency. For example, the acquisition frame rate of the image acquisition device can be controlled to 70Hz or 80Hz, etc.

[0076] When it is detected that the perspective function is not turned on, the image acquisition device is controlled to operate at a third frequency f3, wherein the third frequency f3 is greater than the second frequency f2.

[0077] When it is detected that the perspective function is turned off (the perspective function changes from on to off, and the conditions for turning off the perspective function are determined based on the conditions for turning on the perspective function, and the perspective function is turned off when it is detected that the wearer has returned to the safe range; the perspective function can also be turned off manually, etc.), the image acquisition device is controlled to operate at a third frequency f3, wherein the third frequency f3 is greater than the second frequency f2, and the current screen display frame rate is detected. If the ratio of the third frequency f3 to the screen display frame rate is not within a preset range, the screen display frame rate is adjusted so that the ratio of the adjusted screen display frame rate to the current acquisition frame rate is within the preset ratio range, thereby improving the accuracy of gesture tracking by the terminal device, reducing the delay in gesture interaction, and meeting the user's needs for the display screen.

[0078] In some other exemplary embodiments, as shown in FIG5 , the method further includes:

[0079] In perspective mode, detect whether the frequency of ambient light changes;

[0080] When the frequency of the ambient light changes, the acquisition frame rate of the image acquisition device is controlled to switch to a fourth frequency f4, wherein the fourth frequency f4 is greater than the second frequency f2.

[0081] In the embodiment of the present disclosure, when a change in the luminous frequency of the ambient light is detected, the acquisition frame rate of the image acquisition device is switched to the fourth frequency f4. The fourth frequency f4 is greater than the second frequency f2. The corresponding scenario is, for example, detecting that the wearer walks from indoors to outdoors, or turns off the lights indoors, thereby increasing the update rate of the external environment and obtaining the external scene in a timely manner.

[0082] In an embodiment of the present disclosure, after the acquisition frame rate of the image acquisition device is switched to the fourth frequency f4, it is possible to continue to determine whether the fourth frequency f4 matches the current screen display frame rate (for example, determine whether the ratio of the current screen display frame rate to the fourth frequency f4 is within a preset ratio range, or determine whether the fourth frequency f4 is greater than or equal to the current screen display frame rate). When the fourth frequency f4 does not match the current screen display frame rate, adjust the current screen display frame rate so that the fourth frequency f4 matches the adjusted screen display frame rate; or, adjust the acquisition frame rate of the image acquisition device so that the adjusted acquisition frame rate matches the current screen display frame rate.

[0083] For example, assuming that the current screen display frame rate is the seventh frequency f5, the acquisition frame rate of the image acquisition device is the fourth frequency f4, and the optional screen display frame rate also includes the eighth frequency f6, and the optional image acquisition frame rate also includes the fifth frequency f7, etc., when the fourth frequency f4 is greater than or equal to the current screen display frame rate f5, the image acquisition device is controlled to operate at the fourth frequency f4, and the screen is controlled to operate at the current screen display frame rate f5; when the fourth frequency f4 is less than the current screen display frame rate f5, the image acquisition device is controlled to operate at the fifth frequency f7, where f7 is greater than f4 and f7 is less than f5. Alternatively, the screen display frame rate is controlled to operate at the eighth frequency f6, and the acquisition frame rate of the image acquisition device is the fourth frequency f4, where f6 is less than f5 and f4 is greater than f6.

[0084] In some other exemplary embodiments, as shown in FIG6 , the method further includes:

[0085] When the electronic device has the perspective function turned on, detecting whether the head movement speed is greater than or equal to a preset movement threshold;

[0086] When the head movement speed is greater than or equal to a preset movement threshold, controlling the acquisition frame rate of the image acquisition device to be a fifth frequency f7, and the fifth frequency f7 is greater than the second frequency f2;

[0087] When the head movement speed is less than a preset movement threshold, the step of detecting the luminous frequency of the ambient light is triggered.

[0088] In 3D playback mode, users can freely observe every corner of the virtual view and / or surrounding environment by turning their head. This free observation feature more closely resembles the real-world observation experience. When the head rotation speed is detected to exceed a certain speed (i.e., the rotation speed exceeds a preset motion threshold), the image acquisition device is controlled to operate at a fifth frequency f7, which is greater than the second frequency f2. (Since the process from camera system acquisition to screen display requires the following steps: acquisition → processing → rendering, to prevent image lag, the camera's acquisition frame rate, i.e., the fifth frequency f7, must be greater than the second frequency f2. Furthermore, the fifth frequency f7 must be greater than or equal to the current screen display frame rate.) This increases the frequency of external scene acquisition to match the head rotation speed, preventing image lag caused by excessive head rotation. The head rotation speed is monitored in real time. Once the head rotation speed is detected to be below the preset motion threshold within a preset time, the device switches to detecting whether there is interference between the ambient light frequency and the camera acquisition frame rate.

[0089] In some exemplary embodiments, the method further comprises:

[0090] When the electronic device does not have the perspective function enabled, obtain the frame rate recommended by the application;

[0091] When the frame rate recommended by the application is less than or equal to the third frequency f3, controlling the acquisition frame rate of the image acquisition device to be the frame rate recommended by the application;

[0092] When the frame rate recommended by the application is greater than the third frequency f3, the acquisition frame rate of the image acquisition device is controlled to be a sixth frequency, which is less than the frame rate recommended by the application.

[0093] In non-perspective mode, the gesture interaction process between different applications generally varies. In non-perspective mode, the processing device determines the capture frame rate of the image capture device to be the third frequency f3. However, the third frequency f3 may differ from the frame rate recommended by the application. The frame rate recommended by the application may be greater than, equal to, or less than the third frequency.

[0094] As shown in Figure 7, when the frame rate recommended by the application is less than the third frequency f3, the image acquisition device is controlled to operate at the frame rate recommended by the application, which reduces the processing load of the processing device and can determine the corresponding frame rate of the camera required for the gesture interaction process according to the specific application; for example, when the frame rates supported by the camera are 50, 60, and 70Hz (the highest frame rate supported by the camera is 70Hz), and the frame rate recommended by the application is 60Hz, the camera is controlled to operate at 60Hz at this time; when the frame rate recommended by the application is greater than the third frequency f3, the processing device determines the camera frame rate corresponding to the application based on the complexity of the application scenario, the CPU processing capability, and the screen refresh rate.

[0095] For example, assuming that the image acquisition device (such as a camera) supports acquisition frame rates of 50, 60, 70, 80, 90, and 100 Hz, and in one application, the application recommends an acquisition frame rate of 120 Hz, then the processing device determines the camera frame rate corresponding to the application based on the complexity of the application scenario, the CPU processing power, and the screen refresh rate. Assuming that the processing device determines that the appropriate frame rate for the camera is 90 Hz based on the complexity of the application scenario, the CPU processing power, and the screen refresh rate, then the camera is controlled to operate at a frame rate of 90 Hz. In another application, the application recommends a frame rate of 110 Hz, and the processing device determines that the appropriate frame rate for the camera is 100 Hz based on the complexity of the application scenario, the CPU processing power, and the screen refresh rate, then the camera is controlled to operate at a frame rate of 100 Hz.

[0096] In some exemplary embodiments, the method further comprises:

[0097] Obtaining a maximum screen display frame rate, and when the maximum screen display frame rate is greater than the frame rate recommended by the application, adjusting the acquisition frame rate of the image acquisition device to the frame rate recommended by the application;

[0098] When the maximum screen display frame rate is lower than the frame rate recommended by the application, data statistics are performed based on the amount of data collected by the image acquisition device and the amount of data to be displayed on the display screen, and the acquisition frame rate of the image acquisition device and the screen display frame rate are determined based on the statistical results.

[0099] For example, as shown in Figure 8, assuming the maximum screen display frame rate is N, when N is greater than the application-recommended frame rate M, it indicates that the system frame rate cannot render more data than the current display capacity. In this case, the image acquisition device's capture frame rate is adjusted to the application-recommended frame rate M (the screen display frame rate also uses the application-recommended frame rate M).

[0100] When the value N is less than the application's recommended frame rate M, the camera and display parameters are fed into the data statistics module. The module then calculates the optimal frame rate x that the system can currently execute, based on the total display and camera data volume. The data statistics module is designed to calculate the optimal frame rate based on the current CPU / GPU level and the total amount of data available for rendering. Once the optimal frame rate is determined, the processing device sends a frame rate switching command to the display, ensuring that the image acquisition device and display run at the same frame rate.

[0101] In some exemplary embodiments, the second frequency is generated by a user's selection based on the first selectable frame rate of the electronic device, or the second frequency is generated by a processing device based on the power frequency of the area where the electronic device is located, wherein the first selectable frame rate of the electronic device includes at least one second frequency recommended frame rate, and the second frequency recommended frame rate is consistent with the power frequency of the area where the electronic device is located.

[0102] In some exemplary embodiments, the third frequency is generated by a user's selection according to the second selectable frame rate of the electronic device, or the third frequency is generated by the processing device based on a load condition of the electronic device.

[0103] In the embodiment of the present disclosure, the user can manually set the third frequency and the second frequency of the image acquisition device, such as 50Hz / 90Hz, 50Hz / 72Hz, 50 / 120Hz, 60Hz / 90Hz, 60Hz / 72Hz, 60 / 120Hz, etc., wherein the number before the " / " represents the acquisition frame rate of the image acquisition device in the perspective mode, and the number after the " / " represents the acquisition frame rate of the image acquisition device in the non-perspective mode. For example, 50Hz / 90Hz means that the image acquisition device operates at an acquisition frame rate of 50Hz in the perspective mode and at an acquisition frame rate of 90Hz in the non-perspective mode; 60Hz / 90Hz means that the camera operates at an acquisition frame rate of 60Hz in the perspective mode and at an acquisition frame rate of 90Hz in the non-perspective mode; and so on.

[0104] In an embodiment of the present disclosure, the electronic device can recommend an acquisition frame rate in perspective mode based on the power frequency of the current region. For example, in China, when the user makes a selection, the three options of 50Hz / 90Hz, 50Hz / 72Hz, and 50 / 120Hz are set to gray, recommending that the user prioritize selecting an appropriate acquisition frame rate from these three options. When the electronic device moves to an area with a different power frequency, the acquisition frame rate recommended by the electronic device may change accordingly. For example, in the United States, the three options of 60Hz / 90Hz, 60Hz / 72Hz, and 60 / 120Hz are set to gray, recommending that the user prioritize selecting an appropriate acquisition frame rate from these three options. The above-mentioned second optional frame rate includes 90Hz, 72Hz, and 120Hz. Of course, this disclosure only uses these values ​​as examples for illustrative purposes. The electronic device can recommend other frequency values ​​as needed, or it can also generate a third frequency value based on its own load conditions (load conditions can include the electronic device's screen display capabilities, image acquisition capabilities, application-recommended frame rates, etc.).

[0105] An embodiment of the present disclosure further provides a processing device, which may include a processor and a memory storing a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the display method as described in any of the above items in the present disclosure are implemented.

[0106] As shown in FIG10 , in one example, a processing device may include: a processor 1010, a memory 1020, a bus system 1030, and a transceiver 1040. The processor 1010, the memory 1020, and the transceiver 1040 are connected via the bus system 1030. The memory 1020 is configured to store instructions, and the processor 1010 is configured to execute the instructions stored in the memory 1020 to control the transceiver 1040 to transmit signals. Specifically, under the control of the processor 1010, the transceiver 1040 may receive the luminous frequency of ambient light detected by the ambient light sensor. The processor 1010 detects whether the electronic device has a perspective function enabled. When the electronic device has a perspective function enabled, the processor 1010 controls the image acquisition device to have a capture frame rate of a second frequency, where the second frequency is an integer multiple of a first frequency, and the first frequency is the luminous frequency of the ambient light.

[0107] It should be understood that the processor 1010 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0108] The memory 1020 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1010. A portion of the memory 1020 may also include a non-volatile random access memory. For example, the memory 1020 may also store information about the device type.

[0109] In addition to the data bus, the bus system 1030 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the bus system 1030 in FIG.

[0110] During implementation, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the processor 1010 or by instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1020, and the processor 1010 reads the information in the memory 1020 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0111] The present disclosure also provides an electronic device comprising an ambient light sensor, a display screen, an image acquisition device, and a processing device. The processing device may be any of the processing devices described in the present disclosure. The ambient light sensor is configured to detect the luminous frequency of ambient light; the image acquisition device is configured to acquire an image of the environment and / or a hand in perspective mode; and to acquire an image of the hand in non-perspective mode, and transmit the acquired image to the processing device.

[0112] In the embodiment of the present disclosure, the electronic device may be a head-mounted display or other electronic device, and the embodiment of the present disclosure does not limit this.

[0113] In an exemplary embodiment, the head-mounted display further includes a head motion tracking sensor, which implements X, Y, Z axis and front and back side tracking.

[0114] The graphics card of an existing VR display device is responsible for rendering VR images, and the monitor is responsible for displaying the VR images rendered by the graphics card. The faster the graphics card's rendering speed, the higher the frame rate of the VR images, and the more VR images the graphics card transmits to the monitor. When the monitor's refresh rate is high, the monitor displays more VR images. Specifically, when the frame rate of the image is higher than the refresh rate, the frame rate output by the graphics card is higher than the refresh rate of the monitor, resulting in the monitor being unable to process the output image frames in a timely manner, causing tearing in the frames displayed on the monitor. At the same time, when the monitor is unable to process the output image frames in a timely manner, the VR images output by the graphics card cannot be fully displayed by the monitor, resulting in frame loss. For example, when the graphics card renders 100 images per second (100 frames per second), but the monitor can only display 60 images per second (the monitor's refresh rate is 60Hz), the user can only see 60 images rendered by the graphics card, and the remaining 40 images cannot be displayed by the monitor. This causes the graphics card to do useless work to a certain extent, resulting in frame loss in the images displayed on the monitor. When the frame rate of an image is lower than the refresh rate, the monitor not only fully displays the image rendered by the graphics card, but also displays at least two consecutive frames of the same image. This phenomenon is known as "image jitter." Therefore, when the monitor's refresh rate matches the frame rate of the VR image, the screen can display smooth VR images, giving users a superior visual experience.

[0115] To interact with certain scenes presented in the immersive environment provided by VR display devices, users interact with the VR display devices through head movements, ensuring a realistic user experience. Due to the limitations of the rendering capabilities of VR display devices' graphics processors, when the user's head moves rapidly and the frame rate is lower than the refresh rate in complex immersive environments, image jitter can occur, leading to image delays on the display. For example, when the user keeps their head still, image jitter does not cause discomfort. However, when the user rotates their head from an old position to a new one, image jitter can cause the current frame displayed on the display to still reflect the old position. At this point, the user's brain has switched to the new position, but the image information received by the eyes is still from the old position. This mismatch between the image information received by the eyes and the image information perceived by the brain can cause the user to experience dizziness. Furthermore, the greater the head rotation angle, the more severe this dizziness becomes.

[0116] Time warping (TW) is an image frame correction technique that warps the previous frame to obtain the current frame and transmits the resulting image to the display, thereby reducing image delays and dizziness caused by head motion. Generally, the warping method employed is a direction-based warping technique, which can correct for changes in head rotation. This warping technique is particularly effective for two-dimensional images and requires minimal system resources. For example, for complex scenes, direction-based warping can generate the current frame with minimal data processing. Specifically, before receiving the current frame's vertical synchronization signal, the graphics card must complete TW processing on the previous frame to obtain the current frame. This ensures that the display can display the current frame after receiving the current frame synchronization signal, avoiding image delays during head rotation. In other words, within the rendering time of a single frame, the current frame rendering operation (rendering operation) and the TW processing operation (TW operation) of the current frame must be completed sequentially to ensure that the display can display the TW-processed current frame in the next frame.

[0117] The image rendering process and the time warping process of the existing TW technology are carried out in the same thread, so that the image rendering process and the time warping process cannot be carried out at the same time. If the graphics card performance is relatively low, it cannot complete the current frame image rendering operation and the current frame image TW processing operation in sequence within one frame rendering time. People have improved the TW technology. The Asynchronous Time Warp (ATW) technology is an intermediate frame generation technology, which can not only reduce image delay and solve the problem of head dizziness. It can also complete the current frame image rendering operation and the current frame image TW processing operation within one frame rendering time when the graphics card performance is relatively low. Specifically, the ATW technology arranges the rendering operation and the TW operation in two threads to adapt to the graphics card performance, avoid image delay, and reduce image jitter; the thread where the rendering operation is located is defined as the rendering thread (MRT thread), and the thread where the ATW operation is located is defined as the asynchronous time warp thread (ATW thread).

[0118] In some exemplary embodiments, when the electronic device turns on the perspective function and displays the environment image and the virtual view simultaneously, the electronic device starts a camera thread, a rendering thread, and an asynchronous time warp thread, wherein:

[0119] At a first starting moment, the camera thread acquires image data acquired by an image acquisition device and sends the image data to the asynchronous time warp thread;

[0120] At a first starting time, the rendering thread renders the first screen scene data based on the first predicted user gesture data; at a first intermediate time, the rendered first screen scene data is sent to the asynchronous time warp thread, and the second screen scene data is rendered based on the second predicted user gesture data; at a first display time, the rendered second screen scene data is sent to the asynchronous time warp thread;

[0121] At the first intermediate moment, the asynchronous time warp thread performs a fusion operation on the first screen scene data and the image data and performs correction based on the third predicted user posture data to obtain first screen output data; at the first display moment, the first screen output data is output to the display screen for display, and a fusion operation on the second screen scene data and the image data is performed and correction is performed based on the third predicted user posture data to obtain second screen output data; at the second display moment, the second screen output data is output to the display screen for display, the first intermediate moment is between the first starting moment and the first display moment, and the first display moment is between the first intermediate moment and the second display moment.

[0122] Exemplarily, the first starting moment may be the starting moment of the current frame, the first intermediate moment may be the intermediate moment between the current frame and the next frame, the first display moment may be the starting moment of the next frame, and the second display moment may be the intermediate moment between the next frame and the frame after the next frame.

[0123] In the embodiment of the present disclosure, the first predicted user posture data is the user posture data at the first display moment predicted based on the user posture data at the first starting moment, the second predicted user posture data is the user posture data at the second display moment predicted based on the user posture data at the first intermediate moment, and the third predicted user posture data is the user posture data at the first display moment predicted based on the user posture data at the first intermediate moment.

[0124] In some exemplary embodiments, when the electronic device enables the perspective function and displays only the environment image, the electronic device starts a rendering thread and an asynchronous time warp thread, wherein:

[0125] At the second starting moment, the rendering thread obtains image data acquired by the image acquisition device and sends the image data to the asynchronous time warp thread;

[0126] At a second intermediate moment, the asynchronous time warp thread corrects the image data based on the fourth predicted user posture data; and outputs the corrected data to the display screen at a third display moment, where the second intermediate moment is between the second start moment and the third display moment.

[0127] Exemplarily, the second starting time may be the starting time of the current frame, the second intermediate time may be the intermediate time between the current frame and the next frame, and the third display time may be the starting time of the next frame.

[0128] In the embodiment of the present disclosure, the fourth predicted user posture data is the user posture data at the third display moment predicted based on the user posture data at the second intermediate moment.

[0129] In some exemplary embodiments, when the electronic device turns off the perspective function and displays only the virtual view, the electronic device starts a rendering thread and an asynchronous time warp thread, wherein:

[0130] At a third starting time, the rendering thread renders the first screen scene data based on the fifth predicted user gesture data; at a third intermediate time, the rendered first screen scene data is sent to the asynchronous time warp thread, and the second screen scene data is rendered based on the sixth predicted user gesture data; at a fourth display time, the rendered second screen scene data is sent to the asynchronous time warp thread;

[0131] At the third intermediate moment, the asynchronous time warp thread corrects the rendered first-screen scene data based on the seventh predicted user posture data to obtain first-screen output data; at the fourth display moment, the first-screen output data is output to the display screen for display, and the rendered second-screen scene data is corrected based on the seventh predicted user posture data to obtain second-screen output data; at the fifth display moment, the second-screen output data is output to the display screen for display. The third intermediate moment is between the third starting moment and the fourth display moment, and the fourth display moment is between the third intermediate moment and the fifth display moment.

[0132] Exemplarily, the third starting moment may be the starting moment of the current frame, the third intermediate moment may be the intermediate moment between the current frame and the next frame, the fourth display moment may be the starting moment of the next frame, and the fifth display moment may be the intermediate moment between the next frame and the frame after the next frame.

[0133] In the embodiment of the present disclosure, the fifth predicted user posture data is the user posture data at the fourth display moment predicted based on the user posture data at the third starting moment, the sixth predicted user posture data is the user posture data at the fifth display moment predicted based on the user posture data at the third intermediate moment, and the seventh predicted user posture data is the user posture data at the fourth display moment predicted based on the user posture data at the third intermediate moment.

[0134] In the embodiments of the present disclosure, the “first,” “second,” and “third” in the first starting moment, the second starting moment, and the third starting moment are merely for distinguishing different embodiments, and there is no temporal order relationship among the three. Similarly, the “first,” “second,” and “third” in the first intermediate moment, the second intermediate moment, and the third intermediate moment are merely for distinguishing different embodiments, and there is no temporal order relationship among the three. In addition, although the second display moment is located after the first display moment and the fifth display moment is located after the fourth display moment, the “first,” “third,” and “fourth” in the first display moment, the third display moment, and the fourth display moment are merely for distinguishing different embodiments, and there is no temporal order relationship among the three.

[0135] 12 to 14 show schematic diagrams of the image processing process based on ATW in an embodiment of the present disclosure, wherein VsyncN-1, VsyncN, VsyncN+1, and VsyncN+2 represent the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the N+1th frame, and the vertical synchronization signal of the N+2th frame, respectively. The moment of receiving the vertical synchronization signal of the N-1th frame is defined as the start moment of the N-1th frame, the moment of receiving the vertical synchronization signal of the Nth frame is defined as the end moment of the N-1th frame or the start moment of the Nth frame, the moment of receiving the vertical synchronization signal of the N+1th frame is defined as the end moment of the Nth frame or the start moment of the N+1th frame, and the moment of receiving the vertical synchronization signal of the N+2th frame is defined as the end moment of the N+1th frame or the start moment of the N+2th frame.

[0136] The middle moment between the start moment of the N-1th frame and the end moment of the N-1th frame is defined as the middle moment of the N-1th frame; the middle moment between the start moment of the Nth frame and the end moment of the Nth frame is defined as the middle moment of the Nth frame; the middle moment between the start moment of the N+1th frame and the end moment of the N+1th frame is defined as the middle moment of the N+1th frame.

[0137] Figure 12 is a schematic diagram of the image processing process when only the virtual view is displayed in non-perspective mode. As shown in Figure 12, at the start of the N-1th frame (or the start of the Nth frame), the rendering thread renders the first screen scene data based on the user's posture data at the time of display (i.e., 1.5 frames later). At the middle of the N-1th frame (or the middle of the Nth frame), the rendered first screen scene data is sent to the asynchronous time warp thread. Simultaneously, the second screen scene data is rendered based on the user's posture data at the time of display (i.e., 1.5 frames later). At the start of the Nth frame (or the start of the N+1th frame), the rendered second screen scene data is sent to the asynchronous time warp thread.

[0138] After the asynchronous time warp thread receives the rendered first-screen scene data at the middle of the N-1th frame (or the middle of the Nth frame), it corrects the rendered first-screen scene data based on the user posture data at the display time (that is, 1 frame later) to obtain the first-screen scene output data. After the correction is completed, the first-screen scene output data is output to the display screen at the end of the N-1th frame (or the end of the Nth frame); after receiving the rendered second-screen scene data at the end of the N-1th frame (or the end of the Nth frame), it corrects the rendered second-screen scene data based on the user posture data at the display time (that is, 1 frame later) to obtain the second-screen scene output data. After the correction is completed, the second-screen scene output data is output to the display screen for display.

[0139] Optionally, the first screen may be a screen that displays the image seen by the user's left eye, and the second screen may be a screen that displays the image seen by the user's right eye. The first screen scene data may include various parameters of the objects that should be presented in the field of view of the user's left eye, and the second screen scene data may include various parameters of the objects that should be presented in the field of view of the user's right eye. As described above, the meanings of the first screen and the second screen are interchangeable, and the first screen and the second screen may be various types of display screens, such as liquid crystal displays (LCDs), organic light-emitting diode displays (OLEDs), micro-light-emitting diode (Micro-LED) displays, etc. These display devices refresh a frame of displayed image, for example, by progressive scanning or the like.

[0140] In order to enable the user to use the virtual reality device, the image presented by the virtual reality device can be all objects that the user can observe in the field of view, generally all objects within the user's field of view need to be rendered. At the same time, different images need to be presented to the user's left eye and right eye (that is, parallax is formed), and then the user's brain synthesizes the different images of the left and right eyes, thereby presenting a visually three-dimensional image to the user. Therefore, the parameters for the same object in the first screen scene data and the second screen scene data may also be different. For example, for the same object, the position of the object observed by the left eye and the right eye, and the situation of light refraction through the object to the left and right eyes may also be different. Therefore, in the present disclosure, the MRT thread renders both the first screen scene data and the second screen scene data. In this way, compared to first rendering the object as a whole and then outputting the rendering output results to the first screen and the second screen respectively through affine transformation, a more accurate and more three-dimensional image can be obtained.

[0141] Optionally, the rendering of the first screen scene data and the second screen scene data is multi-render target rendering.

[0142] Multi-render target rendering technology can save the data of multiple pixels into different buffers (for example, a buffer for color mapping, a buffer for normal mapping, and a buffer for depth mapping), so that these data become parameters of subsequent lighting effect shaders, so that the fineness of the output image can be improved. Through multi-render target rendering technology, the rendering of lighting information can be delayed, and the rendering processing speed can be improved. At the same time, the MRT thread renders the first screen scene data and the second screen scene data respectively, so it is necessary to save the pixel data of each object in the first screen scene data and the second screen scene data respectively to the above-mentioned buffer area, and then parse the above data in the lighting effect shading, thereby enhancing the lighting effect of the image displayed on the first screen and the second screen. Compared with the usual use of lighting shaders to perform lighting shading calculations for each pixel in the entire scene, in the embodiment of the present disclosure, using MRT threads to perform multi-render target rendering on the first screen scene data and the second screen scene data respectively can reduce the number of operations of the processor for lighting shading calculations, and can obtain an image with enhanced lighting effects.

[0143] After completing the rendering of the first and second screen scene data, the rendering thread triggers the asynchronous time warp thread (ATW thread) and sends the rendered first and second screen scene data to the ATW thread through thread communication with the ATW thread. The rendering thread then waits for the next vertical synchronization signal (the second vertical synchronization signal) to begin the next scene data rendering cycle.

[0144] It is understood that when rendering a VR image, it is necessary to render the VR image according to the VR image's rendering parameters. VR image rendering parameters include rendering resolution, field of view, and user posture information. The field of view and user posture information enable texture rendering of the VR image, especially for VR images with complex scenes. Scene complexity and rendering resolution play a decisive role in VR image rendering time. When performing TW processing, the image information needs to be processed according to the user posture information to achieve time warping of the image information.

[0145] Figure 13 is a schematic diagram of the image processing process when only the environment image is displayed in perspective mode. As shown in Figure 13, at the start of the N-1th frame (or the start of the Nth frame), the rendering thread obtains the image data collected by the image acquisition device and sends it to the asynchronous time warp thread.

[0146] As shown in Figure 13, at the middle of frame N-1 (or frame N), the asynchronous time warp thread corrects the received camera image data based on the user pose data at the display time (i.e., one frame later) to obtain corrected camera image data. At the start of frame N (or frame N+1), after receiving the vertical synchronization signal for frame N (or frame N+1), the asynchronous time warp thread outputs the corrected camera image data to the display screen.

[0147] Figure 14 is a schematic diagram of the image processing process when displaying both an environment image and a virtual view in perspective mode. As shown in Figure 14, at the start of frame N-1 (or the start of frame N), the camera thread obtains image data captured by the image acquisition device and sends it to the asynchronous time warp thread. The rendering thread renders the first screen scene data based on the user's posture data at the time of display (i.e., 1.5 frames later). At the middle of frame N-1 (or the middle of frame N), the rendered first screen scene data is sent to the asynchronous time warp thread. Simultaneously, the second screen scene data is rendered based on the user's posture data at the time of display (i.e., 1.5 frames later). The rendered second screen scene data is sent to the asynchronous time warp thread at the start of frame N (or the start of frame N+1).

[0148] After receiving the rendered first-screen scene data and the camera image data at the middle of the N-1th frame (or the middle of the Nth frame), the asynchronous time warp thread performs a dual-image superposition and fusion operation on the first-screen scene data and the camera image data to obtain fused first-screen scene data. The fused first-screen scene data is corrected based on the user posture data at the display time (i.e., one frame later) to obtain first-screen scene output data. After the correction is completed, the first-screen scene output data is output to the display screen at the end of the N-1th frame (or the end of the Nth frame). After receiving the rendered second-screen scene data at the end of the N-1th frame (or the end of the Nth frame), the asynchronous time warp thread performs a dual-image superposition and fusion operation on the second-screen scene data and the camera image data to obtain fused second-screen scene data. The fused second-screen scene data is corrected based on the user posture data at the display time (i.e., one frame later) to obtain second-screen scene output data. After the correction is completed, the second-screen scene output data is output to the display screen at the middle of the Nth frame (or the middle of the N+1th frame).

[0149] The present disclosure also provides a computer-readable storage medium storing executable instructions. When executed by a processor, the executable instructions can implement the display method provided in any of the above embodiments of the present disclosure. The display method can be used to control the electronic device provided in the above embodiments of the present disclosure to perform display, thereby resolving the flickering problem caused by interference between the luminous frequency of ambient light and the camera acquisition frame rate, and improving the user experience. The method of driving an electronic device to perform display by executing executable instructions is basically the same as the display method provided in the above embodiments of the present disclosure and will not be described in detail here.

[0150] In the description of the embodiments of the present disclosure, it should be understood that the terms "middle", "upper", "lower", "front", "back", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0151] In the description of the embodiments of the present disclosure, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. A person of ordinary skill in the art will be able to understand the meanings of the above terms in the present disclosure.

[0152] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0153] Although the embodiments disclosed in this disclosure are as described above, the contents described are merely embodiments adopted to facilitate understanding of the disclosure and are not intended to limit the disclosure. Any person skilled in the art to which the disclosure belongs may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope of the disclosure. However, the scope of protection of the disclosure shall still be based on the scope defined by the appended claims.

Claims

1. A display method for an electronic device, the electronic device including an image acquisition device, the method comprising: Detecting whether the electronic device enables a perspective function; When the electronic device enables the perspective function, controlling the acquisition frame rate of the image acquisition device to be a second frequency, where the second frequency is an integer multiple of a first frequency, and the first frequency is the emission frequency of ambient light.

2. The display method according to claim 1, wherein Before the step of controlling the acquisition frame rate of the image acquisition device to be the second frequency, the method further comprises: Detecting the emission frequency of ambient light; When the emission frequency of the ambient light is the first frequency, triggering the step of controlling the acquisition frame rate of the image acquisition device to be the second frequency.

3. The display method according to claim 1 or 2, wherein The detecting whether the electronic device enables the perspective function includes any one or more of the following: Detecting whether an instruction to enable the perspective function is received, detecting whether the user reaches a safety boundary, and detecting whether the distance between an external object and the user is less than or equal to a preset distance threshold.

4. The display method according to claim 1 or 2, the method further comprising: Obtaining the screen display frame rate of the electronic device; Detecting whether the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within a preset ratio range; When the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is not within the preset ratio range, adjusting the screen display frame rate so that the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within the preset ratio range.

5. The display method according to claim 1 or 2, the method further comprising: Detecting whether the electronic device disables the perspective function; When the electronic device disables the perspective function, detecting whether an instruction to enable a gesture recognition function is received; When the instruction to enable the gesture recognition function is received, controlling the acquisition frame rate of the image acquisition device to be a third frequency, and the third frequency is greater than the second frequency.

6. The display method according to claim 1 or 2, the method further comprising: Detecting whether the emission frequency of ambient light changes; When the emission frequency of the ambient light changes, controlling the acquisition frame rate of the image acquisition device to be a fourth frequency, and the fourth frequency is greater than the second frequency, and detecting whether the ratio of the screen display frame rate of the electronic device to the acquisition frame rate of the image acquisition device is within a preset ratio range; When the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is not within the preset ratio range, adjusting the screen display frame rate so that the ratio of the screen display frame rate to the acquisition frame rate of the image acquisition device is within the preset ratio range.

7. The display method according to claim 1 or 2, the method further comprising: When the electronic device enables the perspective function and the emission frequency of the ambient light is not the first frequency, controlling the acquisition frame rate of the image acquisition device to be a fourth frequency, and the fourth frequency is greater than the second frequency.

8. The display method according to claim 1 or 2, the method further comprising: When the electronic device has enabled the perspective function, detecting whether the head movement speed is greater than or equal to a preset movement threshold; When the head movement speed is greater than or equal to a preset movement threshold, control the acquisition frame rate of the image acquisition device to be a fifth frequency, and the fifth frequency is greater than the second frequency; When the head movement speed is less than the preset movement threshold, trigger the step of detecting the emission frequency of the ambient light.

9. The display method according to claim 1 or 2, the method further comprising: When the electronic device does not turn on the perspective function, control the acquisition frame rate of the image acquisition device to be a third frequency, and the third frequency is greater than the second frequency.

10. The display method according to claim 9, the method further comprising: When the electronic device does not turn on the perspective function, obtain the frame rate recommended by the application; When the frame rate recommended by the application is less than or equal to the third frequency, control the acquisition frame rate of the image acquisition device to be the frame rate recommended by the application; When the frame rate recommended by the application is greater than the third frequency, control the acquisition frame rate of the image acquisition device to be a sixth frequency, and the sixth frequency is less than the frame rate recommended by the application.

11. The display method according to claim 10, the method further comprising: Obtain the maximum screen display frame rate of the electronic device, and detect the magnitude relationship between the maximum screen display frame rate and the frame rate recommended by the application; When the maximum screen display frame rate is greater than the frame rate recommended by the application, adjust the acquisition frame rate of the image acquisition device to be consistent with the frame rate recommended by the application; When the maximum screen display frame rate is less than the frame rate recommended by the application, perform data statistics based on the amount of data collected by the image acquisition device and the amount of data to be displayed on the display screen of the electronic device, and determine the acquisition frame rate of the image acquisition device and the screen display frame rate according to the statistical result.

12. The display method according to claim 9, wherein, The second frequency is generated by the user according to the first selectable frame rate of the electronic device, or the second frequency is generated by the electronic device based on the power frequency of the region where the electronic device is located, wherein the first selectable frame rate includes at least one second frequency recommended frame rate, and the second frequency recommended frame rate is consistent with the power frequency of the region where the electronic device is located; The third frequency is generated by the user according to the second selectable frame rate of the electronic device, or the third frequency is generated by the electronic device based on the load condition of the electronic device.

13. The display method according to claim 1 or 2, wherein, When the electronic device turns on the perspective function and simultaneously displays the ambient image and the virtual view, the electronic device starts the camera thread, the rendering thread, and the asynchronous time warp thread, and the method further comprises: At a first start time, the camera thread obtains the image data collected by the image acquisition device and sends it to the asynchronous time warp thread; At the first start time, the rendering thread renders the first screen scene data based on the first predicted user pose data; sends the rendered first screen scene data to the asynchronous time warp thread at a first intermediate time, and renders the second screen scene data based on the second predicted user pose data; sends the rendered second screen scene data to the asynchronous time warp thread at a first display time; At the first intermediate moment, the asynchronous time warp thread performs a fusion operation on the first screen scene data and the image data and performs correction based on the third predicted user pose data to obtain the first screen output data; at the first display moment, the first screen output data is output to the display screen for display, and a fusion operation on the second screen scene data and the image data and correction based on the third predicted user pose data are performed to obtain the second screen output data; at the second display moment, the second screen output data is output to the display screen for display, the first intermediate moment is between the first starting moment and the first display moment, and the first display moment is between the first intermediate moment and the second display moment.

14. The display method according to claim 1 or 2, wherein When the electronic device turns on the perspective function and only displays the environmental image, the electronic device starts the rendering thread and the asynchronous time warp thread, and the method further includes: At the second starting moment, the rendering thread acquires the image data collected by the image acquisition device and sends it to the asynchronous time warp thread; At the second intermediate moment, the asynchronous time warp thread corrects the image data based on the fourth predicted user pose data; at the third display moment, the corrected data is output to the display screen for display, and the second intermediate moment is between the second starting moment and the third display moment.

15. The display method according to claim 1 or 2, wherein When the electronic device turns off the perspective function and only displays the virtual view, the electronic device starts the rendering thread and the asynchronous time warp thread, and the method further includes: At the third starting moment, the rendering thread renders the first screen scene data based on the fifth predicted user pose data; at the third intermediate moment, the rendered first screen scene data is sent to the asynchronous time warp thread, and the second screen scene data is rendered based on the sixth predicted user pose data; at the fourth display moment, the rendered second screen scene data is sent to the asynchronous time warp thread; At the third intermediate moment, the asynchronous time warp thread corrects the rendered first screen scene data based on the seventh predicted user pose data to obtain the first screen output data; at the fourth display moment, the first screen output data is output to the display screen for display, and the rendered second screen scene data is corrected based on the seventh predicted user pose data to obtain the second screen output data; at the fifth display moment, the second screen output data is output to the display screen for display, the third intermediate moment is between the third starting moment and the fourth display moment, and the fourth display moment is between the third intermediate moment and the fifth display moment.

16. A processing device, comprising: A processor and a memory storing a computer program that can run on the processor, wherein when the processor executes the computer program, the steps of the display method according to any one of claims 1 to 15 are implemented.

17. An electronic device, comprising: An image acquisition device and a processing device according to claim 16.

18. A computer-readable storage medium storing computer-executable instructions for executing the display method according to any one of claims 1 to 15.

Images