Display method and electronic device

Abstract

A display method and an electronic device are used for reducing image rendering power consumption. The method comprises: presenting N frames of images to a user by a display device; wherein a first object at a first depth of field on a jth frame of the N frames of images is the same as a first object at the first depth of field on an ith frame of the N frames of images; a second object at a second depth of field on the jth frame of the N frames of images is different from a second object at the second depth of field on the ith frame of the N frames of images; N, i, and j are positive integers, and i is less than j.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronics, and in particular to a display method and an electronic device. BACKGROUND

[0002] Virtual Reality (VR) technology is a kind of human-computer interaction means created by means of computers and sensor technology. The VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other scientific technologies, and can create a virtual environment. A user can immerse in the virtual environment by wearing a VR wearable device.

[0003] The virtual environment is presented by constantly refreshing a plurality of rendered three-dimensional images. The three-dimensional images include objects at different depths of field, which can bring a user a stereoscopic effect. Generally, the higher the image rendering frame rate (the number of frames of images rendered per unit time), the better. However, due to the computing power of a Graphics Processing Unit (GPU), the power consumption of a device and other reasons, it is often difficult to provide a large image rendering frame rate. SUMMARY

[0004] The present application aims to provide a display method and an electronic device for reducing power consumption caused by image rendering.

[0005] In a first aspect, a display method is provided, which can be executed by a display device. The display device can be a VR display device, an Augmented Reality (AR) display device, a Mixed Reality (MR) display device, and can be a wearable device such as a head-mounted device (e.g., an eye, a helmet, etc.). Alternatively, the method can also be executed by an electronic device connected to the display device, such as a host (e.g., a VR host) or a server (e.g., a VR server), etc. In the method, N frames of images are presented to a user by the display device. In the N frames of images, a first object at a first depth of field in a jth frame of image is the same as a first object at the first depth of field in an ith frame of image, and a second object at a second depth of field in the jth frame of image is different from a second object at the second depth of field in the ith frame of image. N, i and j are positive integers, and i is less than j.

[0006] For example, when a user wears a display device (e.g., VR glasses), the user can see a virtual environment, which is presented by constantly refreshing images (three-dimensional images). Therefore, the objects seen by the user have a depth of field, for example, the user can see some objects close to the user and some objects far from the user. In an embodiment of the present application, the first object at the first depth of field in the jth image and the ith image is the same, and the second object at the second depth of field is different. Correspondingly, during the continuous playing of the ith image and the jth image, the first object at the first depth of field appears the same (or unchanged) to the user, and the second object at the second depth of field appears different (or changes). In this way, the first object at the first depth of field can be rendered at a lower frame rate, for example, only one frame of the first object is rendered, and the second frame, the third frame, and the like use this frame of the first object, thereby greatly saving rendering power consumption.

[0007] For example, the first object and the second object are both changing objects. A changing object can be understood as a first object and a second object that appear to change to the user, such as at least one of an action, a position, a shape, a color, or a size.

[0008] For example, the first object at the first depth of field is a boy kicking a ball, and the second object at the second depth of field is a ship on the sea. Both the boy and the ship are changing objects. Through the technical solution of the embodiment of the present application, the user sees the boy changing constantly, while the ship on the sea does not change or changes slowly. It can be simply understood that the boy changes in real time, while the ship changes slowly or does not change.

[0009] In a possible design, the first depth of field is greater than the second depth of field. For example, the first depth of field is greater than a first threshold value, and / or the second depth of field is less than a second threshold value, and the first threshold value is greater than or equal to the second threshold value. The specific values of the first threshold value and the second threshold value are not limited in the embodiment of the present application.

[0010] That is, in the virtual environment seen by the user, the first object (which can be understood as a distant object) far from the user does not change, and the second object (which can be understood as a close object) close to the user changes. Generally, the user often pays more attention to the object close to the user. Therefore, in the embodiment of the present application, the close object changes in real time, and the distant object changes less or can not change. In this way, the user's viewing experience is not affected, and the rendering power consumption is also saved.

[0011] In a possible design, the second depth of field changes when the depth of field of the user's gaze point changes. In other words, the second depth of field changes along with the change of the depth of field of the user's gaze point. For example, when the user's gaze point changes from far to near (for example, from 10 m to 1 m), the second depth of field also changes from far to near. In this way, the second object at the second depth of field changes gradually faster during the process of the user's gaze changing from far to near, avoiding the object corresponding to the user's focus point being unchanged, which affects the user's viewing experience. Specifically, the implementation manner of the background is that the image rendering frame rate of the second object at the second depth of field is increased, so that the number of frames inserted by the second object is reduced, and the second object appears to change faster.

[0012] For example, the second depth of field is the depth of field where the user's gaze point is located. That is, if the user's gaze point is located at which depth of field, the object at the depth of field seen by the user can change in real time, and the object at other depth of field (for example, the first depth of field) can be unchanged or change less.

[0013] For example, the second depth of field can be the depth of field where a preset object is located, and the preset object can be one or more of a virtual object, a display object or an interface. The preset object can be a system default setting or a user setting.

[0014] In some embodiments, the first object at the first depth of field on the jth image in the N images is the same as the first object at the first depth of field on the ith image, including that at least one of the action, position, shape, color or size of the first object on the jth image and the ith image is the same; and the second object at the second depth of field on the jth image is different from the second object at the second depth of field on the ith image, including that at least one of the action, position, shape, color or size of the second object on the jth image and the ith image is different.

[0015] That is, when the user wears a display device (such as a VR glasses), the user can see that in the virtual environment, the first object at the first depth of field is unchanged (such as at least one of the action, position, shape or size being the same), and the second object at the second depth of field changes (such as at least one of the action, position, shape or size being different).

[0016] In a possible design, the first object and the second object are of different types.

[0017] For example, the first object includes one type or multiple types of a virtual object, a display object or an interface; and / or, the second object includes one type or multiple types of a virtual object, a display object or an interface.

[0018] For example, the first object can be a virtual object (such as a VR game character), and the second object is a real object, which refers to an object in the real world captured by a camera. That is, the user sees that the virtual object is included in the real world, wherein the virtual object is real-time changing, and the real world changes slower or even does not change.

[0019] For another example, the first object can be an interface (such as a video playing interface), and the second object can be a background object, such as a virtual cinema, etc. In this way, the user sees that a movie is watched in the virtual cinema. Specifically, the movie is real-time changing, and the virtual application changes slower or even does not change.

[0020] In a possible design, the i-th frame image can be a previous frame image of the j-th frame image, i.e., i = j-1; or the i-th frame image can be a previous n frame image of the j-th frame image, i.e., i = j-n, n > 1, which is not limited by the embodiments of the present application.

[0021] In a possible design, the first object at the first depth of field on the j-th frame image is the same as the first object at the first depth of field on the i-th frame image, including: the first object at the first depth of field on the j-th frame image is a copy of the first object at the first depth of field on the i-th frame image; or the first object at the first depth of field on the j-th frame image is an object after the first object at the first depth of field on the i-th frame image is translated and / or rotated.

[0022] In this way, the first object at the first depth of field on the j-th frame image does not need to be re-rendered, and the first object at the first depth of field on the i-th frame image can be directly used. For example, copying the first object at the first depth of field on the i-th frame image or translating and / or rotating the first object at the first depth of field on the i-th frame image helps to save rendering power consumption.

[0023] In a possible design, the second object at the second depth of field on the j-th frame image is different from the second object at the second depth of field on the i-th frame image, including: the second object at the second depth of field on the j-th frame image is a different object from the second object at the second depth of field on the i-th frame image; and / or the second object at the second depth of field on the j-th frame image is a different form of the second object at the second depth of field on the i-th frame image.

[0024] As described above, in the process of playing N frames of images, the first object at the first depth is the same (or unchanged) and the second object at the second depth is different (or changed) in the view of the user. For example, the object at the second depth in the current frame and the previous frame is changed, that is, a new object enters the second depth of the virtual environment, or the second object at the second depth changes in form, which includes the action, position, shape, size, color, etc. of the second object. In summary, the second object at the second depth changes in real time in the view of the user, and the user has a better viewing experience.

[0025] In a possible design, before the N frames of images are presented to the user by the display device, the method further includes: generating M frames of first object images and N frames of second object images in a time period, M and N are positive integers, and M is less than N; inserting N-M frames of first object images in the M frames of first object images; wherein the inserted N-M frames of first object images are copied from at least one frame of the M frames of first object images or are images obtained by rotating and / or translating the at least one frame of first object images; and fusing the N frames of first object images and the N frames of second object images to obtain the N frames of images.

[0026] It should be understood that M < N, so that N-M frames of first object images need to be inserted. Optionally, the inserted first object images can be copied from the previous frame or the previous frame after rotation and / or translation. For example, M = 3 and N = 6, so that one frame is inserted every other frame in the three frames of first object images, and the inserted one frame can be copied from the previous frame or the previous frame after rotation and / or translation. Alternatively, the inserted first object images can be copied from the previous n frames or the previous n frames after rotation and / or translation, which is not limited in the embodiments of the present application.

[0027] In a possible design, the inserting N-M frames of first object images in the M frames of first object images includes: corresponding the M frames of second object images in the N frames of second object images to the M frames of first object images, the M frames of second object images are adjacent in generation time to the M frames of first object images; and inserting N-M frames of first object images, wherein the inserted N-M frames of first object images correspond to the remaining N-M frames of second object images in the N frames of second object images.

[0028] The M frames of second object images are adjacent in generation time to the M frames of first object images, which can be understood as close in generation time, closest in generation time, or the smallest or less than a threshold in time difference between generation times. It should be understood that M < N, so that N-M frames of first object images need to be inserted. Before the N-M frames of first object images are inserted, the M frames of first object images and the N frames of second object images are aligned according to the generation time, and after the alignment, the N-M frames of first object images are inserted in the empty space. For details, please refer to the following description.

[0029] In a possible design, the M frames of first object images are images of the first object rendered according to poses of the display device at M time instants, and the N frames of second object images are images of the second object rendered according to poses of the display device at N time instants, where the M time instants and the N time instants are within the first time length.

[0030] In an embodiment of the present application, the image rendering frame rate of the first object at the first depth of field is different from that of the second object at the second depth of field. The image rendering frame rate refers to the number of frames of images rendered in a unit of time. Assuming that the rendering frame rate of the first object is M and that of the second object is N, then in a certain time length (for example, a unit of time), M frames of first object images and N frames of second object images are rendered. Taking the rendering of the first object as an example, when the user moves the head, the pose of the VR glasses changes, and the first object is rendered based on the pose of the VR glasses, so that the rendered first object adapts to the movement of the user's head, and the user experience is better.

[0031] In a possible design, the display device presents N frames of images to the user, including: in the case where N is less than the image refresh rate P of the display device, inserting N-P frames of the images into the N frames of images; wherein the inserted N-P frames of images are at least one frame of image copied from the N frames of images or an image obtained by rotating and / or translating at least one frame of image; and the display device presents P frames of images to the user, where P is a positive integer.

[0032] For example, P=90 and N=60, then 30 frames need to be inserted, and the inserted 30 frames can be any one or multiple of the 60 frames, for example, the inserted images can be the image copied from the previous frame or an image obtained by rotating and / or translating the previous frame. Alternatively, the inserted images can be the image copied from the previous n frames or an image obtained by rotating and / or translating the previous n frames, which is not limited in the embodiments of the present application.

[0033] In a possible design, the method further includes: when the user focuses on the first object at the first depth of field, displaying W frames of images by the display device; wherein the object at the second depth of field in the t th frame of image is the same as that in the r th frame of image, and the object at the first depth of field in the t th frame of image is different from that in the r th frame of image, where N, t, and r are positive integers, and r is less than t.

[0034] That is, originally, the first object at the first depth of field is the same (or unchanged) and the second object at the second depth of field is different (or changed) in the view of the user. When the user focuses on the first object at the first depth of field, the first object at the first depth of field is different (or changed) and the second object at the second depth of field is the same (or unchanged). This is because, originally, the image rendering frame rate of the first object at the first depth of field is low, so the interpolation is more, and the change is not obvious or not changed. When the user focuses on the first object at the first depth of field, the image rendering frame rate of the first object is increased, so the interpolation is reduced, and the change speed is accelerated. In order to save power consumption, when the image rendering frame rate of the first object is increased, the image rendering frame rate of the second object is reduced, so the second object at the second depth of field is unchanged or changes slowly.

[0035] The second aspect also provides an electronic device, comprising:

[0036] a processor, a memory, and one or more programs;

[0037] The one or more programs are stored in the memory, and the one or more programs include instructions which, when executed by the processor, cause the electronic device to perform the method steps provided in the first aspect.

[0038] The third aspect provides a computer readable storage medium for storing a computer program, which, when running on a computer, causes the computer to perform the method provided in the first aspect.

[0039] The fourth aspect provides a computer program product comprising a computer program, which, when running on a computer, causes the computer to perform the method provided in the first aspect.

[0040] The fifth aspect provides a graphical user interface on an electronic device, the electronic device having a display screen, a memory, and a processor for executing one or more computer programs stored in the memory, the graphical user interface comprising a graphical user interface displayed when the electronic device performs the method provided in the first aspect.

[0041] The sixth aspect also provides a chip system coupled with the memory in the electronic device, for calling the computer program stored in the memory and executing the technical solution of the first aspect of the embodiments of the present application. In the embodiments of the present application, "coupling" means that two components are directly or indirectly combined with each other.

[0042] The beneficial effects of the above-mentioned second to sixth aspects are described in the beneficial effects of the first aspect, and are not repeated. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A schematic diagram of a system architecture provided by an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a virtual environment as seen by a wearable device when the posture of the wearable device changes, provided by an embodiment of the present application;

[0045] Figure 3 A schematic diagram of an image rendering method provided by an embodiment of the present application;

[0046] Figure 4 A schematic diagram of another image rendering method provided by an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a response delay caused by low rendering frame rate rendering, provided by an embodiment of the present application;

[0048] Figure 6 A schematic diagram of image translation, provided by an embodiment of the present application;

[0049] Figure 7 A schematic diagram of a first application scenario, provided by an embodiment of the present application;

[0050] Figure 8 A schematic diagram of a second application scenario, provided by an embodiment of the present application;

[0051] Figure 9 A schematic diagram of a third application scenario, provided by an embodiment of the present application;

[0052] Figure 10 A structural schematic diagram of a wearable device, provided by an embodiment of the present application;

[0053] Figure 11 A flowchart of an image rendering method provided by an embodiment of the present application;

[0054] Figure 12 A schematic diagram of a close-range object and a far-range object rendered at different frame rates, provided by an embodiment of the present application;

[0055] Figure 13A and Figure 13B A schematic diagram of a processing flow of a close-range object and a far-range object, provided by an embodiment of the present application;

[0056] Figure 14A and Figure 14B A schematic diagram of alignment of a close-range object and a far-range object, provided by an embodiment of the present application;

[0057] Figures 15A to 15C A schematic diagram of an interpolation process, provided by an embodiment of the present application;

[0058] Figure 16A and Figure 16B A schematic diagram of a processing flow of a close-range object, a mid-range object and a far-range object provided by an embodiment of the present application is shown in FIG. 1.

[0059] Figures 17 to 20 A schematic diagram of an interpolation process provided by an embodiment of the present application is shown in FIG. 2.

[0060] Figure 21 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0061] In the following, some terms in the embodiments of the present application are explained to facilitate the understanding of the skilled in the art.

[0062] (1) At least one involved in the embodiments of the present application includes one or more; wherein, more than or equal to two means. In addition, it should be understood that in the description of the present application, "first", "second" and the like are used only for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating or implying order. For example, the first object and the second object do not represent the importance of the two, or represent the order of the two, which is to distinguish the objects.

[0063] In the embodiments of the present application, "and / or" is a description of the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0064] (2) Virtual reality (VR) technology is a kind of man-machine interaction means created by means of computer and sensor technology. VR technology integrates computer graphics technology, computer simulation technology, sensor technology, display technology and other scientific technologies, and can create a virtual environment. The virtual environment includes three-dimensional realistic images generated by the computer and played in real time, which brings visual perception to the user; in addition to the visual perception generated by the computer graphics technology, there are auditory, tactile, force, motion and other perceptions, even including olfactory and gustatory, also known as multi-perception; in addition, the head rotation of the user, eyes, gestures, or other human behavior actions can be detected, and the data corresponding to the user's actions can be processed by the computer, and the user's actions can be responded in real time, and feedback to the user's five senses, and then form a virtual environment. For example, the user wearing a VR wearable device can see the VR game interface, and can interact with the VR game interface through gestures, handles, etc., as if he is in the game.

[0065] (3) Augmented Reality (AR) technology refers to superimposing computer-generated virtual objects onto a real-world scene, thereby achieving augmentation of the real world. That is, in AR technology, the real-world scene needs to be captured, and then a virtual environment is added to the real world.

[0066] Therefore, the difference between VR technology and AR technology is that AR technology creates a complete virtual environment, and the user sees only virtual objects; while AR technology superimposes virtual objects on the real world, that is, it includes both real-world objects and virtual objects. For example, the user wears transparent glasses through which the user can see the real environment around him, and the glasses can also display virtual objects, so the user can see both real objects and virtual objects.

[0067] (4) Mixed Reality (MR) technology is a bridge between virtual environment, real world and user through introducing real scene information (or real scene information) into virtual environment, thereby enhancing the realism of user experience. Specifically, the real object is virtualized (for example, using a camera to scan the real object for three-dimensional reconstruction to generate a virtual object), and the virtualized real object is introduced into the virtual environment, so that the user can see the real object in the virtual environment.

[0068] It should be noted that the technical solutions provided by the embodiments of the present application can be applied to VR scenarios, AR scenarios or MR scenarios.

[0069] Of course, in addition to VR, AR and MR, it can also be applied to other scenarios. For example, naked eye 3D scene (naked eye 3D display screen, naked eye 3D projection, etc.), cinema (such as 3D movie), VR software in electronic devices, etc. In short, it can be applied to any scenario that needs to generate a three-dimensional image, wherein the three-dimensional image includes objects located at different depths (or image depths).

[0070] For convenience of description, the following mainly takes VR scenarios as an example for introduction.

[0071] For example, please refer to Figure 1Fig. 1 is a schematic diagram of a VR system according to an embodiment of the present application. The VR system includes a VR wearable device and a host (e.g., a VR host) or a server (e.g., a VR server). The VR wearable device is connected (wired or wireless) to the VR host or the VR server. The VR host or the VR server can be a device with large computing power. For example, the VR host can be a mobile phone, a tablet computer, a notebook computer, etc., and the VR server can be a cloud server, etc. The VR host or the VR server is responsible for image generation, image rendering, etc., and then sends the rendered image to the VR wearable device for display. The user wearing the VR wearable device can see the image. For example, the VR wearable device can be a head-mounted device (HMD), such as glasses, a helmet, etc.

[0072] For this VR architecture, the VR wearable device, the VR host or the VR server can use the rendering method provided by the present application (the specific principle will be described later) to render the image, so as to save the rendering power consumption of the VR host or the VR server. Optionally, Figure 1 The VR system can also not include the VR host or the VR server. For example, the VR wearable device has the capability of local image generation and rendering, and does not need to obtain the image from the VR host or the VR server for display. In this case, the VR wearable device can use the rendering method provided by the present application to render the image, so as to save the rendering power consumption of the VR wearable device.

[0073] Hereinafter, the local image rendering of the VR wearable device will be mainly introduced as an example.

[0074] (5) Image rendering

[0075] It can be understood that when a user wears a VR wearable device, behaviors such as position movement and head turning may occur. In order to make the virtual environment more realistic, when the VR wearable device moves in position and turns the head, the image needs to be processed accordingly to give the user a real feeling. Therefore, in the VR field, image rendering includes rendering of color, transparency, etc. of the image, and also includes rotating and / or translating the image according to the pose of the VR wearable device. The pose of the VR wearable device includes multiple degrees of freedom such as rotation angle and / or translation distance, wherein the selection angle includes yaw angle, pitch angle, and roll angle, and the translation distance includes translation distance relative to the three-axis direction (X, Y, Z). Therefore, image rendering includes rotating the image according to the rotation angle of the VR wearable device, and / or translating the image according to the translation distance of the VR wearable device. In some embodiments, the pose can include the orientation and position of the user, and when the pose of the user changes, the perspective of the user changes. Specifically, the pose can be the head pose of the user. The pose can be obtained by sensors and / or cameras in the VR wearable device.

[0076] For example, please refer to Figure 2 , which is a schematic diagram of image rendering in the VR field. When the user wears the VR wearable device and faces forward, the rendered image on the screen is located in the front, and the background object (such as mountains, water, etc.) is in the front. When the user's head pose is rotated to the right by an angle (such as 40 degrees), the image on the screen is rotated 40 degrees to the left, and the background object (such as mountains, water, etc.) is rotated 40 degrees to the left. In this way, the virtual environment seen by the user is interactive with the user, and the experience is good.

[0077] It can be understood that the VR wearable device can render (rotate and / or translate) the image according to the current pose. For example, if the VR wearable device renders 60 frames of images within 60 ms, then at the 1st ms, the image (which can be understood as the original image, i.e. the unrendered image) can be rendered according to the pose at the 1st ms. The pose at the 1st ms can be motion data generated by the motion sensor at the 1st ms, such as rotation angle and / or translation distance. At the 2nd ms, the image is rendered according to the pose at the 2nd ms (motion data generated by the motion sensor at the 2nd ms, such as rotation angle and / or translation distance), and so on.

[0078] (6) Depth of Field (DOF)

[0079] 3D images comprise objects at varying depths. For example, when a VR wearable device displays a 3D image, the user sees a 3D scene. Different objects in this scene are at different distances from the user's eyes, creating a sense of depth. Therefore, image depth can be understood as the distance between objects in a 3D image and the user's eyes. The greater the image depth, the farther away objects appear to be, like distant scenery; conversely, the smaller the image depth, the closer objects appear to be, like close-up scenery. Image depth can also be referred to as "depth of field."

[0080] (7) Image rendering frame rate and image refresh frame rate

[0081] Image rendering frame rate refers to the number of image frames rendered per unit time (e.g., 1 second, 60 ms, etc.), that is, how many frames of images can be rendered per unit of time. If the unit time is 1 second, the unit of image rendering frame rate can be fps. The higher the image rendering frame rate, the higher the computing power required by the chip. It should be noted that this application does not limit the specific length (duration) of the unit time; it can be 1 second, 1 ms, 60 ms, etc., as long as it is a fixed period of time.

[0082] Image refresh rate refers to the number of frames a display refreshes per unit of time (e.g., 1 second, 60 ms, etc.), that is, how many frames of images the screen can refresh per unit of time. If the unit of time is 1 second, the unit of image refresh rate can be Hertz (Hz).

[0083] Generally, if the image refresh rate is fixed, then the image rendering frame rate needs to be adapted to that refresh rate. For example, if the image refresh rate is 90Hz, then the image rendering frame rate needs to be at least 90fps to ensure sufficient image refresh on the monitor.

[0084] One approach is to refer to [link / reference]. Figure 3 The process involves rendering each image in the image stream one by one, and then refreshing the rendered image stream on the display screen. Assuming the VR wearable device has an image refresh rate of 90Hz, the image rendering frame rate must reach at least 90fps, requiring a powerful graphics processor. This also means high power consumption, which will reduce the battery life of mobile VR wearable devices with a fixed battery capacity.

[0085] To reduce rendering power consumption, one solution is to lower the image rendering frame rate; for example, the image rendering frame rate can be lower than the image refresh rate. Assuming the image refresh rate is 90Hz, the image rendering frame rate could be 30fps or 60fps. For an example using 30fps, please refer to [link / reference]. Figure 4, the display screen can only refresh 30 images (such as black images) per unit time, but since the image refresh rate is 90Hz, the 30 images rendered are obviously not enough to refresh the display screen per unit time, so the 30 images rendered need to be interpolated, such as inserting 60 images rendered to make the rendered images reach 90 images, so as to ensure that there are enough images refreshed on the display screen per unit time to ensure the display effect.

[0086] This way, due to the low image rendering frame rate, the rendering power consumption is reduced to some extent, but it will cause a higher delay in VR operation. For example, please refer to Figure 5 , multiple images need to be inserted between the i-th frame of rendered image and the i+1-th frame of rendered image, and the inserted images can be a copy of the i-th frame of image. The VR wearable device displays the rendered image stream, assuming that when displaying the i-th frame of rendered image, a trigger operation is detected, and all the images displayed before the i+1-th frame of rendered image are inserted images, since the inserted images are a copy of the previous image (i-th frame of image), during the display of the inserted images, the trigger operation of the user is not responded to, and when the i+1-th frame of rendered image is displayed, the trigger operation is responded to. Therefore, the response time of the user's trigger operation is long, the display effect is poor, and the user experience is poor.

[0087] In addition, the above scheme of reducing the image rendering frame rate will cause the close-range objects on the image to appear to be shaking. This is because, when inserting images, the inserted images can be images processed (translated and / or rotated) according to the pose of the VR wearable device. For example, Figure 5 , the image inserted between the i-th frame of rendered image and the i+1-th frame of rendered image is an image processed according to the pose of the VR wearable device, so there may be a parallax between the inserted image and the i+1-th frame of rendered image, because the i+1-th frame of rendered image is continuous with the i-th frame of rendered image. In this way, the object will visually appear to be shaking, and the lower the image rendering frame rate, the more interpolation is needed, the more obvious the time difference, and the three-dimensional image has the characteristics of near large and far small, so the close-range object shaking phenomenon is more obvious, the display effect is poor, and the experience is poor.

[0088] Moreover, in the above scheme of reducing the image rendering frame rate, black borders will appear on the image. For example, please refer to Figure 6, continue to take an example of inserting a frame image between the image rendered after the i-th frame and the image rendered after the i+1-th frame, the inserted image is obtained by rotating and / or translating the i-th frame image according to the pose of the VR wearable device. For example, when the VR wearable device moves right, the inserted image is an image translated right relative to the i-th frame image, so that the two images are misaligned, and the overlapping part is displayed on the display screen, and the non-overlapping part (diagonal line part) is displayed as black screen, so that a black border is seen on the display screen, affecting the user experience. Therefore, Figure 4 The scheme with a low image rendering frame rate in the prior art has many problems.

[0089] To better improve the display effect, an embodiment of the present application provides a display method, in which a display device presents N frame images to a user; wherein the object at a first depth of field on the j-th frame image is the same as the object at the first depth of field on the i-th frame image; the object at a second depth of field on the j-th frame image is different from the object at the second depth of field on the i-th frame image; i is less than j. For example, the VR wearable device displays N frame images, and the user wearing the VR wearable device sees the N frame images constantly refreshing, wherein the near objects constantly change, and the distant objects are relatively unchanged. This is because the near objects use a higher image rendering frame rate, and the distant objects use a lower image rendering frame rate, so that the number of frames of the near objects rendered per unit time is higher than that of the distant objects, and the missing distant objects can be obtained by using the frame insertion method, but the inserted distant objects will make the distant objects appear unchanged. Generally, the user pays less attention to the distant objects and pays more attention to the near objects, so that using a low rendering frame rate to render the distant objects can save rendering power consumption and does not affect the user experience, and the rendering frame rate of the near objects is high, which ensures the user experience.

[0090] The following describes several application scenarios provided by an embodiment of the present application.

[0091] Exemplarily, Figure 7 The first application scenario provided by an embodiment of the present application is shown in the following schematic diagram.

[0092] The display screen of the VR wearable device displays an image 701, which is a three-dimensional image after rendering, including a mountain, a sea, and a plurality of objects such as a boy kicking a football, so that the user wearing the VR wearable device sees a virtual environment 702 in which the boy is kicking the football in an environment including the mountain and the sea. In this scenario, the VR wearable device can determine the object that the user's eyes focus on, and when rendering the image, the object that the user's eyes focus on can be rendered using a high frame rate, and other objects can be rendered using a low frame rate. In some embodiments, one or more of the close-up object (the boy), the medium shot object (the sea or the ship), or the long shot object (the mountain) can be a real object captured by the camera of the VR wearable device. In some embodiments, the close-up object can also be a user interface (UI) or an interface such as a video playing interface.

[0093] For example, the VR wearable device determines that the object that the user focuses on is the boy, and when rendering the image 701, the VR wearable device renders the boy using a higher image rendering frame rate and renders other objects such as the mountain, the sea, the bird, and the ship using a lower image rendering frame rate. The synthesized image of the objects after rendering is 701. In one implementation, the VR wearable device can default to the close-up object (for example, the boy) as the object that the user focuses on; in another implementation, the VR wearable device can track the user's gaze point to determine the object that the user focuses on, and when the object that the user focuses on is the boy, the VR wearable device renders the boy using a higher image rendering frame rate and renders other objects such as the mountain, the sea, the bird, and the ship using a lower image rendering frame rate.

[0094] Because the image rendering frame rate for the object that the user focuses on is higher than that for other objects, the number of frames of the object that the user focuses on rendered per unit time is higher than that of other objects, that is, some other objects are missing, and the missing other objects can be obtained using the interpolation method. For example, 60 frames of the object that the user focuses on and 30 frames of other objects are rendered per unit time, that is, 30 frames of other objects are missing per unit time, at this time, 30 frames of other objects can be inserted, after interpolation, there are 60 frames of the object that the user focuses on and 60 frames of other objects per unit time, and 60 frames of images can be synthesized and displayed. Because the image rendering frame rate corresponding to other objects is low, the interpolation method is used, so that the user visually sees that the other objects in the virtual environment 702 change slowly, which has little effect on the user experience (the user does not focus on these objects), and can save rendering power consumption. For the object that the user focuses on, the rendering frame rate is high, which can reduce the latency and improve the user experience.

[0095] For example, Figure 8 A schematic diagram of a second application scenario provided for an embodiment of the present application.

[0096] The display screen of the VR wearable device displays an image 801, which is a three-dimensional image after rendering, and the three-dimensional image includes objects such as a virtual cinema, a video playing interface, and the like. Therefore, the user wearing the VR wearable device sees a virtual environment 802 of watching a movie in a cinema. In an implementation manner, in the scenario shown in Figure 8 In the scenario shown in the figure, the VR wearable device can default that the close-range object (for example, the video playing interface) is the object that the user focuses on; in another implementation manner, the VR wearable device can track the gaze point of the user to determine the object that the user focuses on, and when the object that the user focuses on is the video playing interface, the video playing interface is rendered using a higher image rendering frame rate, and other objects such as the virtual cinema are rendered using a lower image rendering frame rate.

[0097] In the scenario, the VR wearable device can render the close-range object using a high frame rate and render the far-range object using a low frame rate when rendering the image.

[0098] Since the image depth h1 of the video playing interface is less than the image depth h2 of the virtual cinema, that is, the video playing interface is a close-range object and the virtual cinema is a far-range object, when the VR wearable device renders the image 801, the close-range object (for example, the video playing interface) is rendered using a higher image rendering frame rate, and the far-range object (for example, the virtual cinema) is rendered using a lower image rendering frame rate. The close-range object and the far-range object after rendering are combined into the image 801. For the far-range object that is lacking in a unit of time, an interpolation method can be used. In this viewing experience, the user pays less attention to the background (that is, the virtual cinema), so a lower rendering frame rate is used to save rendering power consumption, and a higher rendering frame rate is used for the close-range object (the video playing interface) to ensure smooth video playing.

[0099] In the example shown in the figure, the close-range object is taken as an example of the video playing interface. It should be noted that the close-range object can include a close-range object, a UI interface, and the like. In short, it can be any object or UI interface with an image depth less than a first threshold value. Figure 8 In an example, the close-range object is taken as an example of the video playing interface. It should be noted that the close-range object can include a close-range object, a UI interface, and the like. In short, it can be any object or UI interface with an image depth less than a first threshold value.

[0100] Figure 9 A schematic diagram of a third application scenario provided by the embodiment of the present application is shown in the figure.

[0101] The camera on the VR wearable device can capture an image, which can include a real environment (for example, including real objects such as mountains and seas) around the user, and the VR wearable device can combine the image captured by the camera and including the real environment with a virtual object (for example, a UI interface) to generate a three-dimensional image and display the three-dimensional image. The UI interface can be a UI interactive interface, such as a mobile phone desktop, a game operation interface, a video playing interface, and the like.

[0102] In an example, the close-range object is taken as an example of the video playing interface. It should be noted that the close-range object can include a close-range object, a UI interface, and the like. In short, it can be any object or UI interface with an image depth less than a first threshold value. Figure 9 ​The VR wearable device displays an image 901 on the display screen of the VR wearable device, the image 901 is a synthesized image of a real object (including a mountain, a sea, and the like) captured by a camera and a virtual object (including a UI interface). Therefore, the user wearing the VR wearable device sees a scene 902 in which the UI interface is displayed in the real environment. When rendering the image, the VR wearable device can use a high frame rate to render the virtual object and use a low frame rate to render the real object. In an implementation, the VR wearable device can default to the virtual object as the object that the user focuses on. In another implementation, the VR wearable device can track the gaze point of the user to determine the object that the user focuses on. When the object that the user focuses on is the virtual object, the VR wearable device uses a higher image rendering frame rate to render the virtual object and uses a lower image rendering frame rate to render the real object and other objects. When the object that the user focuses on is the real object, the VR wearable device uses a higher image rendering frame rate to render the real object and uses a lower image rendering frame rate to render the virtual object and other objects.

[0103] For example, when rendering the image 901, the VR wearable device uses a higher image rendering frame rate to render the virtual object (such as the UI interface) and uses a lower image rendering frame rate to render the real object (the mountain, the sea, the bird, the ship, and the like). The real object and the virtual object after rendering are synthesized into the image 901. Because the image rendering frame rate of the virtual object is higher than that of the real object, the number of frames of the virtual object rendered in a unit of time is higher than that of the real object. For the real object that is missing, an interpolation method can be used to save rendering power consumption, and the image rendering frame rate of the virtual object (the UI interface) is high, which can reduce the response time delay for operation, and the user experience is better.

[0104] Alternatively, when rendering the image 901, the VR wearable device can use a high frame rate to render the virtual object and part of the real object and use a lower frame rate to render other real objects. For example, the part of the real object is at the same depth of field as the virtual object or is closer to the eyes of the user than the virtual object. In this case, the part of the real object and the virtual object can be rendered using the same high frame rate, and other real objects are rendered using a lower frame rate.

[0105] The structure of the wearable device is described below, which can be a VR wearable device, an AR wearable device, an MR wearable device, or the like.

[0106] Figure 10 FIG. 1 is a structural schematic diagram of a wearable device provided by an embodiment of the present application. As shown in FIG. 1, the wearable device includes a camera 101, a display screen 102, a processor 103, a memory 104, and the like. Figure 10As shown, the wearable device 100 can include a processor 110, a memory 120, a sensor module 130 (which can be used to acquire the posture of the user), a microphone 140, a button 150, an input / output interface 160, a communication module 170, a camera 180, a battery 190, an optical display module 1100, and an eye tracking module 1200, etc.

[0107] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the wearable device 100. In other embodiments of the present application, the wearable device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0108] The processor 110 is generally used to control the overall operation of the wearable device 100, and can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0109] The memory in the processor 110 can also be provided for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. Avoiding repeated access reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0110] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0111] In some embodiments, the processor 110 can render different objects based on different frame rates, such as using a high frame rate to render close-range objects and using a low frame rate to render far-range objects.

[0112] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can contain multiple sets of I2C buses.

[0113] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the communication module 170. For example, the processor 110 communicates with the Bluetooth module in the communication module 170 through the UART interface to realize the Bluetooth function.

[0114] The MIPI interface can be used to connect the processor 110 and the display screen in the optical display module 1100, the camera 180, and other peripheral devices.

[0115] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 180, the display screen in the optical display module 1100, the communication module 170, the sensor module 130, the microphone 140, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc. Optionally, the camera 180 can capture an image including a real object, the processor 110 can fuse the image captured by the camera with a virtual object, and the optical display module 1100 can realize the fused image. The application scenario is shown in the example, which will not be repeated here. Figure 9

[0116] The USB interface is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the wearable device 100, or to transmit data between the wearable device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as a mobile phone, etc. The USB interface can be USB3.0, used to support high-speed display interface (DisplayPort, DP) signal transmission, and can transmit high-speed audio and video data.

[0117] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the wearable device 100. In other embodiments of the present application, the wearable device 100 can also use different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0118] In addition, the wearable device 100 can include a wireless communication function, for example, the wearable device 100 can receive a rendered image from another electronic device (such as a VR host or a VR server) for display, or receive an unrendered image and then the processor 110 renders and displays the image. The communication module 170 can include a wireless communication module and a mobile communication module. The wireless communication function can be realized by an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown), etc.

[0119] The antenna is used to transmit and receive electromagnetic wave signals. The wearable device 100 can include multiple antennas, each of which can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch. ​

[0120] The mobile communication module can provide a solution for wireless communication including 2th generation (2G) network / 3th generation (3G) network / 4th generation (4G) network / 5th generation (5G) network, etc. applied on the wearable device 100. The mobile communication module can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves by an antenna, and perform filtering, amplification, etc. on the received electromagnetic waves, and transfer to a modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor, and radiate as electromagnetic waves by the antenna. In some embodiments, at least part of the function modules of the mobile communication module can be disposed in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module can be disposed in the same device as at least part of the modules of the processor 110.

[0121] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, etc.), or displays an image or a video through a display screen in the optical display module 1100. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110, and disposed in the same device as the mobile communication module or other function modules.

[0122] The wireless communication module can provide a wireless communication solution including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the wearable device 100. The wireless communication module can be one or more devices that integrate at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via an antenna.

[0123] In some embodiments, the antenna of the wearable device 100 and the mobile communication module are coupled, so that the wearable device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0124] The wearable device 100 implements a display function through a GPU, an optical display module 1100, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the optical display module 1100 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0125] The memory 120 can be used to store computer executable program codes including instructions. The processor 110 performs various functional applications of the wearable device 100 and data processing by executing the instructions stored in the memory 120. The memory 120 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (e.g., a sound play function, an image play function, etc.), etc. The data storage area can store data (e.g., audio data, a phonebook, etc.) created during the use of the wearable device 100, etc. In addition, the memory 120 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0126] The wearable device 100 can implement an audio function through an audio module, a speaker, a microphone 140, a headset interface, and an application processor, etc. For example, music play, sound recording, etc.

[0127] The audio module is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module can also be used to encode and decode an audio signal. In some embodiments, the audio module can be provided in the processor 110, or some functional modules of the audio module can be provided in the processor 110.

[0128] The speaker, also referred to as a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The wearable device 100 can listen to music or listen to a hands-free call through the speaker.

[0129] The microphone 140, also referred to as a "microphone", "sound transducer", is used to convert a sound signal into an electrical signal. The wearable device 100 can be provided with at least one microphone 140. In other embodiments, the wearable device 100 can be provided with two microphones 140, in addition to collecting a sound signal, a noise reduction function can also be implemented. In other embodiments, the wearable device 100 can also be provided with three, four or more microphones 140, to collect sound signals, reduce noise, and also identify the source of the sound, implement directional recording functions, etc.

[0130] The headset interface is used to connect a wired headset. The headset interface can be a USB interface, or a 3.5 millimeter (mm) open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0131] In some embodiments, the wearable device 100 can include one or more buttons 150 that can control the wearable device and provide a user with access to functions on the wearable device 100. The buttons 150 can be in the form of push buttons, switches, dials, and touch or near touch sensing devices (e.g., touch sensors). Specifically, for example, a user can turn on the optical display module 1100 of the wearable device 100 by pressing a button. The buttons 150 can include a power button, a volume button, and the like. The buttons 150 can be mechanical buttons. They can also be touch buttons. The wearable device 100 can receive button inputs and generate key signal inputs related to user settings and function controls of the wearable device 100.

[0132] In some embodiments, the wearable device 100 can include an input / output interface 160 that can connect other devices to the wearable device 100 through suitable components. The components can include, for example, audio / video jacks, data connectors, and the like.

[0133] The optical display module 1100 is used to present images to a user under the control of the processor. The optical display module 1100 can convert real pixel image display into virtual image display of near-eye projection through one or more optical devices such as mirrors, lenses, or optical waveguides, to achieve virtual interactive experience or virtual and real combined interactive experience. For example, the optical display module 1100 receives image data information sent by the processor and presents corresponding images to the user.

[0134] In some embodiments, the wearable device 100 can also include an eye movement tracking module 1200 that is used to track the movement of a human eye and determine the gaze point of the human eye. For example, the pupil position can be located through image processing technology, the pupil center coordinates can be obtained, and the gaze point of the human eye can be calculated. For example, the eye movement tracking module 1200 can be implemented by capturing an image of the user's eye through a camera. Through the image of the user's eye, the position coordinates of the display screen that the user's eye is gazing at are calculated, which are the gaze point of the user, and the gaze point is sent to the processor 110. The processor 110 can render the object at the gaze point using a high rendering frame rate. In another embodiment, the eye movement tracking module 1200 can include an infrared emitter that emits infrared light towards the pupil of the user's eye. The cornea of the eye reflects the infrared light, and an infrared camera tracks the reflected infrared light to track the movement of the gaze point.

[0135] The technical solutions provided by the embodiments of the present application are described below in conjunction with the drawings. The following technical solutions can be applied to Figures 7 to 9 a variety of application scenarios.

[0136] Please refer to Figure 11A flowchart of a display information processing method provided by an embodiment of the present application is shown in FIG. 1. The method can be applied to a wearable device (e.g., a VR wearable device) or other electronic device (e.g., a VR host or a VR server) connected to the wearable device. As shown in FIG. 1, the flow of the method includes the following steps. Figure 11

[0137] S1, determining a first object.

[0138] For example, the first object can be a point of interest of a user among all objects to be rendered.

[0139] In mode 1, the point of interest of the user is determined according to an eye tracking technology. For example, as shown in FIG. 2, the VR wearable device determines that the user gazes at the boy according to the eye tracking technology, and determines that the boy is the point of interest. Figure 7

[0140] In mode 2, the point of interest can be a preset object, which includes a UI interface, a close-range object, a virtual object, etc. For example, as shown in FIG. 3, the point of interest is a close-range object or a UI interface. For example, as shown in FIG. 4, the point of interest is a virtual object. Figure 8 Figure 9 In mode 2, the point of interest of the user does not need to be determined in combination with the eye tracking technology.

[0141] Optionally, mode 1 and mode 2 described above can be used alone or in combination, which is not limited in the embodiments of the present application.

[0142] S2, determining a second object.

[0143] For example, the second object is an object other than the first object among all objects to be rendered. For example, the first object is an object of a first depth of field (e.g., a close-range object), and the second object can be an object of a second depth of field (e.g., a long-range object) and / or an object of a third depth of field (e.g., a medium-range object), i.e., the image depth of the second object is greater than that of the first object. For example, the first image depth of the first object is less than a first threshold, and the second image depth of the second object is greater than a second threshold, and the first threshold is less than or equal to the second threshold. The specific values of the first threshold and the second threshold are not limited in the embodiments of the present application. For example, the image depths of the close-range object and the long-range object can be seen from Table 1 below.

[0144] Table 1: Image depth range of close-range object and long-range object

[0145] Object Image depth Close-up object 0.1 m - 10 m Long shot object 100 m - 1000 m

[0146] S3, rendering the first object at a first image rendering frame rate, and the first rendering frame rate is used to indicate the number of frames of the first object that can be rendered within a certain time length. ​​​

[0147] S4, rendering the second object at a second image rendering frame rate, the second rendering frame rate being used to indicate a frame number of the second object that can be rendered in a time length, wherein the first image rendering frame rate is greater than the second image rendering frame rate.

[0148] The following takes the first object as a close-range object and the first image rendering frame rate corresponding to the close-range object as N as an example, and takes the second object as a long-range object and the second image rendering frame rate corresponding to the long-range object as M as an example, to introduce the rendering principle of the first object and the second object, wherein M and N are positive integers, and N is greater than M.

[0149] As shown in Figure 12 , N frames of close-range objects and M frames of long-range objects are rendered in a unit of time. Since N is greater than M, the close-range objects are more than the long-range objects by N-M frames in a unit of time.

[0150] S5, fusing the rendered first object and the second object to obtain a virtual image.

[0151] As an example, continuing to refer to Figure 12 , the frame number M of the long-range objects is less than the frame number N of the close-range objects in a unit of time, and therefore, the long-range objects need to be interpolated before being fused, N-M frames of long-range objects are inserted to ensure that the frame numbers of the close-range objects and the long-range objects are the same, and then the close-range objects and the long-range objects are fused.

[0152] One implementation manner is that, referring to Figure 13A , N frames of close-range objects and M frames of long-range objects are rendered in a time length, and N is greater than M. Since the frame number of the long-range objects is less, N-M frames of long-range objects can be inserted. The inserted N-M frames of long-range objects can be a copy of at least one frame of the M frames of long-range objects. When the N-M frames of long-range objects are inserted, one frame can be inserted every few frames, which is not limited in the embodiments of the application. In this way, the frame numbers of the close-range objects and the long-range objects are consistent, both of which are N, and N frames of close-range objects and N frames of long-range objects can be correspondingly fused to obtain N frames of fused images. If N is less than an image refresh frame rate P, then P-N frames of fused images are continuously inserted to obtain P frames of fused images and display. When the P-N frames of fused images are inserted, the inserted P-N frames of fused images can be images obtained by translating and / or rotating at least one frame in the N frames of fused images according to the posture of the VR wearable device.

[0153] Another implementation manner is that, referring to Figure 13BSince the number of frames of the long-range object is small, N-M frames of the long-range object can be inserted. The inserted N-M frames of the long-range object can be the long-range object after at least one frame of the M frames of the long-range object is rotated and / or translated according to the pose of the VR wearable device. When inserting the N-M frames of the long-range object, a frame can be inserted every few frames, which is not limited in the embodiments of the present application. In this way, the number of frames of the close-range object and the number of frames of the long-range object are both N, and the N frames of the close-range object and the N frames of the long-range object can be correspondingly fused to obtain N frames of the fused image. Figure 13B The difference from Figure 13A is that the inserted N-M frames of the long-range object are different. If the inserted image is the copy of the previous frame, this way has small workload and high efficiency; if the inserted image is the image after the previous frame is translated and / or rotated, this way has good user experience because the inserted image is the image after the previous frame is translated and / or rotated according to the pose of the VR wearable device, and the image seen by the user is adapted to the user pose (the user pose corresponds to the pose of the VR wearable device). Figure 13A Figure 13B

[0154] Specifically, S5 can include the following steps:

[0155] Step 1, align the N frames of the close-range object and the M frames of the long-range object.

[0156] Exemplarily, Figure 12 , the rendering time of the N frames of the close-range object and the rendering time of the M frames of the long-range object can be staggered. For example, the rendering time of the first frame of the close-range object and the first frame of the long-range object is the same, that is, they start rendering at the same time, but the rendering time of the second frame of the close-range object and the second frame of the long-range object is not the same because the frame rates are different. Therefore, the N frames of the close-range object and the M frames of the long-range object can be aligned in step 1.

[0157] The first alignment method is to determine the jth frame of the close-range object that is close in rendering time to the ith frame of the long-range object in the N frames of the close-range object, and align the ith frame of the long-range object with the jth frame of the close-range object.

[0158] Exemplarily, please refer to Figure 12 , it is assumed that the ith frame of the long-range object is the 2nd frame of the long-range object, and the 3rd frame of the close-range object in the N frames of the close-range object is determined to be close in rendering time to the 2nd frame of the long-range object, then the 2nd frame of the long-range object is aligned with the 3rd frame of the close-range object, and the effect after alignment is shown in Figure 14A .

[0159] ​​It can be understood that in some cases, step 1 can not need to be performed, for example, N = 60, M = 30, that is, 60 frames of close-range objects and 30 frames of long-range objects are rendered in a unit time, that is, the rendering speed of the close-range objects is exactly twice the rendering speed of the long-range objects, 1 frame of close-range objects is rendered every Tms, and 1 frame of long-range objects is rendered every 2Tms, for example, the first frame of close-range objects and the first frame of long-range objects are rendered at Tms, the second frame of close-range objects is rendered at 2Tms (at this time, the second frame of long-range objects is not rendered), the third frame of close-range objects and the second frame of long-range objects are rendered at 3Tms, so that the rendering time of the close-range objects and the long-range objects is aligned by itself, and no additional alignment is needed.

[0160] The second alignment manner is to align the M frames of long-range objects with the first M frames of close-range objects in the N frames of close-range objects. For example, see Figure 14B that is, the first frame of long-range objects is aligned with the first frame of close-range objects, the second frame of long-range objects is aligned with the second frame of close-range objects, and so on.

[0161] The above are two alignment manners, and other alignment manners are also feasible, and the embodiments of the present application are not limited.

[0162] Step 2, inserting N-M frames of long-range objects so that the number of frames of the long-range objects reaches N frames.

[0163] The number of frames of the long-range objects is N-M frames less than the number of frames of the close-range objects, so after the long-range objects and the close-range objects are aligned in the foregoing step 1, there are N-M frames of close-range objects that do not correspond to long-range objects, for example, Figure 14A and Figure 14B In the foregoing, part of the close-range objects do not correspond to long-range objects, so N-M frames of long-range objects are inserted, and the inserted N-M frames of long-range objects correspond to the close-range objects that do not correspond to long-range objects in the N frames of close-range objects.

[0164] Since there are the above two alignment manners, different alignment manners have different frame insertion manners, so the following two cases are introduced, the first case is for the first alignment manner, and the second case is for the second alignment manner.

[0165] In the first case, the alignment manner is the first alignment manner (that is, the alignment manner of Figure 14A ).

[0166] For the first case, the first frame insertion manner can be as follows Figure 15AAs shown, one frame of the background object is inserted between the first frame of the background object and the second frame of the background object, and the inserted background object can be the last frame of the background object, i.e., the first frame of the background object. One frame of the background object is inserted between the second frame of the background object and the third frame of the background object, and the inserted background object can be the last frame of the background object, i.e., the second frame of the background object. In this way, after N-M frames of the background object are inserted, the number of frames of the background object reaches N frames. This insertion method can be simply understood as inserting the last frame of the background object at the missing frame.

[0167] For the first case, the second insertion method can be as shown in FIG. 15B. Figure 15A For example, one frame of the background object is inserted between the first frame of the background object and the second frame of the background object, and the inserted background object can be an image processed (rotated and / or translated) from the last frame, i.e., the first frame of the background object, according to the pose of the VR wearable device. The difference between the first insertion method and the second insertion method is that the first insertion method directly inserts the last frame of the background object between the first frame of the background object and the second frame of the background object, while the second insertion method inserts an image processed (rotated and / or translated) from the last frame of the background object according to the pose of the VR wearable device between the first frame of the background object and the second frame of the background object. Similarly, continuing to refer to FIG. 15A, one frame of the background object is inserted between the second frame of the background object and the third frame of the background object, and the inserted background object can be an image processed (rotated and / or translated) from the last frame, i.e., the second frame of the background object, according to the pose of the VR wearable device. In this way, the inserted image at the missing frame is an image processed from the last frame of the background object.

[0168] For the second case, the alignment method is the second alignment method (i.e., the alignment method of FIG. 14B). Figure 14B

[0169] For the second case, the first insertion method is as shown in FIG. 15B. Figure 15B For example, N-M frames of the background object are inserted after the Mth frame of the background object. The inserted N-M frames of the background object can include at least one frame of the M frames of the background object, such as the inserted N-M frames of the background object are all the Mth frame of the background object, i.e., the M+1th frame to the Nth frame are all copies of the Mth frame of the background object.

[0170] For the second case, the second insertion method is as shown in FIG. 15B. Figure 15B ​As shown, N-M frames of the long-range objects are inserted after the Mth frame of the long-range objects, and the inserted N-M frames of the long-range objects can include the long-range objects after the M frames of the long-range objects are processed (rotated and / or translated) according to the VR wearable device. For example, the inserted N-M frames of the long-range objects are all the long-range objects after the Mth frame of the long-range objects are processed (rotated and / or translated) according to the pose of the VR wearable device. Therefore, the difference between the first insertion manner and the second insertion manner is that the first insertion manner directly inserts the Mth frame of the long-range objects at the missing frame, while the second insertion manner inserts the long-range objects after the Mth frame of the long-range objects are rotated and / or translated according to the VR wearable device at the missing frame.

[0171] After the insertion, the number of frames of the long-range objects and the number of frames of the short-range objects are the same, both are N, and step 3 can be performed.

[0172] Step 3, the N frames of the long-range objects are fused with the N frames of the short-range objects.

[0173] For example, as shown in Figure 15A , the first frame of the short-range objects is fused with the first frame of the long-range objects to obtain a first frame of the fusion image, the second frame of the short-range objects is fused with the inserted long-range object to obtain a second frame of the fusion image, and so on, to obtain N frames of the fusion image.

[0174] For example, as shown in Figure 15B , the first frame of the short-range objects is fused with the first frame of the long-range objects to obtain a first frame of the fusion image, the Mth frame of the short-range objects is fused with the Mth frame of the long-range objects to obtain a second frame of the fusion image, the M+1th frame of the long-range objects (the inserted first frame of the long-range objects) is fused with the M+1th frame of the short-range objects to obtain a M+1th frame of the fusion image, and so on, to obtain N frames of the fusion image.

[0175] S6, presenting the virtual image to the user through a virtual display device.

[0176] For example, as shown in Figure 15A , the N frames of the fusion image are displayed through the virtual display device. Among the N frames of the fusion image, the jth frame of the fusion image is the same as the long-range object in the ith frame of the fusion image, and the short-range object is different, i is less than j. For example, i=1, j=2. This is because the long-range object in the jth frame of the fusion image is copied from the long-range object in the ith frame of the fusion image or is the object after the long-range object in the ith frame of the fusion image is rotated and / or translated. Therefore, in the view of the user, the long-range object does not change, and the short-range object changes.

[0177] Optionally, before S6, a step of determining an image refresh frame rate P of the virtual display device (such as a VR wearable device) can also be included, and P is greater than N, the image refresh frame rate is used to indicate the number of frames of refreshing images per unit time, and N frames of fusion images are inserted to make the number of frames of fusion images reach P, so as to ensure that there is enough image refreshing on the display screen.

[0178] For example, referring to Figure 15C , the fusion images include N frames, the image refresh frame rate is P, N is less than P, and P-N frames of fusion images are inserted after the Nth frame of fusion images, and the P-N frames of fusion images inserted here can include at least one frame of the N frames of fusion images, such as all being the Nth frame of fusion images.

[0179] In the above embodiments, the near objects are rendered at a high image rendering frame rate, and the far objects are rendered at a low image rendering frame rate. There is a case that the user wearing the VR wearable device may pay attention to the far objects during watching the rendered images, and if it is determined that the user pays attention to the far objects, the image rendering frame rate corresponding to the far objects can be increased, and / or the image rendering frame rate of the near objects can be reduced.

[0180] That is, the image rendering frame rate of a virtual object can be adjusted according to the change of the attention degree of the user to the virtual object, when the user pays attention to the virtual object, the image rendering frame rate corresponding to the virtual object is increased, and when the user does not pay attention to the virtual object, the image rendering frame rate corresponding to the virtual object is reduced. For example, the VR wearable device can determine the attention degree of the user to the virtual object through the interaction degree of the user and the virtual object. For example, it is determined that the user pays attention to the far objects when it is detected that the user interacts with the far objects more frequently. Or, the VR wearable device determines that the user pays attention to the far objects through eye tracking when it is determined that the user gazes at the far objects.

[0181] In the above embodiments, different rendering frame rates are used for the near objects and the far objects, and in other embodiments, a plurality of virtual objects to be rendered can also be divided into more depth levels according to image depth information, such as a first object, a second object and a third object, wherein the first image depth of the first object is less than the third image depth of the third object, and the third image depth of the third object is less than the second image depth of the second object. The first object can be referred to as a “near object”, the third object can be referred to as a “middle object”, and the second object can be referred to as a “far object”.

[0182] For example, the first image depth of the first object is less than a first threshold, the third image depth of the third object is greater than the first threshold but less than a second threshold, and the second image depth of the second object is greater than the second threshold. The specific values ​​of the first and second thresholds are not limited in this embodiment. For example, the depth threshold ranges for near-field objects, mid-field objects, and far-field objects are shown in Table 2 below:

[0183] Table 2: Image Depth Range for Foreground, Midground, and Background Objects

[0184] Object Image depth Close-up object 0.1-10m Medium shot object 10-100m Long shot object 100-1000m

[0185] Among them, the first image rendering frame rate N of the near-field object is greater than the third image rendering frame rate K of the mid-field object, and the third image rendering frame rate K of the mid-field object is greater than the second image rendering frame rate M of the far-field object.

[0186] One possible approach is, such as Figure 16A As shown, N frames of close-up objects, K frames of mid-range objects, and M frames of distant objects are rendered within a certain time period, where N is greater than K and greater than M. Since the distant and mid-range objects have fewer frames, frame interpolation is necessary. For example, NK frames of mid-range objects are inserted (the inserted NK frames can be copies of at least one frame from the K frames of mid-range objects), and NM frames of distant objects are inserted (the inserted NM frames of mid-range objects can be copies of at least one frame from the M frames of mid-range objects). In this way, the number of frames for close-up, mid-range, and distant objects is always N, allowing for corresponding fusion to obtain an N-frame fused image. If N is less than the image refresh rate P, then PN frames of fused images are continued to be inserted to obtain a P-frame fused image, which is then displayed. When inserting PN frames of fused images, the inserted PN frames can be images obtained by translating and / or rotating at least one frame from the N-frame fused image according to the posture of the VR wearable device.

[0187] Another possible approach is, such as Figure 16B As shown, N frames of close-up objects, K frames of mid-range objects, and M frames of distant objects are rendered within a certain time period, where N is greater than K, which is greater than M. Since the number of frames for distant and mid-range objects is small, frame interpolation is necessary. For example, NK frames of mid-range objects are inserted (the inserted NK frames can be mid-range objects rotated and / or translated based on the VR wearer's posture at least one frame from the K frames), and NM frames of distant objects are inserted (the inserted NM frames of distant objects can be distant objects rotated and / or translated based on the VR wearer's posture at least one frame from the M frames). In this way, the number of frames for close-up, mid-range, and distant objects is always N, allowing for corresponding fusion to obtain an N-frame blended image.

[0188] The following is based on Figure 7The application scenario shown is an example, and the first image rendering frame rate N of the close-up object is 60, the third image rendering frame rate K of the medium shot object is 45, and the second image rendering frame rate M of the long shot object is 30. The rendering process is introduced taking Figure 7 The close-up object in the scene is a small boy, the medium shot object is a boat, and the long shot object is a mountain.

[0189] As shown in Figure 17 , taking 60 ms as an example, 60 frames of close-up objects, 45 frames of medium shot objects, and 30 frames of long shot objects are rendered in a unit time. Specifically, the close-up object is rendered one frame per 1 ms, the medium shot object is rendered one frame per 1.33 ms, and the long shot object is rendered one frame per 2 ms. For example, the close-up object, the medium shot object, and the long shot object start rendering at the same time, so the first frame of the close-up object, the first frame of the medium shot object, and the first frame of the long shot object are rendered at the first ms, the second frame of the close-up object is rendered at the second ms, the second frame of the medium shot object is rendered at 2.33 ms, the third frame of the close-up object and the second frame of the long shot object are rendered at the third ms, and so on. Therefore, the number of frames of the close-up object rendered in a unit time is the most, the number of frames of the medium shot object is the second, and the number of frames of the long shot object is the least. It should be noted that the unit time is 60 ms in this paper, and in fact, the unit time can be any length of time period, such as 1 s (i.e., 1000 ms).

[0190] After rendering, 60 frames of close-up objects, 45 frames of medium shot objects, and 30 frames of long shot objects can be aligned. The principle of alignment can be referred to the two alignment methods provided in the foregoing, and the first method is taken as an example for introduction, that is, the close-up object, the medium shot object, and the long shot object with close rendering time are aligned.

[0191] For example, continuing to refer to Figure 17 , the first frame of the medium shot object has been aligned with the first frame of the close-up object, the rendering time of the second frame of the medium shot object is 2.33 ms, which is relatively close to the second frame of the close-up object, so the second frame of the medium shot object is aligned with the second frame of the close-up object, as shown in Figure 18 . The rendering time of the third frame of the medium shot object is 3.66 ms, which is relatively close to the rendering time of the fourth frame of the close-up object (i.e., the fourth ms), so the third frame of the medium shot object is aligned with the fourth frame of the close-up object, and so on. Since the long shot object has been aligned with the close-up object, it can not be aligned again.

[0192] After the close-up object, the medium shot object, and the long shot object are aligned, the missing objects can be interpolated.

[0193] For example, referring to FIG. 19, the middle-scene object is missing 60-45=15 frames, so the middle-scene object needs to be inserted for 15 frames. The inserted middle-scene object can be the middle-scene object in the second frame or the middle-scene object in the second frame after being processed (rotated and / or translated) according to the pose of the VR device. Similarly, after the insertion of the middle-scene object for 15 frames, the number of middle-scene object frames reaches 60 frames.

[0194] Continuing to refer to FIG. 19, the far-scene object is missing 60-30=30 frames, so the far-scene object needs to be inserted for 30 frames. As shown in FIG. 19, Figure 19 , a far-scene object is inserted between the first frame far-scene object and the second frame far-scene object. The inserted far-scene object can be the far-scene object in the first frame or the far-scene object in the first frame after being processed (rotated and / or translated) according to the pose of the VR device. Similarly, a far-scene object is inserted between the second frame far-scene object and the third frame far-scene object. The inserted far-scene object can be the far-scene object in the second frame or the far-scene object in the second frame after being processed (rotated and / or translated) according to the pose of the VR device. Similarly, after the insertion of the far-scene object for 30 frames, the number of far-scene object frames reaches 60 frames.

[0195] When the number of near-scene object frames, middle-scene object frames, and far-scene object frames all reach 60 frames, the corresponding fusion can be performed.

[0196] For example, referring to FIG. 19, Figure 20 , the first frame near-scene object, the first frame middle-scene object, and the first frame far-scene object are fused to obtain a first frame fused image. The second frame near-scene object, the second frame middle-scene object, and the inserted far-scene object are fused to obtain a second frame fused image. Similarly, 60 frame fused images are obtained.

[0197] It should be understood that, because the middle-scene object and the far-scene object are inserted, the middle-scene object and the far-scene object change slowly on different fused images. For example, Figure 20 , the virtual display device displays the 60 frame fused images in sequence. The middle-scene object in the third frame fused image is the same as the middle-scene object in the second frame fused image, because the middle-scene object in the third frame fused image is copied from the middle-scene object in the second frame fused image or is the middle-scene object in the second frame fused image after being processed (rotated and / or translated), so the middle-scene object does not change in the user's view from the second frame fused image to the third frame fused image. However, the near-scene object in the third frame fused image is different from the near-scene object in the second frame fused image, as shown in FIG. 19. Figure 20, the appearance of the close-up object on the third frame of the fused image has changed relative to the appearance of the close-up object on the second frame of the fused image (the appearance of the small boy). It should be understood that the distant object is interpolated more, so the distant object appears to change the slowest. Thus, when the 60 frames of the fused image are refreshed, the user sees the distant object change the slowest, the mid-distant object second slowest, and the close-up object change the fastest. Generally, the user pays more attention to the close-up object, so ensuring that the close-up object changes in real time improves the viewing experience. The user pays relatively less attention to the mid-distant or distant object, so the mid-distant or distant object changes relatively slowly and does not affect the user experience, and also saves rendering power consumption.

[0198] Therefore, if the same virtual object is set at different image depths, the virtual object at different depth positions presents different effects. For example, the same virtual object (such as a video playing interface in Figure 9 ) is set at a close-up position and a distant position. Since the rendering frame rate corresponding to the virtual object at the close-up position is higher, the virtual object at the close-up position changes faster and more smoothly. The rendering frame rate corresponding to the virtual object at the distant position is lower, so the object at the distant position changes slowly and is more jittery.

[0199] It should be noted that the foregoing introduces a scheme that currently exists to render images using a low rendering frame rate, such as in Figure 4 , the image rendering frame rate is 30, which is less than the image refresh frame rate of 90. However, this low rendering frame rate is for the entire image. In other words, all virtual objects of each image correspond to the same rendering frame rate, that is, 30 frames. This scheme can cause large trigger delays and jittering phenomena due to the too low rendering frame rate of close-up objects. However, in the embodiments of the present application, different virtual objects on one image correspond to different rendering frame rates. Close-up objects can use a larger rendering frame rate to ensure the viewing experience of close-up objects. Mid-distant objects and distant objects can use a relatively low rendering frame rate to reduce rendering power consumption without affecting the user experience.

[0200] In addition, in the current scheme, when the pose of the VR wearable device changes, a black border appears, as shown in Figure 6 . This is because the image inserted during interpolation is an image after processing (rotation and / or translation), so the non-overlapping part of the inserted image and the rendered image (for example, the i-th image) has a black border. Since the current scheme uses the same image rendering frame rate for all virtual objects on the entire image, all virtual objects need to insert the same number of image frames, and the inserted images are all rotated and / or translated by the same amount, so the black borders corresponding to all virtual objects are the same.

[0201] In the embodiments of the present application, the near objects and the far objects correspond to different rendering frame rates, so the number of inserted frames for the near objects and the far objects is different, and correspondingly, the black and white corresponding to the near objects and the far objects is different. For example, one frame of the near object is inserted between the i th frame of the near object and the i+1 th frame of the near object, and the inserted one frame of the near object is processed according to the posture of the VR wearable device. For example, the non-overlapping part between the inserted one frame of the near object and the i th frame of the near object has a width equal to the displacement of the VR wearable device. Since the image rendering frame rate corresponding to the near object is high, the time interval between the i th frame of the near object and the i+1 th frame of the near object is short, and under the condition that the moving speed of the VR wearable device is certain, the displacement of the VR wearable device is small, so the width of the non-overlapping part between the inserted one frame of the near object and the i th frame of the near object is small. By the same token, when the far object is inserted between the i th frame of the far object and the i+1 th frame of the far object, the inserted far object is processed according to the posture of the VR wearable device. Assuming that the non-overlapping part between the inserted far object and the i th frame of the far object has a width equal to the displacement of the VR wearable device. Since the image rendering frame rate corresponding to the far object is low, the time interval between the i th frame of the far object and the i+1 th frame of the far object is long, and under the condition that the moving speed of the VR wearable device is certain, the displacement of the VR wearable device is large, so the width of the non-overlapping part between the inserted far object and the i th frame of the far object is large. Therefore, the width of the black and white corresponding to the near object is smaller than the width of the black and white corresponding to the far object.

[0202] The above embodiments are described by taking the first object as the near object and the second object as the far object as an example. It can be understood that the first object and the second object can also be determined based on other manners instead of the image depth, for example, the first object is a virtual object at a center position on the to-be-rendered image, and the second object is a virtual object at an edge position on the to-be-rendered image. Alternatively, the first object can be an object or an object type specified by the system or the user, and the second object is all objects on the to-be-rendered image except the first object, and the like.

[0203] In summary, different image rendering frame rates can be used for different virtual objects to be rendered, and the rendering principle is the same as that for the near object and the far object.

[0204] Based on the same concept, Figure 21 An electronic device 2000 provided by the present application is shown. The electronic device 2000 can be the mobile phone in the foregoing. As shown in FIG. 20, the electronic device 2000 includes a processor 2001, a memory 2002, a communication interface 2003, and a display 2004. Figure 21As shown, the electronic device 2000 can include one or more processors 2001, one or more memories 2002, a communication interface 2003, and one or more computer programs 2004, which can be connected through one or more communication buses 2005. Among them, the one or more computer programs 2004 are stored in the above-mentioned memory 2002 and are configured to be executed by the one or more processors 2001, and the one or more computer programs 2004 include instructions that can be used to execute the related steps of the method of the mobile phone in the above-mentioned corresponding embodiments. The communication interface 2003 is used to realize the communication with other devices, such as the communication interface can be a transceiver.

[0205] In the above embodiments of the present application, the method provided by the embodiments of the present application is introduced from the perspective of the electronic device (such as a mobile phone) as the execution subject. In order to realize each function in the above-mentioned method provided by the embodiments of the present application, the electronic device can include a hardware structure and / or a software module, and the above-mentioned each function is realized in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether a certain function in the above-mentioned each function is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application of the technical solution and the design constraint conditions.

[0206] In the above embodiments, according to the context, the term "when" or "after" can be interpreted as meaning "if" or "after" or "in response to determining" or "in response to detecting". Similarly, according to the context, the phrase "upon determining" or "if detecting (the stated condition or event)" can be interpreted as meaning "if determining" or "in response to determining" or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)". In addition, in the above embodiments, relational terms such as first, second, etc. are used to distinguish one entity from another entity, without limiting any actual relationship and order between the entities.

[0207] In this specification, the reference "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in yet some embodiments", etc. appearing in various places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically noted. The terms "include", "contain", "have" and their variants mean "including but not limited to", unless otherwise specifically noted.

[0208] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc. In the case of no conflict, the schemes of the above embodiments can be combined.

[0209] It should be noted that a part of this patent application file contains content protected by copyright. The copyright owner has no objection to the photocopy reproduction of the patent document or the patent file or record of the patent office for the purpose of the patent document content.

Claims

1. A display method characterized by comprising: The method comprises: presenting N frames of images to a user through a display device; wherein a first object at a first depth of view in a jth frame of the N frames of images is the same as a first object at the first depth of view in an ith frame of the N frames of images; a second object at a second depth of view in the jth frame of the N frames of images is different from a second object at the second depth of view in the ith frame of the N frames of images; N, i, and j are positive integers, and i is less than j; the second depth of view is a depth of view at which a gaze point of the user is located, and the second depth of view changes with a change in the gaze point of the user.

2. The method of claim 1, wherein, The first depth of view is greater than the second depth of view.

3. The method of claim 1, wherein, The first depth of view is greater than a first threshold value, and / or the second depth of view is less than a second threshold value, the first threshold value being greater than or equal to the second threshold value.

4. The method according to any of claims 1 to 3, characterized in that, i = j-1.

5. The method of any one of claims 1-3, wherein the first object at the first depth of view in the jth frame of the N frames of images is the same as the first object at the first depth of view in the ith frame of the N frames of images comprises: at least one of an action, a position, a shape, a color, or a size of the first object is the same in the jth frame of the N frames of images and in the ith frame of the N frames of images; the second object at the second depth of view in the jth frame of the N frames of images is different from the second object at the second depth of view in the ith frame of the N frames of images comprises: at least one of an action, a position, a shape, a color, or a size of the second object is different in the jth frame of the N frames of images and in the ith frame of the N frames of images.

6. The method according to any one of claims 1 to 3, characterized in that, The first object and the second object are both changing objects.

7. The method of any one of claims 1-3, wherein the first object comprises one type or multiple types of virtual objects, display objects, or interfaces; and / or the second object comprises one type or multiple types of virtual objects, display objects, or interfaces.

8. The method of any one of claims 1-3, wherein, The first object and the second object are of different types.

9. The method according to any one of claims 1 to 3, characterized in that, the first object at the first depth of view in the jth frame of the N frames of images is the same as the first object at the first depth of view in the ith frame of the N frames of images comprises: the first object at the first depth of view in the jth frame of the N frames of images is a copy of the first object at the first depth of view in the ith frame of the N frames of images; or the first object at the first depth of view in the jth frame of the N frames of images is an object obtained by translating and / or rotating the first object at the first depth of view in the ith frame of the N frames of images.

10. The method of any one of claims 1-3, wherein, the second object at the second depth of view in the jth frame of the N frames of images is different from the second object at the second depth of view in the ith frame of the N frames of images comprises: the second object at the second depth of view in the jth frame of the N frames of images is a different object from the second object at the second depth of view in the ith frame of the N frames of images; and / or the second object at the second depth of view in the jth frame of the N frames of images is a different form of the same object as the second object at the second depth of view in the ith frame of the N frames of images.

11. The method of any one of claims 1-3, wherein, Before the presenting of the N frames of images to the user through the display device, the method further comprises: generating M frames of first object images and N frames of second object images within a time period, M and N being positive integers, and M being less than N; inserting N-M frames of the first object images in the M frames of the first object images, wherein the inserted N-M frames of the first object images are copied from at least one frame of the M frames of the first object images or are images after rotation and / or translation of the at least one frame of the M frames of the first object images; fusing the N frames of the first object images and the N frames of the second object images to obtain the N frames of images.

12. The method of claim 11, wherein, inserting N-M frames of the first object images in the M frames of the first object images, wherein the inserted N-M frames of the first object images are copied from at least one frame of the M frames of the first object images or are images after rotation and / or translation of the at least one frame of the M frames of the first object images; corresponding the M frames of the second object images in the N frames of the second object images with the M frames of the first object images, the M frames of the second object images being adjacent to the M frames of the first object images in generation time; inserting N-M frames of the first object images, wherein the inserted N-M frames of the first object images correspond to the remaining N-M frames of the second object images in the N frames of the second object images.

13. The method of claim 12, wherein: the M frames of the first object images are respectively images rendered according to poses of the display device at M time instants; the N frames of the second object images are respectively images rendered according to poses of the display device at N time instants, the M time instants and the N time instants being within a first time duration.

14. The method of claim 11, wherein, presenting N frames of images to a user by a display device, comprising: in a case that the N is less than a picture refresh rate P of the display device, inserting N-P frames of the images in the N frames of images, wherein the inserted N-P frames of images are copied from at least one frame of the N frames of images or are images after rotation and / or translation of the at least one frame of the N frames of images; presenting P frames of images to a user by a display device, P being a positive integer.

15. The method of any one of claims 1-3, wherein, The method further comprises: displaying W frames of images by the display device when the user focuses on the first object at the first depth of field; wherein an object at the second depth of field on a t-th frame of images in the W frames of images is the same as an object at the second depth of field on an r-th frame of images, and an object at the first depth of field on the t-th frame of images is different from an object at the first depth of field on the r-th frame of images, N, t, and r being positive integers, and r being less than t.

16. An electronic device, comprising: comprising: a processor, a memory, and one or more programs; wherein the one or more programs are stored in the memory and comprise instructions that, when executed by the processor, cause the electronic device to perform the method steps of any one of claims 1-15.

17. A computer readable storage medium characterized by: The computer readable storage medium is configured to store a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-15.

18. A computer program product, characterised in that, The computer readable storage medium is configured to store a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-15.

Images