Display method, device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose a display method, device, electronic equipment and computer readable storage medium, the method can be applied to the virtual reality / augmented reality device provided with a zoom lens group. The method can comprise: obtaining a human eye gaze area of a user; then obtaining depth information of the human eye gaze area; then the focal length value can be determined according to the depth information of the human eye gaze area, and the focal length of the zoom lens group is adjusted to the focal length value; finally, the multimedia information can be scaled based on the focal length value, and the scaled multimedia information is displayed. Embodiments of the present application can solve the problem of visual vergence accommodation conflict by dynamically adjusting the focal length of the zoom lens group.

Description

Technical Field

[0001] This invention relates to the field of display technology, and more particularly to a display method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Virtual reality (VR) technology is an emerging multimedia technology in recent years. It uses computer hardware, software and sensors to create a virtual reality environment, enabling users to experience and interact with the virtual world through VR devices.

[0003] VR's display principle is based on binocular parallax. The screen displays two-dimensional (2D) images of the same scene from different angles in front of the observer's left and right eyes. By merging the parallax of the left and right eyes, a scene with a certain stereoscopic effect is created. During VR use, the observer's eyes focus on different areas, causing them to converge. When looking at near objects, the eyes converge; when looking at distant objects, the eyes diverge. However, the image distance of VR devices, achieved through lenses, is fixed, and the light emitted from the display lacks depth information, resulting in a fixed focal length for the observer's eyes. In the real world, human eye convergence and focal length accommodation are coordinated. Therefore, if the observer's focal length is fixed, and only convergence is performed, a visual convergence-accommodation conflict occurs. Summary of the Invention

[0004] This invention discloses a display method, apparatus, electronic device, and computer-readable storage medium for resolving the visual convergence-accommodation conflict problem.

[0005] The first aspect discloses a display method applicable to VR / AR (augmented reality) devices, modules (e.g., chips) within VR / AR devices, and logic modules or software capable of implementing all or part of the functions of a VR / AR device. The VR / AR device is equipped with a zoom lens group. The following description uses an application to a VR device as an example. The method may include: acquiring the user's eye-gaze region; acquiring depth information of the eye-gaze region; determining a focal length value based on the depth information; adjusting the focal length of the zoom lens group to the focal length value; scaling multimedia information based on the focal length value; and displaying the scaled multimedia information.

[0006] In this embodiment of the invention, the VR device can first acquire the user's gaze area, and then acquire the depth information of the gaze area. Subsequently, the VR device can determine a suitable focal length value based on this depth information, and adjust the focal length of the zoom lens group to this value. This ensures that the VR device's virtual imaging matches the depth information of the user's gaze area, thereby guaranteeing consistency between the user's convergence and focal length adjustments and avoiding visual convergence conflict. Furthermore, since the content seen by the user changes before and after adjusting the focal length value, which may cause eye discomfort, the VR device can scale the multimedia information based on the focal length value and display the scaled multimedia information to maintain a consistent view of the user's content, thus avoiding eye discomfort caused by rapid changes in the content seen.

[0007] As one possible implementation, obtaining the user's eye gaze area includes: acquiring the user's eye movement data based on eye-tracking technology; and determining the user's eye gaze area based on the eye movement data.

[0008] In this embodiment of the invention, the VR device can first acquire the user's eye movement data through eye tracking technology, and then the VR device can accurately determine the user's eye gaze area based on the eye movement data.

[0009] As one possible implementation, acquiring depth information of the area being gazed at by the human eye includes: acquiring depth information of the area being gazed at by the human eye through Simultaneous Localization and Mapping (SLAM) technology or a depth sensor.

[0010] In this embodiment of the invention, the VR device can accurately obtain depth information of the area being gazed at by the human eye through real-time localization and mapping (SLAM) technology or a depth sensor.

[0011] As one possible implementation, scaling multimedia information based on the focal length value includes: determining the image distance q based on the focal length value and the object distance p, wherein the VR / AR device is further provided with a display module, the object distance is the distance between the display module and the zoom lens group, and the image distance is the distance between the virtual imaging position and the zoom lens group; calculating the scaling value m according to the formula m = p / q; and scaling the multimedia information according to the scaling value m.

[0012] In this embodiment of the invention, the VR device can determine the image distance based on the focal length value and the object distance, and then determine the scaling value based on the image distance and the object distance. After that, the multimedia information can be scaled according to the scaling value so that the content seen by the user can remain unchanged, thereby improving the user experience.

[0013] As one possible implementation, scaling the multimedia information according to the scaling value includes: when the multimedia information is two-dimensional, performing layer scaling or rendering scaling on the multimedia information according to the scaling value; when the multimedia information is three-dimensional, performing view rendering scaling on the multimedia information according to the scaling value.

[0014] In this embodiment of the invention, the VR device can perform different scaling adjustments for two-dimensional information and three-dimensional (3D) information respectively. For two-dimensional information, the VR device can perform layer scaling or rendering scaling of the multimedia information according to the scaling value; for three-dimensional information, the VR device can perform field-of-view rendering scaling of the multimedia information according to the scaling value.

[0015] As one possible implementation, the method further includes: acquiring depth information of the non-focused region; and blurring the multimedia information of the non-focused region based on the depth information of the human eye's focused region and the non-focused region.

[0016] In this embodiment of the invention, the VR device can also acquire depth information of the non-focused area. Then, based on the depth information of the human eye's focused area and the non-focused area, the multimedia information of the non-focused area can be blurred so that the image seen by the user matches the visual effect of the real world, making the image more realistic and thus improving the user experience.

[0017] As one possible implementation, the method of blurring multimedia information in the non-focused area based on depth information of the human eye's gaze area and the non-focused area includes: determining a depth difference based on the depth information of the human eye's gaze area and the non-focused area; determining a blurring value based on the depth difference; and blurring multimedia information in the non-focused area based on the blurring value.

[0018] In this embodiment of the invention, the VR device can determine the depth difference based on the depth information of the human eye's gaze area and the non-gaze area. Then, it can determine the blurring value based on the depth difference, so that the VR device can perform more accurate blurring processing on the multimedia information of the non-gaze area based on the blurring value.

[0019] The second aspect discloses a display device, which can be a VR / AR device or a module (e.g., a chip) within a VR / AR device, the VR / AR device being provided with a zoom lens group. The device may include:

[0020] The first acquisition unit is used to acquire the user's eye gaze area;

[0021] The second acquisition unit is used to acquire depth information of the area gazed at by the human eye.

[0022] A determining unit is used to determine the focal length value based on the depth information;

[0023] The adjustment unit is used to adjust the focal length of the zoom lens group to the specified focal length value.

[0024] A scaling unit is used to scale multimedia information based on the focal length value;

[0025] The display unit is used to display scaled-down multimedia information.

[0026] As one possible implementation, the first acquisition unit is specifically used for:

[0027] The user's eye movement data is obtained based on eye-tracking technology;

[0028] Based on this eye-tracking data, the user's eye gaze area is determined.

[0029] As one possible implementation, the second acquisition unit is specifically used for:

[0030] Depth information of the area being viewed by the human eye can be obtained through real-time localization and mapping (SLAM) technology or depth sensors.

[0031] As one possible implementation, the scaling unit scales the multimedia information based on the focal length value, including:

[0032] The image distance q is determined based on the focal length value and the object distance p. The VR / AR device is also equipped with a display module. The object distance is the distance between the display module and the zoom lens group, and the image distance is the distance between the virtual imaging position and the zoom lens group.

[0033] The scaling value m is calculated using the formula m = p / q.

[0034] The multimedia information is scaled according to the scaling value m.

[0035] As one possible implementation, the scaling unit scales the multimedia information according to the scaling value, including:

[0036] If the multimedia information is two-dimensional, the multimedia information can be scaled by layer or by rendering based on the scaling value.

[0037] When the multimedia information is three-dimensional, the viewpoint rendering scales the multimedia information according to the scaling value.

[0038] As one possible implementation, the display device may further include:

[0039] The third acquisition unit is used to acquire depth information of the non-focused region;

[0040] The processing unit is used to perform blurring processing on the multimedia information of the non-focused area based on the depth information of the human eye's gaze area and the non-focused area.

[0041] As one possible implementation, the processing unit performs blurring processing on the multimedia information in the non-focused area based on the depth information of the human eye's gaze area and the non-focused area, including:

[0042] Determine the depth difference based on the depth information of the area the person is fixating on and the area they are not fixating on;

[0043] The blur value is determined based on this depth difference;

[0044] Based on this blurring value, the multimedia information in the non-focused area is blurred.

[0045] The third aspect discloses an electronic device, which can be a VR / AR device or a module (e.g., a chip) within a VR / AR device. The electronic device may include a display module, a processor, and a memory. The display module is used to display content, the memory is used to store a computer program, and the processor is used to invoke the computer program. When the processor executes the computer program stored in the memory, it causes the processor to perform the display method disclosed in the first aspect or any embodiment of the first aspect.

[0046] The fourth aspect discloses an electronic device, which can be a VR / AR device or a module (e.g., a chip) within a VR / AR device. The electronic device may include: an eye-tracking module, a depth information module, an adjustment module, a display module, and a processor. Specifically, the eye-tracking module is used to acquire the user's eye gaze area; the depth information module is used to acquire depth information of the eye gaze area; the processor is used to determine a focal length value based on the depth information; the adjustment module is used to adjust the focal length of the zoom lens group to the focal length value; the processor is also used to scale multimedia information based on the focal length value; and the display module is used to display the scaled multimedia information.

[0047] The fifth aspect discloses a computer-readable storage medium storing a computer program or computer instructions that, when executed, implement the display methods disclosed in the above aspects.

[0048] The sixth aspect discloses a chip including a processor for executing a program stored in a memory, which, when executed, causes the chip to perform the methods described above.

[0049] As one possible implementation, the memory is located outside the chip.

[0050] The seventh aspect discloses a computer program product comprising computer program code, which, when executed, causes the aforementioned display method to be performed.

[0051] It is understood that the display device provided in the second aspect, the electronic device provided in the third aspect, the electronic device provided in the fourth aspect, the computer-readable storage medium provided in the fifth aspect, the chip provided in the sixth aspect, and the computer program product provided in the seventh aspect are all used to execute the display method provided in the first aspect of this application and any possible implementation thereof. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a display method disclosed in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of a view rendering scaling disclosed in an embodiment of the present invention;

[0054] Figures 3A-3C This is a schematic diagram of multimedia information scaling disclosed in an embodiment of the present invention;

[0055] Figures 4A-4D This is a schematic diagram of another multimedia information scaling method disclosed in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the structure of a display device disclosed in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the structure of another electronic device disclosed in an embodiment of this application. Detailed Implementation

[0059] This invention discloses a display method, apparatus, electronic device, and computer-readable storage medium for resolving the visual convergence-accommodation conflict problem. The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0060] Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The reference to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily indicate the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand, explicitly and implicitly, that the embodiments described herein can be combined with other embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects and are not used to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, it may include a series of steps or units, or optionally, steps or units not listed, or optionally other steps or units inherent to these processes, methods, products, or devices.

[0061] The accompanying drawings show only the portions relevant to this application, not all of them. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict operations (or steps) as sequential processes, many of these operations may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0062] The terms “component,” “module,” “system,” “unit,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or distributed between two or more computers. Furthermore, these units can be executed from various computer-readable media on which various data structures are stored. Units can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from a second unit interacting with another unit between a local system, a distributed system, and / or a network; for example, the Internet interacting with other systems via signals).

[0063] To better understand the embodiments of the present invention, some terms and related technologies used in the embodiments of the present invention will be described below.

[0064] Augmented reality (AR) is a technology that integrates virtual information with the real world. It involves various technologies such as multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensors. It simulates and applies computer-generated text, images, 3D models, music, videos, and other virtual information to the real world, thereby "enhancing" the real world.

[0065] In the real world, to see a target clearly, a person first needs to adjust their two eyeballs, moving their line of sight towards the target (i.e., convergence-focusing accommodation). When looking at a nearby target, the eyeballs converge relatively; when looking at a distant object, the eyeballs diverge relatively. Then, to see the target clearly, the eyeballs need to adjust to the correct focal length (i.e., focal length accommodation). Therefore, in the process of observing a target in the real world, the eye's focal length accommodation and convergence accommodation work together.

[0066] Current AR and VR display technologies primarily employ stereoscopic display principles. Stereoscopic display is based on binocular parallax, using the fusion of visual perception from the left and right eyes to create a scene with a certain degree of stereoscopic effect. Based on this principle, VR and AR devices mainly consist of a display screen and lenses. Virtual imaging is achieved through the lenses, displaying 2D images with parallax in front of the observer's or user's left and right eyes respectively. The parallax of the left and right eyes is then fused to create a 3D visual experience. However, because the positions of the display screen and lenses are fixed, and the focal length of the lenses is fixed, the image distance of the 2D images observed by the user's left and right eyes is also fixed. Furthermore, the light emitted from the display screen lacks depth information; therefore, the focal length of the user's eyes remains fixed (i.e., the user's eye refractive power does not change). Thus, regardless of where the user looks in the VR world (i.e., the virtual world), the user's eye refractive power does not change; there is no adjustment of refractive power. In contrast, when the user looks at targets at different distances in the VR world, their eyes will undergo convergence. Therefore, users' eyes experience vergence-accommodation conflict (VAC), meaning a mismatch between the user's refractive accommodation and convergence accommodation. When users engage in prolonged use of VR / AR devices, this VAC conflict can easily lead to eye fatigue, nausea, dizziness, and other problems.

[0067] Currently, to address the issue of visual convergence-accommodation conflict, variable focal length lens groups can be used in VR / AR devices. By dynamically adjusting the focal length of these lenses, the image distance of the VR virtual image can be altered, matching the eye's focus adjustment with binocular convergence-accommodation and improving the user experience. However, dynamically adjusting the focal length of the variable focal length lens group can cause a mismatch between the user's left and right eye fields of view, resulting in a change in the displayed image. Specifically, changes in the refractive power of the variable focal length lens group (i.e., changes in focal length) alter the user's field of view and the image distance of the VR virtual image. In the VR world, for near scenes (i.e., close-up views), changing the focal length of the variable focal length lens group results in a smaller near-field field of view and magnified objects within that field of view; conversely, for distant scenes (i.e., far-field views), the same applies, resulting in a wider far-field field of view and smaller objects within that field of view. Furthermore, rapid changes in the focal length of the variable focal length cause rapid switching of the user's view (i.e., changes in field of view and the size of objects within that field of view), which can cause eye discomfort and negatively impact the user experience.

[0068] The display method provided in this embodiment of the invention can be executed by a VR / AR device, which will be referred to as an electronic device below. The electronic device may include an image processing device and a head-mounted display device (such as VR glasses or a VR headset). The head-mounted display device may include a zoom lens group, a sensor module, and a display module. The sensor module is used to collect data, which may include, but is not limited to, image data, angle data, and orientation data. It should be understood that the sensor module may include, but is not limited to, image sensors, cameras, accelerometers, distance sensors, gyroscopes, light sensors, temperature sensors, heart rate sensors, pedometers, microphones, etc. The image processing device may be a smartphone, tablet computer, laptop computer, etc. The image processing device and the head-mounted display device can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the invention.

[0069] Please see Figure 1 , Figure 1 This is a flowchart illustrating a display method disclosed in an embodiment of the present invention. Figure 1 As shown, the display method may include the following steps.

[0070] 101. Obtain the area of ​​the user's eye gaze.

[0071] To dynamically adjust the focal length of the zoom lens group and match the user's eye focus adjustment (i.e., focal length adjustment) with the binocular convergence adjustment (i.e., convergence adjustment), the electronic device can acquire the user's eye gaze area in real time during use. The eye gaze area is the area on the display module corresponding to the user's line of sight.

[0072] Electronic devices can acquire users' eye movement data based on eye-tracking technology, and determine the user's gaze area based on this data. Specifically, electronic devices can track and record the user's eye movements using eye-tracking technology to obtain eye movement data, which may include the coordinates of the user's gaze point, dwell time, pupil diameter, iris angle changes, etc. The electronic device can identify a gaze point on the display module where the user's eye dwell time exceeds a certain threshold; for example, a gaze point with a dwell time exceeding 20ms can be identified as a gaze point.

[0073] Electronic devices can determine a user's gaze area based on the coordinates of their gaze point. There are several ways to do this. One way is to directly define the location corresponding to the user's gaze point's coordinates as the gaze area. Another way is to define a range of areas based on the user's gaze point's coordinates; for example, using the coordinates as the center of a circle with a fixed radius (e.g., 2 mm), the area of ​​that circle can be defined as the user's gaze area. Yet another way is to pre-divide the display module into multiple areas, and then define the area containing the user's gaze point as the user's gaze area.

[0074] It should be understood that electronic devices can be equipped with eye-tracking devices (such as eye trackers) to obtain the movement status of the user's eyes.

[0075] 102. Obtain depth information of the area the person is looking at.

[0076] After acquiring the user's gaze area, the electronic device can further acquire the depth information of that gaze area. The depth information of the gaze area can be understood as the distance information of the multimedia information within that gaze area. This distance information is also the image distance corresponding to the multimedia information in the gaze area, which is the VR imaging image distance that the multimedia information in that gaze area should satisfy. In the VR world, targets at different distances can have different depths (i.e., distances). For example, distant mountains presented in the VR world have a relatively large depth (i.e., a relatively far distance from the user's eyes), while nearby objects have a relatively small depth (i.e., a relatively close distance from the user's eyes). The multimedia information can be images, videos, etc., and this embodiment of the invention is not limited thereto.

[0077] Electronic devices can acquire depth information of the area being gazed at by the human eye using simultaneous localization and mapping (SLAM) technology or depth sensors. Alternatively, they can determine the depth information of the area being gazed at by the human eye based on a depth information table. Specifically, the multimedia information displayed on the display module can correspond to a depth information table. Taking an image as an example, it can include multiple pixels, each of which can correspond to a depth, or multiple adjacent pixels (such as a 2x2 pixel block) can correspond to a depth. Therefore, the electronic device can obtain the depth of the area being gazed at by the human eye based on the depth information table corresponding to the currently displayed multimedia information. For example, if the area being gazed at by the human eye can include X pixels, and assuming that each pixel corresponds to a depth, the electronic device can determine the depth information of the area being gazed at by the average depth of these X pixels.

[0078] 103. Determine the focal length value based on this depth information.

[0079] After acquiring the depth information of the area being gazed at by the user's eye, the electronic device can determine the focal length value based on the depth information of the area being gazed at in order to adjust the focal length of the zoom lens group so that the user's eye focus adjustment matches the binocular convergence adjustment.

[0080] Specifically, the image distance of the VR image formed by the zoom lens should match the depth information of the human eye's gaze area (i.e., the VR image distance should be the same as the image distance corresponding to the multimedia information in the human eye's gaze area). Therefore, the electronic device can determine the image distance that the VR image should satisfy based on the depth information of the human eye's gaze area. Then, the electronic device can determine the focal length that the zoom lens group should achieve to satisfy this image distance using the lens imaging formula (i.e., the Gaussian imaging formula). The electronic device can determine the depth of the human eye's gaze area as the image distance that the VR image should satisfy, and then determine the focal length value that the zoom lens group should satisfy based on this image distance.

[0081] 104. Adjust the focal length of the zoom lens group to this focal length value.

[0082] Once the electronic device determines the required focal length of the zoom lens group, it can adjust the focal length of the zoom lens group to that value. It should be understood that adjusting the focal length of the zoom lens group is equivalent to adjusting its diopter.

[0083] It should be noted that the zoom lens group may include a main lens and a variable focal length lens. Adjusting the focal length of the zoom lens group by the electronic device is equivalent to adjusting the focal length of the variable focal length lens. The variable focal length lens may include any suitable lens, such as a glass lens, polymer lens, liquid lens, liquid crystal lens, electrodeformation lens, or some combination thereof. The variable focal length lens can adjust the direction of light emitted from the display module so that the light emitted from the display module appears at a specific focal length / image plane away from the user. In some embodiments, the variable focal length lens may include a liquid crystal lens, which can adjust the optical power more quickly to keep up with the speed of eye accommodation, thereby resolving the visual convergence-accommodation conflict and improving the user experience. It should be noted that the main lens and the variable focal length lens in the zoom lens group may be parallel and coaxial.

[0084] 105. Scale the multimedia information based on the focal length value.

[0085] After an electronic device adjusts the focal length of its zoom lens group, it can scale the multimedia information based on that focal length value in order to keep the content seen by the user unchanged.

[0086] Specifically, the electronic device can determine the scaling value based on the focal length and object distance, and scale the multimedia information accordingly. The object distance is the distance between the display module and the zoom lens group. The scaling value m can be calculated using the following formula (1).

[0087] m = p / q(1)

[0088] Where p is the object distance (i.e., the distance between the display module and the zoom lens group), and q is the image distance (i.e., the distance between the virtual imaging position and the zoom lens group).

[0089] Electronic devices can determine the image distance of VR imaging based on the focal length and object distance of the zoom lens group, i.e., q in the above formula (1). It should be understood that the distance between the display module and the zoom lens group can be constant (i.e., p above is constant). Therefore, when the focal length of the zoom lens group changes, only the image distance of VR imaging will change.

[0090] It should be noted that the scaling value determined by the electronic device based on the focal length and object distance is a scaling value for the original multimedia information (such as the scaling value of the length and width of an image). The original multimedia information can be understood as multimedia information that has not been scaled.

[0091] The multimedia information displayed by the display module can be two-dimensional or three-dimensional. Different multimedia information can be scaled in different ways. When the multimedia information is two-dimensional, the electronic device can perform layer scaling or rendering scaling based on the scaling value. When the multimedia information is three-dimensional, the electronic device can perform viewpoint rendering scaling based on the scaling value.

[0092] Layer scaling refers to directly changing the size of the layer to be output to the display module, thereby correspondingly changing the size of multimedia information. Layer scaling can be handled directly by the processor of the electronic device at a low level, without requiring user-defined settings, resulting in fast processing speed. However, when scaling a layer, if the multimedia information exceeds the display area threshold (i.e., the size of the multimedia information exceeds the maximum size that the display module can display), the electronic device will still render the area outside the display range.

[0093] Rendering scaling treats the entire display area of ​​the display module as a canvas. When multimedia information is scaled (e.g., an image is enlarged or reduced), if the size of the multimedia information exceeds a display area threshold, only the corresponding multimedia information within the user's field of vision will be rendered; areas outside the display range will not be rendered. If the size of the multimedia information does not exceed the display area threshold, the complete multimedia information will be displayed. Rendering scaling needs to be handled by the application layer program of the electronic device.

[0094] View-of-view rendering scaling refers to rendering images (i.e., rendering 3D multimedia information) based on different viewpoints. 3D multimedia information can have a complete environment model; therefore, view-of-view rendering scaling allows for scaling the viewpoint based on the original environment model. Please see [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of a view rendering scaling method disclosed in an embodiment of the present invention. For example... Figure 2 As shown, since 3D multimedia information has complete environmental modeling, view rendering scaling can include scaling in three dimensions: X, Y, and Z, so that the 3D multimedia information after view rendering scaling also has complete environmental modeling.

[0095] It should be noted that electronic devices can also scale multimedia information based on depth information. Therefore, it can be understood that the two steps of adjusting the focal length of the zoom lens group and scaling the multimedia information can be performed simultaneously. That is, after obtaining the depth information, the electronic device can scale the multimedia information based on the depth information while performing the above steps 103 and 104.

[0096] 106. Display scaled-down multimedia information.

[0097] After the electronic device scales the multimedia information based on the focal length value, the scaled multimedia information can be displayed through the display module. The display module can be an electronic display, which can include a single electronic display or multiple electronic displays (for example, it can include two electronic displays, respectively located in front of the user's left and right eyes). The electronic display can include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode display (AMOLED), a transparent organic light-emitting diode display, a projector, or a combination thereof.

[0098] Please refer to Figures 3A-3C , Figures 3A-3C which is a schematic diagram of multimedia information scaling disclosed in an embodiment of the present invention. As Figure 3A shown, the maximum width of the screen capable of displaying the picture is x, and the maximum height is y. The original multimedia information can be an image or a video frame, and the width and height of the original picture displayed on the screen are x1 and y1 respectively, where x1 < x and y1 < y. Before adjusting the focal length of the zoom lens group, the focal length of the zoom lens group can be f1. At this time, the picture seen by the user can be the original picture, that is, the height and width of the user's visible range are x1 and y1 respectively. After that, assuming that the focal length of the zoom lens group is adjusted to f2, the user's visible range is reduced to x2 * y2. x2 < x1 and y2 < y1. If the picture displayed on the screen is not adjusted, the picture seen by the user will change. As Figure 3B shown, the user can only see a part of the original picture, and the items seen in the VR world will be enlarged or reduced accordingly. Therefore, the electronic device can scale the multimedia information accordingly according to the change of the user's visible range (that is, determine the scaling value based on the focal length value of the zoom lens group and scale the multimedia information according to the scaling value). As Figure 3C shown, the length and width of the original picture can be scaled proportionally to x2 and y2. At this time, the user's visible range is also x2 * y2, so as to ensure that the picture content seen by the user remains unchanged before and after adjusting the focal length of the zoom lens group (that is, Figure 3A and Figure 3C the picture content in the user's visible range area is the same), improving the user experience. It should be understood that Figure 3C the scaling shown can be layer scaling, that is, scaling the original layer from x1 * y1 to x2 * y2, or rendering scaling.

[0099] Please refer to again Figures 4A-4D , Figures 4A-4D is another schematic diagram of multimedia information scaling disclosed in an embodiment of the present invention. As Figure 4A shown, the width and height of the original screen can be x and y4, its width is equal to the maximum width that the display screen can display the picture, and its height y4 is less than the maximum height y that the display screen can display the picture. Before adjusting the focal length of the zoom lens group, the focal length of the zoom lens group can be f3. At this time, the height and width of the user's visible range are x3 and y3 respectively, and the picture seen by the user can be a part of the original picture. x3 < x, y3 < y4. After that, assume that the focal length of the zoom lens group is adjusted to f4. At this time, the user's visible range becomes x5 * y5, x5 > x3, y5 > y3. If the picture displayed on the display screen is not adjusted, the picture seen by the user will change. As Figure 4B shown, the user can see a larger picture, and the items seen in the VR world will be enlarged or reduced accordingly. Therefore, the electronic device can determine the scaling value based on the focal length of the zoom lens group, and according to the scaling value, it can be determined that the original picture needs to be enlarged to x6 * y. As Figure 4C shown, the electronic device can enlarge the original multimedia information to x6 * y through layer scaling, that is, enlarge the original layer from x * y4 to x6 * y. At this time, the picture content that the user can see is the same as the content seen before the focal length changes (that is, Figure 4A and Figure 4C the picture content in the user's visible range area is the same). As can be seen from Figure 4C , when the electronic device performs layer scaling, the layer size (x * y4) is larger than the maximum picture (x * y) that the display screen can display. The electronic device will render the area beyond the display range, but it cannot be displayed on the display screen. The electronic device can also scale the multimedia information through rendering scaling. As Figure 4D shown, compared with layer scaling, when performing rendering scaling, the electronic device will not render the area beyond the display range, and only render the picture of x * y.

[0100] It should be understood that the above Figures 3A-4D shown multimedia information scaling is only an exemplary illustration and does not constitute a limitation thereto.

[0101] Optionally, the electronic device can also obtain the depth information of the non-gazed area, and based on the depth information of the human eye's gazed area and non-gazed area, blur the multimedia information in the non-gazed area. Blurring the multimedia information in the non-gazed area by the electronic device can more realistically simulate the user's viewing experience in the real world, thereby improving the user's viewing comfort.

[0102] Specifically, electronic devices can determine the depth difference (i.e., the depth difference between the gaze area and the non-gaze area) based on the depth information of the gaze area and the non-gaze area. Then, the electronic device can determine the blur value based on this depth difference and perform blurring processing on the multimedia information in the non-gaze area based on this blur value. Each depth difference corresponds to a blur value; the larger the depth difference, the larger the corresponding blur value, and vice versa. One way for the electronic device to blur the non-gaze area is as follows: the electronic device divides the non-gaze area into multiple non-gaze area blocks, calculates the depth difference between each non-gaze area block and the gaze area, obtaining the depth difference value corresponding to each non-gaze area block. Then, the electronic device can determine the blur value corresponding to each non-gaze area block based on the depth difference value, and then perform blurring processing on each non-gaze area block separately based on its corresponding blur value.

[0103] It should be understood that when electronic devices determine the blur value, they can also consider the depth information of the area the human eye is focused on. That is, electronic devices can determine the blur value of the non-focused area based on the depth information and depth difference of the area the human eye is focused on, thus determining the blur value more accurately.

[0104] In this embodiment of the invention, the electronic device may be equipped with a zoom lens group. By adjusting the focal length of the zoom lens group, the visual convergence-accommodation conflict can be improved. Furthermore, the electronic device can also scale multimedia information to maintain the same image seen by the user before and after the focal length adjustment of the zoom lens group, thereby reducing eye fatigue and improving the user experience. In addition, the electronic device can also blur the non-focused areas of the human eye to more realistically simulate the user's observation experience in the real world.

[0105] It should be understood that Figure 1 This application uses a VR / AR device (i.e., the aforementioned electronic device) as an example to illustrate the display method, but it does not limit the execution subject of the display method. For example, Figure 1 The VR / AR device mentioned can also be a chip, chip system, or processor that supports the VR / AR device in implementing the method, or it can be a logic module or software that can implement all or part of the functions of the VR / AR device.

[0106] Please see Figure 5 , Figure 5 This is a schematic diagram of a display device disclosed in an embodiment of the present invention. The display device can be a VR / AR device, or a module (e.g., a chip) within a VR / AR device, which is equipped with a zoom lens group. Figure 5 As shown, the device may include:

[0107] The first acquisition unit 501 is used to acquire the user's eye gaze area;

[0108] The second acquisition unit 502 is used to acquire depth information of the area of ​​the human eye's gaze.

[0109] The determining unit 503 is used to determine the focal length value based on the depth information;

[0110] The adjustment unit 504 is used to adjust the focal length of the zoom lens group to the focal length value.

[0111] The scaling unit 505 is used to scale multimedia information based on the focal length value;

[0112] Display unit 506 is used to display scaled multimedia information.

[0113] In one embodiment, the first acquisition unit 501 is specifically used for:

[0114] The user's eye movement data is obtained based on eye-tracking technology;

[0115] Based on this eye-tracking data, the user's eye gaze area is determined.

[0116] In one embodiment, the second acquisition unit 502 is specifically used for:

[0117] SLAM technology or depth sensors are used to obtain depth information of the area being viewed by the human eye.

[0118] In one embodiment, the scaling unit 505 scales the multimedia information based on the focal length value, including:

[0119] The image distance q is determined based on the focal length value and the object distance p. The VR / AR device is also equipped with a display module. The object distance is the distance between the display module and the zoom lens group, and the image distance is the distance between the virtual imaging position and the zoom lens group.

[0120] The scaling value m is calculated using the formula m = p / q.

[0121] The multimedia information is scaled according to the scaling value m.

[0122] In one embodiment, the scaling unit 505 scales the multimedia information according to the scaling value, including:

[0123] If the multimedia information is two-dimensional, the multimedia information can be scaled by layer or by rendering based on the scaling value.

[0124] When the multimedia information is three-dimensional, the viewpoint rendering scales the multimedia information according to the scaling value.

[0125] In one embodiment, the display device may further include:

[0126] The third acquisition unit 507 is used to acquire depth information of the non-focused region;

[0127] The processing unit 508 is used to perform blurring processing on the multimedia information of the non-focused area based on the depth information of the human eye's gaze area and the non-focused area.

[0128] In one embodiment, the processing unit 508 performs blurring processing on the multimedia information in the non-focused area based on the depth information of the human eye's gaze area and the non-focused area, including:

[0129] Determine the depth difference based on the depth information of the area the person is fixating on and the area they are not fixating on;

[0130] The blur value is determined based on this depth difference;

[0131] Based on this blurring value, the multimedia information in the non-focused area is blurred.

[0132] For a more detailed description of the first acquisition unit 501, the second acquisition unit 502, the determination unit 503, the adjustment unit 504, the scaling unit 505, the display unit 506, the third acquisition unit 507, and the processing unit 508, please refer directly to the above description. Figure 1 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0133] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. The electronic device can be a VR / AR device, or a module (e.g., a chip) within a VR / AR device, and the VR / AR device is equipped with a zoom lens group. Figure 6 As shown, the electronic device may include an eye-tracking module, a depth information module, an adjustment module, a display module, and a processor. Furthermore, the eye-tracking module, depth information module, adjustment module, and display module can each be connected to the processor.

[0134] The eye-tracking module described above can be used to acquire eye movement data of the user's eyes, and can also acquire the user's eye gaze area through the eye movement data (i.e., execute step 101 above). The processor can acquire the depth information of the eye gaze area through the depth information module (i.e., execute step 102 above). Then, the processor can determine the focal length value based on the depth information (i.e., execute step 103 above). Next, the processor can adjust the focal length of the zoom lens group to the specified focal length value through the adjustment module (i.e., execute step 104 above). Then, the processor can scale the multimedia information based on the depth information or the focal length value, and display the scaled multimedia information through the display module (i.e., execute steps 105 and 106 above).

[0135] In some embodiments, after the eye-tracking module acquires the user's eye movement data, it can send the eye movement data to the processor. The processor can then acquire the user's eye gaze area through the eye movement data.

[0136] Optionally, the eye-tracking module can be an eye tracker, the depth information module can be a SLAM module or a depth sensor, and the display module can be an electronic display. In one embodiment, the electronic device may include a head-mounted display device (such as VR glasses or a VR helmet), and the eye-tracking module, depth information module, adjustment module, and display module may be disposed within the head-mounted display device.

[0137] In some embodiments, the electronic device may further include a camera module (such as a webcam) capable of capturing images or videos of the surrounding environment in real time. The electronic device may also include a position sensor, such as one or more accelerometers, one or more gyroscopes, one or more magnetometers, or a combination thereof. In one embodiment, the position sensor may include multiple accelerometers for measuring translational motion (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational motion (e.g., pitch, yaw, roll). Optionally, the electronic device may also include at least one memory.

[0138] For a more detailed description of the aforementioned eye-tracking module, depth information module, adjustment module, display module, and processor, please refer directly to the above. Figure 1 The relevant descriptions of the electronic devices in the method embodiments shown are directly obtained and will not be repeated here.

[0139] Please see Figure 7 , Figure 7 This is a schematic diagram of another electronic device disclosed in an embodiment of this application. This electronic device can be a VR / AR device, or a module (e.g., a chip) within a VR / AR device. Figure 7As shown, the electronic device 700 may include: at least one processor 701, such as a central processing unit (CPU), at least one memory 705, and at least one communication bus 702. Optionally, the electronic device 700 may also include at least one network interface 704, a user interface 703, and a display module 706. The communication bus 702 is used to enable communication between these components. Optionally, the user interface 703 may include a handheld device or a keyboard, and the network interface 704 may include a standard wired interface or a wireless interface (such as a Bluetooth interface or a Wi-Fi interface). The memory 705 may be high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 705 may also be at least one storage device located remotely from the aforementioned processor 701. Figure 7 As shown, the memory 705, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0140] exist Figure 7 In the electronic device 700 shown, the network interface 704 provides network communication functions; the user interface 703 is mainly used to provide an input interface for users.

[0141] In one embodiment, the processor 701 can be used to invoke a device control application stored in the memory 705, which can achieve the following:

[0142] Obtain the user's eye gaze area; obtain the depth information of the eye gaze area; determine the focal length value based on the depth information; adjust the focal length of the zoom lens group to the focal length value; scale the multimedia information based on the focal length value; and display the scaled multimedia information.

[0143] It should be understood that the electronic device 700 described in the embodiments of this application can be used to perform the above-described functions. Figure 1 The methods executed by the electronic device in the method embodiments can be directly referred to the relevant descriptions, and will not be elaborated in detail here.

[0144] This invention also discloses a computer-readable storage medium storing instructions thereon, which, when executed, perform the methods described in the above-described method embodiments.

[0145] This invention also discloses a computer program product including instructions that, when executed, perform the methods described in the above method embodiments.

[0146] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A display method, characterized in that, The method is applied to a virtual reality / augmented reality device, the device being equipped with a zoom lens group, and the method includes: Obtain the user's eye gaze area; Obtain depth information of the region of human eye gaze; The focal length value is determined based on the depth information; Adjust the focal length of the zoom lens group to the focal length value; The multimedia information is scaled based on the focal length value; Display scaled multimedia information; The scaling of multimedia information based on the focal length value includes: The image distance q is determined based on the focal length value and the object distance p. The virtual reality / augmented reality device is also provided with a display module. The object distance is the distance between the display module and the zoom lens group, and the image distance is the distance between the virtual imaging position and the zoom lens group. The scaling value m is calculated using the formula m=p / q; The multimedia information is scaled according to the scaling value m; The scaling of the multimedia information according to the scaling value includes: When the multimedia information is two-dimensional, the multimedia information is scaled by layers or by rendering according to the scaling value; When the multimedia information is three-dimensional, the view rendering of the multimedia information is scaled according to the scaling value.

2. The method according to claim 1, characterized in that, The area of ​​the user's eye gaze includes: The user's eye movement data is obtained based on eye-tracking technology; The user's eye gaze area is determined based on the eye movement data.

3. The method according to claim 1, characterized in that, The process of obtaining the depth information of the human eye's gaze region includes: Depth information of the area being viewed by the human eye can be obtained through real-time localization and mapping (SLAM) technology or depth sensors.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain depth information of non-focal regions; Based on the depth information of the human eye's gaze area and the non-gaze area, the multimedia information in the non-gaze area is blurred.

5. The method according to claim 4, characterized in that, The process of blurring multimedia information in the non-focused region based on depth information of the human eye's gaze region and the non-focused region includes: Determine the depth difference based on the depth information of the human eye's gaze area and the non-gaze area; The blurring value is determined based on the depth difference; The multimedia information in the non-focused region is blurred based on the blurring value.

6. A display device, characterized in that, The device is a virtual reality / augmented reality device or a module within a virtual reality / augmented reality device, wherein the virtual reality / augmented reality device is equipped with a zoom lens group, and the device includes: The first acquisition unit is used to acquire the user's eye gaze area; The second acquisition unit is used to acquire depth information of the human eye's gaze area; A determining unit is used to determine the focal length value based on the depth information; An adjustment unit is used to adjust the focal length of the zoom lens group to the focal length value. A scaling unit is used to scale multimedia information based on the focal length value; The display unit is used to display scaled multimedia information; The scaling unit scales the multimedia information based on the focal length value, including: The image distance q is determined based on the focal length value and the object distance p. The virtual reality / augmented reality device is also provided with a display module. The object distance is the distance between the display module and the zoom lens group, and the image distance is the distance between the virtual imaging position and the zoom lens group. The scaling value m is calculated using the formula m=p / q; The multimedia information is scaled according to the scaling value m; The scaling unit scaling the multimedia information according to the scaling value includes: When the multimedia information is two-dimensional, the multimedia information is scaled by layers or by rendering according to the scaling value; When the multimedia information is three-dimensional, the view rendering of the multimedia information is scaled according to the scaling value.

7. An electronic device, characterized in that, include: The system comprises an eye-tracking module, a depth information module, an adjustment module, a display module, and a processor; among which: The eye-tracking module is used to obtain the area of ​​the user's eye gaze; The depth information module is used to acquire depth information of the human eye's gaze area; The processor is used to determine the focal length value based on the depth information; The adjustment module is used to adjust the focal length of the zoom lens group to the focal length value; The processor is also configured to scale multimedia information based on the focal length value; The display module is used to display scaled-down multimedia information; The processor scaling the multimedia information based on the focal length value includes: The image distance q is determined based on the focal length value and the object distance p, where the object distance is the distance between the display module and the zoom lens group, and the image distance is the distance between the virtual imaging position and the zoom lens group. The scaling value m is calculated using the formula m=p / q; The multimedia information is scaled according to the scaling value m; The scaling of the multimedia information by the processor according to the scaling value includes: When the multimedia information is two-dimensional, the multimedia information is scaled by layers or by rendering according to the scaling value; When the multimedia information is three-dimensional, the view rendering of the multimedia information is scaled according to the scaling value.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or computer instructions that, when executed, implement the method as described in any one of claims 1-5.

Images