Display method and electronic device

Abstract

A display method and an electronic device. The electronic device displays a first image through a first display screen, displays the first image through the first display screen, the first display screen corresponding to a first eye of a user; a second display screen displays a second image, the second display screen corresponding to a second eye of the user; wherein the first image and the second image have an overlapping area, and the overlapping area includes at least one same object; on the first image, the center point of the overlapping area is located at a first position; on the second image, the center point of the overlapping area is located at a second position; the distance from the first position to the center point of the first image is not equal to the distance from the second position to the center point of the second image, and / or the direction from the first position to the center point of the first image is not equal to the direction from the second position to the center point of the second image. In this way, the assembly deviation of the display screen on the electronic device (such as a head-mounted display device) can be compensated for, and the display effect can be improved.

Description

Technical Field

[0001] This specification relates to the field of electronic technology, and in particular to a display method and electronic device. Background Technology

[0002] With the development of terminal display technology, the application scenarios of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies are becoming increasingly diverse. VR devices can simulate a three-dimensional (3D) virtual world scene and provide simulated experiences in terms of sight, hearing, touch, or other senses, making users feel as if they are actually there. Furthermore, users can interact with this simulated virtual world scene. AR devices can overlay virtual images onto the user while they view a real-world scene, and users can also interact with these virtual images to achieve augmented reality effects. MR combines AR and VR, providing users with a view that merges the real and virtual worlds.

[0003] Head-mounted displays are devices worn on a user's head that provide a new visual environment. They can emit optical signals to present different effects such as VR, AR, or MR.

[0004] Typically, head-mounted displays have two display devices: one for the left eye and the other for the right eye. The left and right eye displays show images separately. In this way, the left and right eyes separately capture the images, which are then fused by the brain to create the virtual world. However, in practical applications, users wearing head-mounted displays are prone to blurred images, dizziness, or visual fatigue, severely impacting the comfort and overall experience. Summary of the Invention

[0005] The purpose of this specification is to provide a display method and electronic device for improving the comfort of head-mounted display devices.

[0006] In a first aspect, a display method is provided, applied to an electronic device. The electronic device includes a first display screen and a second display screen. The first display screen displays a first image, corresponding to a user's first eye, and the second display screen displays a second image, corresponding to the user's second eye. An overlapping area exists between the first and second images, and the overlapping area includes at least one identical object. On the first image, the center point of the overlapping area is located at a first position. On the second image, the center point of the overlapping area is located at a second position. The distance from the first position to the center point of the first image is a first distance, and the distance from the second position to the center point of the second image is a second distance. The direction from the first position to the center point of the first image is a first direction, and the direction from the second position to the center point of the second image is a second direction. The first distance is not equal to the second distance, and / or, the first direction is different from the second direction.

[0007] Taking a head-mounted display device as an example, generally, there is an overlapping area between the image displayed on the first screen and the image displayed on the second screen of the head-mounted display device. For example, this overlapping area is symmetrical about the center line of the face (or the central plane of the electronic device), that is, the first distance from the first position (the position of the center point of the overlapping area on the first screen) to the center point of the first screen is equal to the second distance from the second position (the position of the center point of the overlapping area on the second screen) to the center point of the second screen, and the first direction from the first position to the center point of the first screen is opposite to the second direction from the second position to the center point of the second screen. Considering that assembly deviations may occur during the production of head-mounted display devices, which may cause the first screen to not align with the corresponding human eye, and / or the second screen to not align with the corresponding human eye, when the first screen and the second screen display images respectively, the overlapping areas on the first screen and the overlapping areas on the second screen are asymmetrical with respect to the center line of the face. Thus, the user cannot merge the overlapping areas on the two images.

[0008] Therefore, in this embodiment, the overlapping areas on the images of the two displays are asymmetrical to compensate for assembly deviations. For example, the first distance from the first position (the center point of the overlapping area on the first image) to the center point of the first image is not equal to the second distance from the second position (the center point of the overlapping area on the second image) to the center point of the second image, and / or, the first direction from the first position to the center point of the first image is different from the second direction from the second position to the center point of the second image. Thus, when the first display shows the first image and the second display shows the second image, the user can merge the overlapping areas on the first and second images, which helps improve the comfort of the head-mounted display.

[0009] In one possible design, the electronic device further includes a first optical component and a second optical component, the first optical component corresponding to the first display screen and the second optical component corresponding to the second display screen, the first optical component and the second optical component being symmetrical with respect to an intermediate plane; the first position and the second position being symmetrical with respect to the intermediate plane. When the first position and the second position are symmetrical with respect to the intermediate plane, the overlapping areas can be better blended, achieving a better visual effect.

[0010] In one possible design, the electronic device is a head-mounted display device. When the electronic device is worn by a user, the first position and the second position are symmetrical with respect to the center line of the user's face, which allows the overlapping areas to blend better and achieve a better visual effect.

[0011] In one possible design, the first position changes as the position of the first display screen changes. For example, if the first display screen moves in a third direction, the overlapping area on the first image moves in the opposite direction to the third direction. In another possible design, the second position changes as the position of the second display screen changes. For example, if the second display screen moves in a fourth direction, the overlapping area on the first image moves in the opposite direction to the fourth direction.

[0012] In other words, the positions of the first display screen and / or the second display screen can change dynamically. As the display screens change dynamically, the position of the overlapping area also changes dynamically to ensure that the overlapping area can be blended.

[0013] In one possible design, before displaying the first image on the first display screen and the second image on the second display screen, the method further includes adjusting the interpupillary distance of the first display screen and the second display screen, wherein the interpupillary distance adjustment includes: moving the first display screen a certain distance along a fifth direction, and moving the second display screen the same distance along a sixth direction opposite to the fifth direction; wherein the fifth direction is the direction in which the first display screen moves away from the second display screen, or the fifth direction is the direction in which the first display screen moves closer to the second display screen.

[0014] It should be noted that VR glasses with assembly deviations still exhibit these deviations after interpupillary distance (IPD) adjustment. For example, suppose IPD adjustment is performed on the VR glasses (i.e., the first and second display devices move the same distance in opposite directions, such as moving closer to each other or moving further apart). After IPD adjustment, when displaying the first and second images, the distance difference between the first and second distances remains unchanged compared to before the IPD adjustment, and the relative relationship between the first and second directions remains unchanged compared to before the IPD adjustment, ensuring that overlapping areas can be blended before and after IPD adjustment.

[0015] In one possible design, the at least one identical object includes a first object and a second object; on the first image, a first feature point of the first object is located at a first coordinate, and a second feature point of the second object is located at a second coordinate; on the second image, a first feature point of the first object is located at a third coordinate, and a second feature point of the second object is located at a fourth coordinate; the coordinate difference between the first coordinate and the third coordinate is different from the coordinate difference between the second coordinate and the fourth coordinate.

[0016] In other words, the overlapping region includes two objects, and these two objects can have different offsets. For example, the offsets of the two objects are different when at least one of the following conditions is met:

[0017] Condition 1: The first object is located within the area where the user's gaze point is located, while the second object is not located within the area where the user's gaze point is located;

[0018] Condition 2: Both the first object and the second object are located within the area where the user gaze point is located, and the second object is closer to the edge of the area where the user gaze point is located than the first object;

[0019] Condition 3: The distance between the first object and the center point of the first image is less than the distance between the second object and the center of the second image;

[0020] Condition 4: The number of user interactions corresponding to the first object is greater than the number of user interactions corresponding to the second object;

[0021] Condition 5: The first object is a user-specified object, and the second object is not a user-specified object.

[0022] In this embodiment, the second object is an object that the user pays little attention to or is not interested in (with low interaction frequency). Therefore, the offset of the second object is small, or the second object is not offset at all, which will not affect the user's perception and can save the computing power of electronic devices and improve efficiency.

[0023] In one possible design, the method further includes: the electronic device includes a first display module and a second display module; the first display module includes a first display screen and a first optical device, the second display module includes a second display screen and a second optical device, a first offset exists between the position of the first display screen and the position of the first optical device, and a second offset exists between the position of the second display screen and the position of the second optical device; the method further includes: acquiring three-dimensional image data; acquiring a first coordinate transformation matrix and a second coordinate transformation matrix, the first coordinate transformation matrix corresponding to the first optical device and the second coordinate transformation matrix corresponding to the second optical device; acquiring the first offset and the second offset; processing the three-dimensional image data into a first image based on the first coordinate transformation matrix and the first offset; and processing the three-dimensional image data into a second image based on the second coordinate transformation matrix and the second offset. It is understood that the first image and the second image are taken from the same three-dimensional image data and transformed according to the positions of the first optical device (corresponding to the position of the first human eye) and the second optical device (corresponding to the position of the second human eye), which helps to fuse the overlapping areas on the first image and the second image, ensuring that the user can clearly see the virtual environment.

[0024] In one possible design, the first coordinate transformation matrix changes when the position of the first display module changes; or, the second coordinate transformation matrix changes when the position of the second display module changes. When the position of the human eye changes, the display module typically changes accordingly, thus adjusting the viewing angle of the display screen based on the change in the human eye's position.

[0025] Secondly, a calibration method is also provided, applied to a calibration device, the calibration device including an image capturing module. The method includes: when a first display screen of an electronic device to be calibrated displays a first image and a second display screen displays a second image, the image capturing module captures a third image of the first display screen and a fourth image of the second display screen; wherein the first image and the second image have overlapping areas, the overlapping areas include at least one identical object, the at least one identical object including a calibration object, and the center point of the overlapping area on the first image is located at a first position, and the center point of the overlapping area on the second image is located at a second position; the distance from the first position to the center of the first image is equal to the distance from the second position to the center of the second image, and the direction from the first position to the center of the first image is equal to the direction from the second position to the center of the second image; the third image and the fourth image are fused to obtain a fifth image, the fifth image including two calibration objects; a first offset of the first display screen and / or a second offset of the second display screen are determined based on the distance difference between the two calibration objects.

[0026] In this way, the assembly deviation of the electronic device, that is, the offset between the two display devices, can be calibrated so that the electronic device can compensate for the assembly deviation.

[0027] In one possible design, the first offset includes a first displacement and a first direction, and the second offset includes a second displacement and a second direction; wherein the sum of the first displacement and the second displacement is equal to the distance difference; and the first direction is opposite to the second direction.

[0028] In one possible design, the first displacement is half of the distance difference, and the second displacement is half of the distance difference.

[0029] In one possible design, the method further includes: writing the first offset and the second offset into the electronic device to be calibrated, so that the electronic device to be calibrated processes the image displayed on the first display screen based on the first offset, and processes the image displayed on the second display screen based on the second offset.

[0030] Thirdly, a display method is also provided. The electronic device includes a first display module and a second display module. The first display module includes a first display screen and a first optical device, and the second display module includes a second display screen and a second optical device. A first offset exists between the position of the first display screen and the position of the first optical device, and a second offset exists between the position of the second display screen and the position of the second optical device. The method further includes: acquiring three-dimensional image data; acquiring a first coordinate transformation matrix and a second coordinate transformation matrix, the first coordinate transformation matrix corresponding to the first optical device and the second coordinate transformation matrix corresponding to the second optical device; acquiring the first offset and the second offset; processing the three-dimensional image data into a first image based on the first coordinate transformation matrix and the first offset, and displaying the first image on the first display module; and processing the three-dimensional image data into a second image based on the second coordinate transformation matrix and the second offset, and displaying the second image on the second display module. It is understood that the first image and the second image are taken from the same three-dimensional image data and transformed according to the positions of the first optical device (corresponding to the position of the first human eye) and the second optical device (corresponding to the position of the second human eye), respectively. This helps to fuse the overlapping areas on the first image and the second image, ensuring that the user can clearly see the virtual environment.

[0031] In one possible design, the first coordinate transformation matrix changes when the position of the first display module changes; or, the second coordinate transformation matrix changes when the position of the second display module changes. When the position of the human eye changes, the display module typically changes accordingly, thus adjusting the viewing angle of the display screen based on the change in the human eye's position.

[0032] Fourthly, an electronic device is also provided, comprising:

[0033] Processor, memory, and one or more programs;

[0034] The one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the processor, cause the electronic device to perform the method steps provided in the first aspect above.

[0035] Fifthly, a calibration device is also provided, comprising:

[0036] Processor, memory, and one or more programs;

[0037] The one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the processor, cause the electronic device to perform the method steps provided in the second aspect above.

[0038] Sixthly, a system is also provided, comprising:

[0039] An electronic device for performing the steps of the method as described in the first aspect above, and,

[0040] Used to perform the calibration apparatus as described in the second aspect above.

[0041] In a seventh aspect, a computer-readable storage medium is also provided for storing a computer program that, when run on a computer, causes the computer to perform the method described in the first or second aspect above.

[0042] Eighthly, a computer program product is also provided, comprising a computer program that, when run on a computer, causes the computer to perform the method described in the first or second aspect above. Attached Figure Description

[0043] Figure 1 A schematic diagram of a VR system provided in one embodiment of this specification;

[0044] Figure 2A and Figure 2B This is a schematic diagram of a VR head-mounted display device provided in one embodiment of this specification;

[0045] Figure 3A This is a schematic diagram of another structure of the VR head-mounted display device provided in one embodiment of this specification;

[0046] Figure 3B This is a schematic diagram of the software structure of a VR head-mounted display device provided in one embodiment of this specification;

[0047] Figures 4A to 4B A schematic diagram of the human eye observation mechanism provided in one embodiment of this specification;

[0048] Figure 5 This is a schematic diagram illustrating the display principle of VR glasses provided in one embodiment of this specification;

[0049] Figures 6A to 6B A schematic diagram illustrating binocular non-fusion as provided in one embodiment of this specification;

[0050] Figures 7A to 7B Another schematic diagram illustrating binocular non-fusion as an embodiment of this specification;

[0051] Figures 8A to 8B This is a schematic diagram illustrating a display method provided in one embodiment of this specification;

[0052] Figures 9 to 10This is a schematic diagram illustrating another display method provided in one embodiment of this specification;

[0053] Figure 11 This is a schematic flowchart of a display method provided in one embodiment of this specification;

[0054] Figures 12A to 12B This is a schematic diagram illustrating the process of obtaining a two-dimensional image from a three-dimensional image provided in one embodiment of this specification.

[0055] Figures 12C to 13 A schematic diagram illustrating a calibration method provided in one embodiment of this specification;

[0056] Figure 14 This is a schematic diagram of another display method provided in one embodiment of this specification;

[0057] Figure 15 This is a schematic diagram illustrating an embodiment of the present specification with assembly deviations.

[0058] Figure 16 This is a schematic diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0059] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0060] The embodiments of this application involve at least one, including one or more; where "multiple" means two or more. Furthermore, it should be understood that in the description of this specification, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order. For example, "first object" and "second object" do not represent the degree of importance or order between them, but are merely for descriptive distinction. In the embodiments of this application, "and / or" merely describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0061] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation" and "connection" should be interpreted broadly. For example, "connection" can be a detachable connection or a non-detachable connection; it can be a direct connection or an indirect connection through an intermediate medium. The directional terms mentioned in the embodiments of this application, such as "upper," "lower," "left," "right," "inner," and "outer," are only for reference to the directions in the accompanying drawings. Therefore, the directional terms used are for better and clearer explanation and understanding of the embodiments of this application, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application. "Multiple" refers to at least two.

[0062] References to "one embodiment" or "some embodiments" as used in this specification mean that one or more embodiments of this specification include a particular feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0063] Virtual Reality (VR) technology is a human-computer interaction method created using computer and sensor technologies. VR technology integrates various scientific technologies such as computer graphics, computer simulation, sensor technology, and display technology to create virtual environments. These virtual environments include computer-generated, dynamically displayed two-dimensional or three-dimensional virtual objects, providing users with simulations of their senses, such as vision, making them feel as if they are actually there. Moreover, in addition to visual perception generated by computer graphics, it also includes auditory, tactile, force, and motion perceptions, and even smell and taste, also known as multi-sensory perception. Furthermore, it can detect the user's head movements, eye movements, gestures, or other human behaviors. The computer processes the data corresponding to the user's actions and responds to them in real time, providing feedback to the user's five senses, thus forming the virtual environment. For example, when a user wears a VR headset (such as VR glasses or a VR helmet), they can see a VR game interface and interact with it through gestures, controllers, etc., as if they were actually in the game.

[0064] Augmented Reality (AR) technology refers to superimposing computer-generated virtual objects onto real-world scenes to enhance the real world. In other words, AR technology requires capturing real-world scenes and then adding a virtual environment to them. Therefore, the difference between VR and AR technologies is that VR creates a completely virtual environment where users see only virtual objects; while AR overlays virtual objects onto the real world, allowing users to see both real and virtual objects. For example, a user wearing transparent glasses can see their surroundings through the lenses, which can also display virtual objects, thus allowing them to see both real and virtual objects.

[0065] Mixed Reality (MR) technology enhances the realism of the user experience by introducing real-world scene information (or authentic scene information) into a virtual environment, creating an interactive feedback bridge between the virtual environment, the real world, and the user. Specifically, it virtualizes real-world objects (for example, using a camera to scan real-world objects and perform 3D reconstruction to generate virtual objects), and then introduces these virtualized real objects into the virtual environment, allowing users to see the real objects within the virtual environment.

[0066] Binocular fusion, also known as binocular image fusion, is a visual phenomenon. It occurs when both eyes simultaneously observe the same object, forming two images of that object on their respective retinas. These images are then transmitted via the optic nerves of both eyes to the same area of ​​the visual cortex, where they fuse to form a complete, single image.

[0067] In this article, virtual images or virtual environments can include various objects, also known as targets. Objects can include things or objects that can appear in the real world environment, such as people, animals, or furniture. Objects can also include virtual elements such as virtual icons, navigation bars, software buttons, or windows, which can be used to interact with the user.

[0068] It should be noted that the technical solutions provided in this application embodiment can be applied to head-mounted display devices such as VR, AR, or MR; or, they can also be applied to other scenes or electronic devices that need to display a three-dimensional environment to the user, in addition to VR, AR, and MR. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0069] For ease of understanding, the following text will mainly use VR head-mounted display devices as an example.

[0070] For example, the VR head-mounted display device 100 can be applied to, for example, Figure 1 The VR system shown includes a VR headset 100 and a processing device 200. This VR system can be referred to as a VR split-type device. The VR headset 100 can be connected to the processing device 200. The connection between the VR headset 100 and the processing device 200 can be wired or wireless. The wireless connection can be Bluetooth (BT), traditional Bluetooth or Bluetooth Low Energy (BLE), wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Zigbee, frequency modulation (FM), near field communication (NFC), infrared (IR), or general 2.4G / 5G frequency band wireless communication connections, etc.

[0071] In some embodiments, the processing device 200 can perform processing calculations. For example, the processing device 200 can generate an image and process the image (the processing method will be described later), and then send the processed image to a VR head-mounted display device for display. The processing device 200 may include a host (e.g., a VR host) or a server (e.g., a VR server). The VR host or VR server can be a device with significant computing power. For example, a VR host can be a mobile phone, tablet, laptop, or other device, and a VR server can be a cloud server, etc.

[0072] In some embodiments, the VR head-mounted display device 100 can be glasses, a helmet, etc. The VR head-mounted display device 100 generally has two display devices, namely display device 110 and display device 120. The display devices of the VR head-mounted display device 100 can display images to the human eye. Figure 1 In the illustrated embodiment, display devices 110 and 120 are enclosed inside the VR glasses, so Figure 1 The arrows used to indicate display device 110 and display device 120 are represented by dashed lines.

[0073] In some embodiments, the VR head-mounted display device 100 also has image generation and processing functions, that is, the VR head-mounted display device 100 does not require Figure 1 The processing device 200 in the VR headset 100 can be called a VR all-in-one machine.

[0074] Please see Figure 2A This is a schematic diagram of a VR head-mounted display device 100. Figure 2AAs shown, the VR headset 100 includes a display module 1 and a display module 2. Display module 1 includes a display device 110 and an optical device 130. Display module 2 includes a display device 120 and an optical device 140. Display modules 1 and 2 can also be referred to as lens tubes. When a user wears the VR headset 100, display module 1 displays the image to the user's right eye. Display module 2 displays the image to the user's left eye. It is understood that... Figure 2A The VR head-mounted display device 100 shown may also include other components, such as a support 30 and a bracket 20, wherein the support 30 is used to support the VR head-mounted display device 100 on the bridge of the nose, and the bracket 20 is used to support the VR head-mounted display device 100 on the ears to ensure that the VR head-mounted display device 100 is worn stably.

[0075] In some embodiments, optical devices 130 and 140 are symmetrical with respect to the intermediate plane C. Figure 2A In the diagram, the central plane C is a plane perpendicular to the paper. In some embodiments, the VR head-mounted display device 100 can be a left-right symmetrical structure, and the support 30 and / or the bracket 20 can be left-right symmetrical with respect to the central plane C respectively. The support 30 can fix the position of the face, which is beneficial for the optical devices 130 and 140 to align with the user's left and right eyes respectively.

[0076] When the VR headset 100 is worn by a user, optical components 130 and 140 are aligned with the user's left and right eyes, respectively. Generally, the human face is basically symmetrical, with the left and right eyes symmetrical relative to the center line of the face. When the VR headset 100 is worn by a user, the left and right eyes are symmetrical with respect to the central plane C, and optical components 130 and 140 are symmetrical with respect to the center line of the face. The center line of the face lies within the central plane C, meaning it overlaps with the central plane C.

[0077] In the embodiments of this application, "symmetry" can be strict symmetry or it can have a slight deviation. For example, optical devices 130 and 140 can be strictly symmetrical with respect to the intermediate plane C, or they can be substantially symmetrical with respect to the intermediate plane C. The substantial symmetry can have a certain deviation, which is within a small range.

[0078] For ease of description, please refer to Figure 2B , Figure 2B This can be understood as... Figure 2A A simplified version of the VR head-mounted display device 100, for example, Figure 2B Only display module 1 and display module 2 are shown; other components are not shown. Figure 2BWhen a user wears the VR headset 100, the display device 120 is located on the side opposite to the left eye of the optical device 140, and the display device 110 is located on the side opposite to the right eye of the optical device 130. The optical devices 130 and 140 are symmetrical with respect to the center line D of the human face. When the display device 110 is displaying an image, the light emitted by the display device 110 is converged to the user's right eye through the optical device 130. When the display device 120 is displaying an image, the light emitted by the display device 120 is converged to the user's left eye through the optical device 140.

[0079] It should be noted that, Figure 2A or Figure 2B The VR head-mounted display device 100 shown is only a logical illustration. In specific implementations, the number of optical devices and / or display devices can be flexibly set according to different needs. For example, in some embodiments, display device 110 and display device 120 can be two independent display devices, or two display areas on the same display device. In some embodiments, display device 110 and display device 120 can be displays, such as liquid crystal displays, light-emitting diode (LED) displays, or other types of display devices, which are not limited in this application embodiment. In other embodiments, optical device 130 and optical device 140 can be two independent optical devices, or different parts of the same optical device. In some embodiments, optical device 130 or 140 can be one or more optical devices such as reflectors, transmissive mirrors, or optical waveguides, which can also improve the field of view. For example, optical device 130 or 140 can be eyepieces composed of multiple transmissive mirrors. For example, optical devices can be Fresnel lenses and / or aspherical lenses, which are not limited in this application embodiment. Normally, optical devices 130 and 140 are aligned with the user's two eyes respectively. When IPD adjustment is performed, the two optical devices are adjusted to the same distance but in opposite directions.

[0080] To facilitate understanding of the technical solutions in this specification, the following text mainly uses... Figure 2B Taking the VR head-mounted display device 100 as an example for introduction, but Figure 2A This technical solution can also be implemented in VR head-mounted display devices. Figure 2B Only for Figure 2A A simplification).

[0081] It is understandable that the VR head-mounted display device 100 may include more components; see details below. Figure 3A .

[0082] For example, please refer to Figure 3AThis is a schematic diagram of the structure of a head-mounted display device 100 provided in an embodiment of this application. The head-mounted display device 100 can be a VR head-mounted display device, an AR head-mounted display device, a MR head-mounted display device, etc. Taking a VR head-mounted display device as an example, ... Figure 3A As shown, the VR head-mounted display device 100 may include a processor 101, a memory 102, a sensor module 103 (e.g., which can be used to acquire the user's posture), a microphone 104, a button 150, an input / output interface 160, a communication module 170, a camera 180, a battery 190, an optical display module 106, and an eye-tracking module 105, etc.

[0083] Processor 101 is typically used to control the overall operation of VR head-mounted display device 100, and may include one or more processing units. For example, processor 101 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0084] The processor 101 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. This memory can store instructions or data that the processor 101 has just used or that are used repeatedly. If the processor 101 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 101, and thus improves the efficiency of the system.

[0085] In some embodiments of this specification, the processor 101 can be used to control the optical power of the VR head-mounted display device 100. For example, the processor 101 can be used to control the optical power of the optical display module 106, thereby adjusting the optical power of the head-mounted display device 100. For instance, the processor 101 can adjust the relative positions of the various optical components (such as lenses) in the optical display module 106, thereby adjusting the optical power of the optical display module 106, and consequently adjusting the position of the corresponding virtual image plane when the optical display module 106 images onto the human eye. This achieves the effect of controlling the optical power of the head-mounted display device 100.

[0086] In some embodiments, the processor 101 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0087] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 101 may include multiple I2C buses.

[0088] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 101 and the communication module 170. For example, the processor 101 communicates with the Bluetooth module in the communication module 170 via the UART interface to implement Bluetooth functionality.

[0089] The MIPI interface can be used to connect the processor 101 to peripheral devices such as the display device and camera 180 in the optical display module 106.

[0090] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 101 to the camera 180, the display device in the optical display module 106, the communication module 170, the sensor module 103, the microphone 104, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. In some embodiments, the camera 180 can capture images including real objects, and the processor 101 can fuse the captured images with virtual objects, displaying the fused image through the optical display module 106. In some embodiments, the camera 180 can also capture images including human eyes. The processor 101 uses these images for eye tracking.

[0091] The USB interface conforms to the USB standard specification and can be a Mini USB interface, Micro USB interface, USB Type-C interface, etc. The USB interface can be used to connect a charger to charge the VR headset 100, and can also be used for data transfer between the VR headset 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as mobile phones. The USB interface can be USB 3.0, used for compatibility with high-speed display port (DP) signal transmission, enabling the transmission of high-speed audio and video data.

[0092] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the head-mounted display device 100. In other embodiments of this specification, the head-mounted display device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0093] Additionally, the VR headset 100 may include wireless communication functionality. For example, the VR headset 100 may receive and display images from other electronic devices (such as a VR host), or it may directly acquire data from base stations or other sites. The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication functionality can be implemented using an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The VR headset 100 may include multiple antennas, each of which can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0094] The mobile communication module can provide wireless communication solutions for VR head-mounted display devices 100, including 2G, 3G, 4G, 5G, and 6G networks. The mobile communication module may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves via an antenna, filter and amplify the received electromagnetic waves, and transmit them to a modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor and radiate it as electromagnetic waves via the antenna. In some embodiments, at least some functional modules of the mobile communication module may be housed in the processor 101. In some embodiments, at least some functional modules of the mobile communication module and at least some modules of the processor 101 may be housed in the same device.

[0095] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to a speaker), or displays images or videos through a display device in the optical display module 106. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 101 and may be housed in the same device as the mobile communication module or other functional modules.

[0096] The wireless communication module can provide solutions for wireless communication applications in the VR head-mounted display device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signal to the processor 101. The wireless communication module can also receive signals to be transmitted from the processor 101, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0097] In some embodiments, the antenna of the VR headset 100 is coupled to the mobile communication module, enabling the VR headset 100 to communicate with networks and other devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), 5G, 6G, BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0098] The VR head-mounted display device 100 implements display functions through a GPU, an optical display module 106, and an application processor. The GPU is a microprocessor for image processing, connected to the optical display module 106 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 101 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0099] The memory 102 can be used to store computer executable program code, which includes instructions. The processor 101 executes various functional applications and data processing of the VR head-mounted display device 100 by running the instructions stored in the memory 102. The memory 102 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of the head-mounted display device 100 (such as audio data, phone book, etc.). In addition, the memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0100] The VR headset 100 can implement audio functions through an audio module, speakers, a microphone 104, a headphone jack, and an application processor. Examples include music playback and recording. The audio module is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module can also be used for encoding and decoding audio signals. In some embodiments, the audio module can be located in the processor 101, or some functional modules of the audio module can be located in the processor 101. The speaker, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. The headset 100 can listen to music or make hands-free calls through the speaker.

[0101] Microphone 104, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. The VR headset 100 may be equipped with at least one microphone 104. In some embodiments, the VR headset 100 may be equipped with two microphones 104, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, the VR headset 100 may also be equipped with three, four, or more microphones 104, enabling sound signal collection, noise reduction, sound source identification, and directional recording, among other functions.

[0102] The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface, or a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0103] In some embodiments, the VR headset 100 may include one or more buttons 150 that can control the VR headset and provide users with the ability to interact with it. The buttons 150 may take the form of buttons, switches, dials, and touch or proximity sensors (such as touch sensors). Specifically, for example, a user can press a button to turn on the optical display module 106 of the VR headset 100. Buttons 150 may include a power button, volume buttons, etc. Buttons 150 may be mechanical buttons or touch buttons. The headset 100 can receive button input and generate key signal inputs related to user settings and function control of the headset 100.

[0104] In some embodiments, the VR headset 100 may include an input / output interface 160, which can connect other devices to the VR headset 100 via suitable components. Components may include, for example, audio / video jacks, data connectors, etc.

[0105] The optical display module 106, under the control of the processor 101, presents images to the user. The optical display module 106 can use one or more optical devices, such as mirrors, transmissive mirrors, or optical waveguides, to convert a real-pixel image display into a near-eye projection virtual image display, achieving a virtual interactive experience or a combined virtual and real interactive experience. For example, the optical display module 106 receives image data information sent by the processor 101 and presents the corresponding image to the user.

[0106] For example, the optical display module 106 can be referred to above. Figure 2A The structure shown, for example, includes two display devices, namely display device 110 and display device 120, in the optical display module 106. Alternatively, the optical display module 106 can also be referred to in the preceding text. Figure 2B The structure shown, for example, includes display module 1 and display module 2 in optical display module 106. Display module 1 includes display device 110 and optical device 130, and display module 2 includes display device 120 and optical device 140.

[0107] In some embodiments, the VR head-mounted display device 100 may further include an eye-tracking module 1200, which tracks the movement of the human eye to determine the gaze point. For example, image processing technology can be used to locate the pupil position, obtain the pupil center coordinates, and then calculate the gaze point. In some embodiments, the eye-tracking system can determine the user's gaze point position (or determine the user's gaze direction) using methods such as video eye diagrams, photodiode response methods, or pupil-corneal reflection methods, thereby achieving eye tracking.

[0108] In some embodiments, the user's gaze direction is determined using the pupillary corneal reflex method. The eye-tracking system may include one or more near-infrared light-emitting diodes (LEDs) and one or more near-infrared cameras. The near-infrared LEDs and near-infrared cameras are not... Figure 3A As shown in the diagram. In different examples, the near-infrared LED can be positioned around the optics to provide comprehensive illumination of the human eye. In some embodiments, the center wavelength of the near-infrared LED can be 850 nm or 940 nm. The eye-tracking system can obtain the user's gaze direction by illuminating the human eye with a near-infrared LED, capturing an image of the eyeball with a near-infrared camera, and then determining the optical axis direction of the eyeball based on the position of the reflective point of the near-infrared LED on the cornea and the center of the pupil in the eyeball image, thereby obtaining the user's gaze direction.

[0109] It should be noted that in some embodiments of this specification, separate eye-tracking systems can be set up for each of the user's eyes to perform eye tracking synchronously or asynchronously. In other embodiments of this specification, an eye-tracking system can be set up only near one of the user's eyes. The eye-tracking system obtains the gaze direction of the corresponding eye, and based on the relationship between the fixation points of the two eyes (e.g., when a user observes an object through both eyes, the fixation points of the two eyes are generally close or the same), combined with the user's interocular distance, the gaze direction or fixation point position of the user's other eye can be determined.

[0110] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the VR head-mounted display device 100. In other embodiments of this specification, the VR head-mounted display device 100 may include more than Figure 3A The embodiments of this application do not limit the number of components, the combination of certain components, the separation of certain components, or the different arrangement of components.

[0111] Figure 3B This is a software structure block diagram of a VR head-mounted display device 100 according to an embodiment of this application.

[0112] like Figure 3B As shown, the software structure of the VR head-mounted display device 100 can be a layered architecture. For example, the software can be divided into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, it is divided into five layers, from top to bottom: application layer 210, application framework layer (framework, FWK) 220, Android runtime 230 and system library 240, kernel layer 250 and hardware layer 260.

[0113] The application layer 210 may include a series of application packages. For example, such as... Figure 3B As shown, the application layer includes Gallery 211 application, Game 212 application, and so on.

[0114] The application framework layer 220 provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer may include some predefined functions. For example... Figure 3B As shown, the application framework layer may include a resource manager 221, a view system 222, etc. For example, the view system 222 includes visual controls, such as controls for displaying text and controls for displaying images. The view system 222 can be used to build the application. The display interface can consist of one or more views. For example, a display interface including message notification icons may include views for displaying text and views for displaying images. The resource manager 221 provides the application with various resources, such as localized strings, icons, images, layout files, video files, etc.

[0115] Android Runtime 230 consists of the core libraries and the virtual machine. Android Runtime 230 is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that Java needs to call, and the other part contains the core Android libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0116] System library 240 may include multiple functional modules, such as a surface manager 241, media libraries 242, a 3D graphics processing library (e.g., OpenGL ES) 243, and a 2D graphics engine 244 (e.g., SGL). The surface manager 241 manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media libraries support playback and recording of various common audio and video formats, as well as still image files. Media library 242 supports various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing library 243 is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine 244 is a 2D drawing engine.

[0117] Furthermore, the system library 240 may also include a VR algorithm integration module 245. The VR algorithm integration module 245 includes a first offset of the first display device, a second offset of the second display device, a coordinate transformation matrix, and related algorithms for coordinate transformation based on the coordinate transformation matrix, etc. The first offset, second offset, coordinate transformation matrix, etc., will be described in detail later. It should be noted that... Figure 3B Taking the VR algorithm integration module 245 located in the system library as an example, it can be understood that the VR algorithm integration module 245 can also be located in other layers, such as the application framework layer 220. This application embodiment does not limit this.

[0118] Kernel layer 250 is the layer between hardware and software. Kernel layer 250 includes at least display driver 251, camera driver 252, audio driver 253, sensor driver 254, etc.

[0119] The hardware layer may include a first display device 110, a second display device 120, and various sensor modules, such as an accelerometer 201, a gravity sensor 202, a touch sensor 203, etc.

[0120] Understandable, Figure 3B The software structure shown does not constitute a specific limitation on the software structure of the VR head-mounted display device 100. For example, in other embodiments, the software structure of the VR head-mounted display device 100 may include more than... Figure 3B More or fewer layers, such as an adaptation layer located between the application framework layer 220 and the system library 240, are used to implement the adaptation between the upper layer (i.e., the application framework layer) and the lower layer (i.e., the system library), for example, to implement interface matching between the upper and lower layers to ensure that the upper and lower layers can communicate with each other.

[0121] by Figure 3B Taking the software structure shown as an example, an exemplary flow of the display method provided in this application includes:

[0122] After the game 212 application in the application layer 210 starts, it calls the system library 240 through the framework layer 220. The system library 240 converts the 3D image generated by the game 212 application into a first planar image and a second planar image, where the first planar image corresponds to the first display device 110 and the second planar image corresponds to the second display device 120. The system library obtains the first offset and the second offset from the VR algorithm integration module 240, uses the first offset to process the first planar image (e.g., coordinate transformation) to obtain a third planar image, and uses the second offset to process the second planar image (e.g., coordinate transformation) to obtain a fourth planar image. The system library 240 drives the first display device 110 to display the third planar image and drives the second display device 120 to display the fourth planar image through the display driver 251 in the kernel layer, so as to display the virtual environment to the user through the third and fourth planar images.

[0123] For ease of description, the following text will use VR head-mounted display device 100, which is VR glasses, as an example.

[0124] In order to clearly explain the technical solution of this specification, the mechanism of human vision generation will be briefly explained below.

[0125] Figure 4A A schematic diagram of the components of the human eye. (For example...) Figure 4A The human eye consists of the lens, ciliary muscle, and retina at the back of the eye. The lens acts as a focusing lens, converging incoming light onto the retina to create a clear image of objects in a scene. The ciliary muscle adjusts the lens's shape; by contracting or relaxing, it regulates the lens's refractive power, thus adjusting its focal length. This allows objects at varying distances to be clearly imaged on the retina.

[0126] In the real world, when a user (without wearing VR glasses or other head-mounted electronic devices) views an object, the perspectives of the left and right eyes are different, so the images captured by the left and right eyes are different. Generally, there is an overlap in the field of view of the left and right eyes, so there is an overlapping area in the images captured by the left and right eyes. The overlapping area includes the images of objects located within the overlapping field of view of the user's two eyes.

[0127] For example, please see Figure 4BThe real-world environment 400 includes multiple observed objects, such as a tree 410, a soccer ball 420, and a dog 430. When a user observes the real-world environment 400, assuming the soccer ball 420 is within the field of view of the left eye but not the field of view of the right eye, the tree 410 is within the overlapping area 440 of the fields of view of the left and right eyes, and the dog 430 is within the field of view of the right eye but not the field of view of the left eye, then the left eye captures image 4100, which is the image formed on the retina of the left eye. The right eye captures image 4200, which is the image formed on the retina of the right eye. There is an overlapping area between images 4100 and 4200. The overlapping area includes the images of objects within the overlapping area 440 of the field of view. For example, the overlapping area on image 4100 is region 4110, and the overlapping area on image 4200 is region 4210. In this image, region 4110 includes image 4101 of tree 410, and region 4210 includes image 4201 of tree 410 (because tree 410 is within the overlapping field of view of the left and right eyes). The non-overlapping region 4120 of image 4100 includes image 4102 of soccer ball 420 (because soccer ball 420 is within the field of view of the left eye). Since dog 430 is not within the field of view of the left eye, image 4100 does not include an image of dog 430. The non-overlapping region 4220 of image 4200 includes image 4202 of dog 430 (because dog 430 is within the field of view of the right eye). Since soccer ball 420 is not within the field of view of the right eye, image 4200 does not include an image of soccer ball 420.

[0128] It can be understood that the center point Q1 of image 4100 is aligned with the center W1 of the left eyeball; that is, center point Q1 and the center W1 of the left eyeball lie on the same straight line K1, which passes through the center W1 of the left eyeball and is perpendicular to the left eyeball. The center W1 of the left eyeball can be understood as the center of the left pupil. Similarly, the center point Q2 of image 4200 is aligned with the center W2 of the right eyeball; that is, Q2 and W2 lie on the same straight line K2, which passes through the center W2 of the right eyeball and is perpendicular to the right eyeball. The center W2 of the right eyeball can be understood as the center of the right pupil.

[0129] Continue as Figure 4BThe distance from the center point P of the visual field overlap region 440 of both eyes to line K1 is L1, and the distance to line K2 is L2. Distance L1 equals distance L2, and the direction from center point P to line K1 is opposite to the direction to line K2. Correspondingly, the overlap region 4110 includes center point P1, which is the image point corresponding to point P in image 4100. The overlap region 4210 includes center point P2, which is the image point corresponding to point P in image 4200. Therefore, the distance from point P1 to center point Q1 of image 4100 is L1', and the distance from point P2 to center point Q2 of image 4200 is L2'. Distance L1' equals distance L2', and the direction from point P1 to point Q1 is opposite to the direction from point P2 to point Q2. Center points P1 and P2 are symmetrical with respect to the center line D of the face, that is, center points P1 and P2 are also symmetrical with respect to the intermediate plane. In the embodiments of this application, the center point of the image can be understood as the exact center of the image in the vertical direction and the exact center of the image in the horizontal direction. Similarly, the center point of the overlapping region can be understood as the exact center of the overlapping region in the vertical direction and the exact center of the overlapping region in the horizontal direction.

[0130] After the left eye acquires image 4100 and the right eye acquires image 4200, the brain fuses images 4100 and 4200 to obtain a single image, which is the image the user actually sees. This fusion of images 4100 and 4200 includes the fusion of images within overlapping regions 4110 and 4210. For example, the images of the same observed object within overlapping regions 4110 and 4210 will be merged into one; for instance, images 4101 and 4201 will be merged into one. This way, the user sees an image including a tree, consistent with the real-world environment.

[0131] Under this human visual mechanism, the overlapping areas of the images seen by the left and right eyes can be merged (i.e., binocular fusion can be achieved), so the scene seen by the user is clear and there will be no ghosting (such as the image of a tree being ghosted), and the human eye is in a comfortable state.

[0132] In some embodiments, head-mounted display devices such as VR glasses utilize the aforementioned human visual generation mechanism to display a virtual environment to the user.

[0133] For ease of comparison, the virtual environment displayed to the user by VR glasses is used as an example. Figure 4B Taking environment 400 as an example, the two display devices of the VR glasses each display an image. These two images include images of various objects in environment 400 (such as tree 410, football 420, dog 430). In this way, the user can experience environment 400 through these two images.

[0134] For example, such as Figure 5The VR glasses generate two images, image 5100 and image 5200. Images 5100 and 5200 include images of various objects in the environment 400 (such as tree 410, soccer ball 420, dog 430). Images 5100 and 5200 have overlapping regions. The overlapping region on image 5100 is region 5110, and the overlapping region on image 5200 is region 5210. Objects in region 5110 are all contained within region 5210, and vice versa. Image 5100 is displayed on display device 120. Image 5200 is displayed on display device 110. When display device 120 displays image 5100, the center point R1 of image 5100 is aligned with the center point S1 of display device 120, meaning R1 and S1 lie on the same straight line K3, which is a line passing through the center point S1 of display device 120 and perpendicular to display device 120. When display device 110 displays image 5200, the center point R2 of image 5200 is aligned with the center point S2 of display device 110, meaning R2 and S2 lie on the same straight line K4, which is a line passing through the center point S2 of display device 110 and perpendicular to display device 110. The distance from the center point P3 of the overlapping area 5110 on image 5100 to straight line K3 is L3. The distance from the center point P4 of the overlapping area 5210 on image 5200 to straight line K4 is L4. Distance L3 is equal to distance L4, and the direction from center point P3 to straight line K3 is opposite to the direction from center point P4 to straight line K4.

[0135] When a user wears VR glasses, the center point W1 of the user's left eyeball aligns with the center point S1 of the display device 120, capturing image 5300. That is, the center point T1 of image 5300 is aligned with the center point S1 of the display device. In other words, the center point R1 of image 5100, the center point S1 of display device 120, the center point W1 of the user's left eyeball, and the center point T1 of image 5300 are all on the same straight line K3. Specifically, point T1 on image 5300 is the image point corresponding to R1 on image 5100.

[0136] When a user wears VR glasses, the center point W2 of the user's right eyeball is aligned with the center point S2 of the display device 110, capturing image 5400. That is, the center point T2 of image 5400 is aligned with the center point S2 of the display device 110. In other words, the center point R2 of image 5200, the center point S2 of display device 110, the center point W2 of the user's right eyeball, and the center point T2 of image 5400 are all on the same straight line K4. Specifically, point T2 on image 5400 is the image point corresponding to R2 on image 5200.

[0137] The images 5300 and 5400 acquired by the left eye and right eye respectively contain overlapping regions. The overlapping region in image 5300 is region 5310. The overlapping region in image 5400 is also region 5410. Overlapping region 5310 includes a center point P3', which is the image point corresponding to point P3 in image 5100 on image 5300. Overlapping region 5410 also includes a center point P4', which is the image point corresponding to point P4 in image 5200 on image 5400. The distance from point P3' to line K3 is L3'. The distance from point P4' to line K4 is L4'. Distance L3' is equal to distance L4', and the direction from point P3' to line K3 is opposite to the direction from point P4' to line K4. Center points P3' and P4' are symmetrical with respect to the center line D of the face, meaning they are also symmetrical with respect to the intermediate plane. In this way, the brain can fuse image 5300 with image 5400 to obtain... Figure 4B The environment shown in image 400 simulates a real environment, and because images 5300 and 5400 can be fused, the user's eyes are comfortable. In other words, when a user wears VR glasses, the images captured by both eyes can be fused, and the eyes are comfortable.

[0138] It should be noted that, in Figure 5 In the illustrated embodiment, taking the alignment of the center point W1 of the user's left eyeball with the center point S1 of the display device 120 and the alignment of the center point W2 of the user's right eyeball with the center point S2 of the display device 110 as an example, in some embodiments, there may be situations where the center point of at least one eyeball cannot be aligned with the center point of the corresponding display device. For example, there are three situations: Situation 1: The center point W1 of the left eyeball can be aligned with the center point S1, but the center point W2 of the right eyeball cannot be aligned with the center point S2. Situation 2: The center point W1 of the left eyeball cannot be aligned with the center point S1, but the center point W2 of the right eyeball can be aligned with the center point S2. Situation 3: The center point W1 of the left eyeball cannot be aligned with the center point S1, and the center point W2 of the right eyeball also cannot be aligned with the center point S2.

[0139] One possible scenario is that during the manufacturing process of VR glasses, assembly errors result in at least one display device (i.e., the screen) not being centered properly in alignment with the eyes. Alternatively, during use, loose parts may cause at least one display device to misalign with the eyes; for example, during assembly, the screen and corresponding optics may not be aligned, and while the optics are generally aligned with the eyes when the VR glasses are worn, the center of the screen may not be.

[0140] In some embodiments, the distance between the two display devices of a VR headset is adjustable, referred to as interpupillary distance (IPD) adjustment, to accommodate changes in interpupillary distance among different users. For example, buttons or handles on the VR headset can be used to adjust the distance between the two display modules as the positions of the display screen and optics change with the position of the display modules. For instance, when family member A wears the VR headset, A may adjust the distance between the two display devices using the buttons or handles. After A finishes wearing the headset, when family member B wears it, the distance between the two display devices may not match B's interpupillary distance. In this case, B can readjust the distance between the two display devices (which can be understood as the distance between the two display modules). Generally, during IPD adjustment, the two display devices are adjusted by the same distance but in opposite directions. For example, if display device 110 moves 1 cm to the left, display device 120 moves 1 cm to the right; or, if display device 110 moves 2 cm to the right, display device 120 moves 2 cm to the left. It should be understood that for VR glasses with assembly deviations, since at least one display device in the VR glasses is inherently misaligned with the eyes, even after IPD adjustment (adjusting the two display devices to the same distance), at least one display device will still be misaligned with the eyes. Therefore, VR glasses with assembly deviations can still be used in the technical solution of this application after IPD adjustment.

[0141] In summary, the technical solutions provided in the embodiments of this application can be applied to any scenario that could cause misalignment between points W1 and S1 and / or between points W2 and S2.

[0142] The following will take case 1 (the center point W1 of the left eyeball can be aligned with the center point S1, but the center point W2 of the right eyeball cannot be aligned with the center point S2) as an example.

[0143] For example, please see Figure 6A When a user wears VR glasses, the center of the left eyeball W1 is aligned with the center point S1 of the display device 120, meaning that the center point S1 and the center point W1 of the left eyeball are on the same straight line K5. Line K5 passes through the center point W1 of the left eyeball and is perpendicular to the left eyeball. The center point W2 of the right eyeball cannot be aligned with the center point S2 of the display device 110. For example, the center point W2 of the right eyeball is aligned with point S2' on the display device 110 (point S2' is located N distance to the right of point S2), meaning that point W2 and point S2' are on the same straight line K6'. Line K6' passes through the center point W2 of the right eyeball and is perpendicular to the right eyeball.

[0144] Continue to pass Figure 6AThe virtual environment shown to the user by the VR glasses is Figure 4B Taking environment 400 as an example, the two display devices of the VR glasses each display an image. These two images include the images of various objects in environment 400 (such as the tree 410, the football 420, and the dog 430). In this way, the user can experience environment 400 through these two images.

[0145] For example, such as Figure 6B The VR glasses generate two images, image 6100 and image 6200. Images 6100 and 6200 include overlapping regions. The overlapping region on image 6100 is region 6110, and the overlapping region on image 6200 is region 6210. Objects in region 6110 are all contained within region 6210, and vice versa. Image 6100 is displayed on display device 120. Image 6200 is displayed on display device 110. When display device 120 displays image 6100, the center point R3 of image 6100 is aligned with the center point S1 of display device 120; that is, center point R3 and center point S1 are on the same straight line K5. When display device 110 displays image 6200, the center point R4 of image 6200 is aligned with the center point S2 of display device 110; that is, center point R4 and center point S2 are on the same straight line K6. Line K6 is a line passing through the center point S2 of display device 110 and perpendicular to display device 110. Line K6 and line K6' are different lines, and the distance between them is N. The distance from the center point P5 of the overlapping region 6110 on image 6100 to line K5 is L5. The distance from the center point P6 of the overlapping region 6210 on image 6200 to line K6 is L6. Distance L5 is equal to distance L6, and the direction from center point P5 to line K5 is opposite to the direction from center point P6 to line K6.

[0146] When the user wears VR glasses, the center W1 of the left eyeball aligns with the center point S1 of the display device 120. The left eye captures image 6300. Image 6300 includes the center point T3. Point T3 is the image point corresponding to the center point R3 on image 6100. That is, points R3, S1, W1, and T3 are on the same straight line K5. The center W2 of the right eyeball aligns with point S2' on the display device 110. S2' aligns with point R4' on image 6200. Point R4' is located at a distance N to the right of the center point R4. The right eye captures image 6400. Image 6400 includes the center point T4. Point T4 is the image point corresponding to point R4' on image 6200. That is, points R4', S2', W2, and T4 are on the same straight line K6'.

[0147] It should be noted that because the center of the right eyeball W2 cannot be aligned with the center point S2 of the display device 110, it cannot be aligned with the center point R4 of the image 6200, but instead aligns with point R4', which is a distance N to the right of R4. This can be understood as the image 6200 shifting to the left by a distance N along with the display device 110, causing the right eye to align with point R4' instead of the center point R4 of the image 6200. As a result, a portion of the image 6200 will move out of the right eye's field of vision. For example, the left-hand region 6201 (the shadow area) of the image 6200 moves out of the right eye's field of vision. Therefore, this portion is not included in the image 6400 captured by the right eye. Since the field of view of the right eye remains constant (e.g., 110 degrees), even if region 6201 moves out of the right eye's field of vision, the size of the image captured by the right eye remains unchanged. For example, the image 6400 includes the right-hand region 6430 (the shadow area), which is not part of the image 6200, but is a portion of the image captured by the right eye. For example, region 6430 contains no image of any object, such as a black area.

[0148] Understandably, if area 6201 (the shaded area) is moved out of the right eye's field of vision, the user will not be able to obtain a comfortable field of vision, and the field of vision observed by the user will be smaller.

[0149] Please compare. Figure 6B and Figure 5 understand. Figure 5 When a user wears VR glasses, with the center W1 of the left eyeball aligned with the center S1 of the display device 120 and the center W2 of the right eyeball aligned with the center S2 of the display device 120, the user sees image 5300 with the left eye and image 5400 with the right eye. The brain can fuse the overlapping regions 5310 and 5410, so the user can see the environment 400 clearly and the human eye is comfortable. Figure 6B In this scenario, when a user wears VR glasses, if the center point W1 of the left eyeball aligns with the center point S1 of the display device 120, but the center point W2 of the right eyeball cannot align with the center point S2 of the display device 110, the left eye sees image 6300 and the right eye sees image 6400. The brain cannot fuse the overlapping regions 6310 and 6410. This is because the overlapping region 6410 in image 6400 is larger than... Figure 5 In the image 5400, the overlapping region 5410 is missing some content, causing the overlapping regions 6310 and 6410 to not fully merge. Because the overlapping regions cannot fully merge, the user cannot clearly see the environment 400. At this time, the brain will instinctively control the right eye muscles to move, causing the right eyeball to turn to the left in an attempt to align with the center point R4 of the image 6200. This will cause the left eye to look straight ahead and the right eye to look to the left, resulting in inconsistent lines of sight between the two eyes, which will cause dizziness and a poor experience.

[0150] Figure 6A and Figure 6B Taking case 1 (where W1 and S1 can be aligned, but W2 and S2 cannot) as an example, it can be understood that the same principle applies to case 2 (where the center point W1 of the left eyeball cannot be aligned with the center point S1, but the center point W2 of the right eyeball can be aligned with the center point S2), so it will not be repeated.

[0151] The following example illustrates case 3 (the center point W1 of the left eyeball cannot be aligned with the center point S1, and the center point W2 of the right eyeball cannot be aligned with the center point S2).

[0152] For example, please see Figure 7A When a user wears VR glasses, the distance B1 between the center point S1 of display device 120 and the center point S2 of display device 110 is less than the distance B2 between the center W1 of the left eyeball and the center W2 of the right eyeball. The distance B2 between the center W1 of the left eyeball and the center W2 of the right eyeball can also be understood as the distance between the pupils, also known as interpupillary distance (IPD). That is, as... Figure 7A As shown, the center W1 of the left eyeball cannot be aligned with the center point S1, and the center W2 of the right eyeball cannot be aligned with the center point S2.

[0153] For example, the center of the left eyeball W1 is aligned with point S1' on the display device (point S1' is a distance N1 to the left of point S1), meaning the center of the left eyeball W1 and point S1' are on the same straight line K7, which passes through the center of the left eyeball W1 and is perpendicular to the center of the left eyeball. The center of the right eyeball W2 is aligned with point S2' on the display device (point S2' is a distance N2 to the right of point S2), meaning the center of the right eyeball W2 and point S2' are on the same straight line K8, which passes through the center of the right eyeball W2 and is perpendicular to the center of the right eyeball. The distance N1 from point S1' to point S1 plus the distance N2 from point S2' to point S2 equals the distance difference between B1 and B2. Distance N1 may or may not be equal to distance N2. In some embodiments, due to assembly deviations or other reasons (e.g., display device assembly deviations), distance N1 may not be equal to distance N2.

[0154] The following section will use the example of distance N2 being greater than distance N1.

[0155] To pass Figure 7A The virtual environment shown to the user by the VR glasses is Figure 4BTaking environment 400 as an example, the two display devices of the VR glasses each display an image. These two images include images of various objects in environment 400 (such as tree 410, football 420, dog 430). In this way, the user can experience environment 400 through these two images to simulate a real environment.

[0156] For example, please see Figure 7B The VR glasses generate two images, image 7100 and image 7200. Images 7100 and 7200 include overlapping regions. The overlapping region on image 7100 is region 7110, and the overlapping region on image 7200 is region 7210. The distance L7 from the center point P7 of the overlapping region 7110 to the center point R5 of image 7100 is equal to the distance L8 from the center point P8 of the overlapping region 7210 to the center point R6 of image 7200, and the direction from P7 to R5 is opposite to the direction from P8 to R6.

[0157] Image 7100 is displayed on display device 120. Image 7200 is displayed on display device 110. When display device 120 displays image 7100, the center point S1 of display device 120 is aligned with the center point R5 of image 7100; that is, center point R5 and center point S1 are on the same straight line K7'. Line K7' and line K7 are different lines, and the distance between them is N1. When display device 110 displays image 7200, the center point S2 of display device 110 is aligned with the center point R6 of image 7200; that is, center point R6 and center point S2 are on the same straight line K8'. Line K8' and line K8 are different lines, and the distance between them is N2. N2 is greater than N1.

[0158] When a user wears VR glasses, the center W1 of the left eyeball aligns with point S1' on the display device. Point S1' aligns with point R5' on image 7100. Point R5' is located at a distance N1 to the left of the center point R5. The left eye captures image 7300. Image 7300 includes the center point T5. Point T5 is the image point corresponding to point R5' on image 7100. That is, points R5', S1', W1, and T5 are on the same straight line K7.

[0159] When the user wears VR glasses, the center W2 of the right eyeball aligns with point S2' on display device 110. Point S2' aligns with point R6' on image 7200. Point R6' is located at a distance N2 to the right of the center point R6 in image 7200. The right eye captures image 7400. Image 7400 includes center point T6. Point T6 is the image point corresponding to R6' on image 7200. That is, points R6', S2', W2, and T6 are on the same straight line K8.

[0160] Please compare image 7100 and image 7300. Because image 7100, along with display device 120, is offset to the right by a distance N1 relative to the corresponding optical device 140, the left eye cannot align with the center point R5 of image 7100, but instead aligns with point R5', located N1 to the left of the center point R5. Thus, the right-hand region 7101 (the shadow area) of image 7100 will move out of the left eye's field of vision. For example, if the left-hand region 7101 includes an image 7102 of a small tree (which can be understood as object 7102), and this image 7102 moves out of the left eye's field of vision, then the image 7300 captured by the left eye will not include the image 7102 of the small tree. Since the field of view of the left eye remains constant (e.g., 110 degrees), even though region 7101 moves out of the left eye's field of vision, the size of the image captured by the left eye remains unchanged. For example, image 7300 includes a left-side region 7330 (a shadow area). This region 7330 is not an image of image 7100, but is part of the image captured by the left eye. For instance, region 7330 contains no image of any object; it is, for example, a black area. Understandably, if region 7101 (the shadow area) moves out of the left eye's field of vision, the user will not have a comfortable field of vision, and the user's field of view will be smaller. That is, after fusing images 7300 and 7400, the image 7102 of the small tree cannot be seen, affecting the VR experience.

[0161] Please compare images 7200 and 7400. Because image 7200, along with display device 110, is offset to the left by a distance N2 relative to the corresponding optical device 130, the right eye cannot align with the center point R6 of image 7200, but instead aligns with point R6', located N2 to the right of the center point R6. Therefore, the left region 7201 (the shadow area) of image 7200 will move out of the right eye's field of vision. For example, the right region 7201 includes a portion of the tree image 7202 (e.g., the left side), which will also move out of the right eye's field of vision. Therefore, the image 7400 acquired by the right eye only includes the portion of the tree image 7202 that is not within region 7201. Since the right eye's field of view remains constant (e.g., 110 degrees), even if region 7201 moves out of the right eye's field of vision, the size of the image acquired by the right eye remains unchanged. For example, image 7400 includes a right-side region 7430 (a shadow area). This region 7430 is not an image of image 7200, but is part of the image captured by the right eye. For instance, region 7430 contains no image of any object; it is, for example, a black area. Understandably, if region 7201 (the shadow area) moves out of the right eye's field of vision, it will prevent the user from obtaining a comfortable field of vision, and the user's observed field of vision will be smaller, affecting the VR experience.

[0162] Taking N2 being greater than N1 as an example, the width of region 7201 in image 7200 that is outside the right eye's field of vision is greater than the width of region 7101 in image 7100 that is outside the left eye's field of vision. Correspondingly, the width of region 7430 in image 7400 acquired by the right eye is greater than the width of region 7330 in image 7300 acquired by the left eye.

[0163] Figure 7B In this process, the brain cannot fuse the overlapping regions on images 7300 and 7400. This is because the overlapping regions 7310 and 7410 on images 7300 do not contain exactly the same objects. For example, overlapping region 7410 only includes half of the image of a tree. Therefore, overlapping regions 7310 and 7410 cannot be completely fused. Because overlapping regions 7310 and 7410 cannot be completely fused, the user cannot clearly see the environment 400. At this time, the brain will instinctively control the left eye muscles to move, causing the left eyeball to turn to the right in an attempt to align with the center R5 of image 7100, and will also control the right eye muscles to move, causing the right eyeball to turn to the left in an attempt to align with the center R6 of image 7200. This will cause the left eye to look to the right and the right eye to look to the left, resulting in different lines of sight between the two eyes and causing dizziness. Furthermore, the user's field of vision becomes smaller, and after fusing images 7300 and 7400, the image 7102 of the small tree cannot be seen.

[0164] The above lists three situations that cause binocular fusion problems, namely situations 1 to 3. To solve problems such as binocular fusion problems or poor field of vision, Figure 8A and Figure 8B One possible implementation is shown.

[0165] like Figure 8A The VR glasses display device 110 includes two regions, region 1 and region 2. Region 1 can be a central region, and region 2 can be an edge region (shadow region), with region 2 surrounding region 1. That is, the center point of region 1 can overlap with the center point of display device 110, which is S2. The area of ​​region 1 can be preset, and the distance N4 between the inner edge of region 2 and the outer edge of region 1 can be preset. The display device 120 includes two regions, region 3 and region 4. Region 3 can be a central region, and region 4 can be an edge region (shadow region), with region 4 surrounding region 3. That is, the center point of region 3 can overlap with the center point of display device 120, which is S1. The area of ​​region 3 can be preset, and the distance N5 between the inner edge of region 4 and the outer edge of region 3 can be preset.

[0166] Assume that when a user wears VR glasses, the center W1 of the left eyeball is aligned with the center point S1 of the display device 120. Image 8100 is displayed in area 3 of the display device 120. Area 4 displays nothing (e.g., area 4 is black). That is, the center W1 of the left eyeball can be aligned with the center point S1 of image 8100. The left eye captures image 8300. When the user wears VR glasses, the center W2 of the right eyeball is aligned with the center point S2 of the display device 110. Image 8200 is displayed in area 1 of the display device 110. Area 2 displays nothing (e.g., area 2 is black). That is, point W2 can be aligned with the center point S2 of image 8200. The right eye captures image 8400. Since points W1 and S1 are aligned, and points W2 and S2 are aligned, images 8300 and 8400 can be blended, preventing dizziness.

[0167] Figure 8A In some embodiments, the center point W1 of the left eyeball is aligned with the center point S1 of the display device 120, and the center point W2 of the right eyeball is aligned with the center point S2 of the display device 110. Figure 8A The VR glasses shown may experience one of the three situations described above, resulting in misalignment between the center W1 of the left eyeball and the center point S1 of the display device 120, and / or misalignment between the center W2 of the right eyeball and the center point S2 of the display device 110. When one of these three situations occurs, the VR glasses can be used... Figure 8B The solution is as follows. For ease of description, we will take case 1 (the center W1 of the left eyeball is aligned with the center point S1 of the display device 120, but the center W2 of the right eyeball cannot be aligned with the center point S2 of the display device 110) as an example.

[0168] like Figure 8B The center of the user's right eye, W2, cannot be aligned with the center point S2 of the display device 110. For example, the center of the right eye, W2, is aligned with point S2' on the display device 110. Point S2' is located at a distance N6 to the left of point S2. In this case, image 8200 is not displayed in region 1 on the display device 110, but in region 5. The center point of region 5 is point S2', so when image 8200 is displayed in region 5, the center point of image 8200 is point S2', and therefore point W2 can be aligned with the center point of image 8200. Thus, image 8400 acquired by the right eye can be fused with image 8300 acquired by the left eye.

[0169] Therefore, by reserving a display area on the display device, the problem of misalignment between points W1 and S1 and / or between points W2 and S2 can be solved.

[0170] Figure 9This demonstrates another feasible solution to binocular non-fusion. This method does not require reserving a display area on the display device, and can solve the binocular non-fusion problem through image processing. It is also more conducive to reducing the size of the display device, achieving miniaturization and portability of the device.

[0171] Continue with case 1 (i.e.) Figure 6A For example, see the following case: Figure 9 The center point W1 of the left eyeball can be aligned with the center point S1 of the display device 120, while the center point W2 of the right eyeball cannot be aligned with the center point S2 of the display device 110, but is aligned with point S2' on the display device (point S2' is located at a distance N to the right of point S2). The display devices 110 and 120 are asymmetrical with respect to the intermediate plane (or the center line of the face).

[0172] by Figure 9 Taking the VR glasses displaying environment 400 to the user as an example, the VR glasses generate two images, namely image 9100 and image 9200. Image 9100 is displayed in full screen on display device 120, and image 9200 is displayed in full screen on display device 110. That is, no reserved display area is needed on the display devices, allowing for a relatively small display device size, saving costs. Furthermore, a smaller display device size is beneficial to the trend of lightweight and compact device design. When display device 120 displays image 9100, the center point S1 of display device 120 is aligned with the center point R9 of image 9100, meaning S1 and R9 are on the same straight line K9. When display device 110 displays image 9200, the center point S2 of display device 110 is aligned with the center point R10 of image 9200, meaning S2 and R10 are on the same straight line K10.

[0173] Images 9100 and 9200 have overlapping regions. The overlapping region in image 9100 is region 9110, and the overlapping region in image 9200 is region 9210. The distance from the center point P9 of overlapping region 9110 to the center point R9 of image 9100 is L9, and the first direction from P9 to R9 is left. The distance from the center point P10 of overlapping region 9210 to R10 of image 9200 is L10, and the first direction from P10 to R10 is right. Distance L9 is not equal to distance L10. The difference between distances L9 and L10 is N. It should be noted that... Figure 9In this embodiment, the center point R9 of image 9100 and the center point R10 of image 9200 are used as references. In other embodiments, other points on image 9110 (e.g., the left vertex) and other points on image 9200 (e.g., the right vertex) can also be used as references. That is, the distance from the center point P9 of the overlapping region 9110 to the left vertex of image 9100 is not equal to the distance from the center point P10 of the overlapping region 9210 to the right vertex of image 9200. The center points P9 and P10 are symmetrical with respect to the intermediate plane (or the center line of the face).

[0174] like Figure 9 When a user wears VR glasses, the center point W1 of the left eyeball can be aligned with the center S1 of the display device 120. The left eye captures image 9300. Image 9300 includes a center point T9. Point T9 is the image point corresponding to the center point R9 on image 9100. That is, points R9, S1, W1, and T9 are on the same straight line K9.

[0175] like Figure 9 The center of the right eyeball, W2, is aligned with point S2' on the display device (point S2' is located N distance to the right of point S2). Point S2' is aligned with point R10' on image 9200, which is located N distance to the right of the center point R10. Image 9400 is acquired by the right eye. Image 9400 includes the center point T10. Point T10 is the image point corresponding to point R10' on image 9200. That is, points T10, W2, S2', and R10' are on the same straight line K10'. Line K10' is different from line K10, and the distance between them is N.

[0176] In image 9300, the overlapping region 9310 includes a center point P9'. Point P9' is the image point corresponding to point P9 in image 9100. The distance from point P9' to line K9 is L9'. In image 9400, the overlapping region 9410 includes a center point P10', which is the image point corresponding to point P10 in image 9200. The distance from point P10' to line K10' is L10'. The distance L9' is equal to L10'. Furthermore, the direction from point P9' to line K9 is opposite to the direction from point P10' to line K10', and the center points P9' and P10' are symmetrical with respect to the intermediate plane (or the center line of the face).

[0177] Although Figure 9 The display device 110 is offset to the left by a distance N relative to the corresponding optical device 130. However, since the overlapping area 9210 is shifted to the right by a distance N, the offset of the display device 110 is compensated (or can be called canceled). Therefore, the overlapping area 9410 in the image acquired by the right eye and the overlapping area 9310 in the image acquired by the left eye can be merged.

[0178] When the overlapping region 9210 shifts to the right by a distance N, a region 9230 with a width of N will be left on the leftmost side of the display device 110. In some embodiments, region 9230 displays a portion of the image to the left of the overlapping region 9110. For example, in image 9100, if a background object (e.g., a blue sky and white clouds) is displayed in a region with a width of N near the left side of the overlapping region 9110, then region 9230 will also display a background object (e.g., a blue sky and white clouds). Therefore, the objects in region 9230 and the portion of the overlapping region to the left are the same, and these two regions can be understood as overlapping regions. In some embodiments, region 9230 includes a new object that is not present in image 9100. For example, if the display device 110 is shifted upwards by a distance N relative to the corresponding optical device 130, then the overlapping region 9210 shifts downwards by a distance N, and region 9230 will appear on the upper part of the display device 110. Since image 9200 is an image patch on a panoramic image, the object within region 9230 can be an object located above the overlapping region 9210 in the panoramic image patch (an object not present in image 9100). In some embodiments, region 9230 can display a first color, the type of which is not limited, such as black, white, etc. It is understood that as the display device 110 shifts to the left by a distance N relative to the corresponding optical device (130), region 9230 may move out of the right eye's field of vision. Therefore, region 9230 may not display any content; for example, region 9230 may not be powered on, i.e., region 9230 is black, which saves power consumption. It should be noted that when region 9230 moves out of the right eye's field of vision, since the right eye's field of view remains unchanged, the image 9400 acquired by the right eye includes the right region 9430 (the shadow area). Region 9430 is not an image of image 9200; for example, it is a black area, representing that no image of any object is displayed in this part.

[0179] It should be noted that, Figure 9 Let's take case 1 as an example. Below, we'll discuss case 3 (i.e....) Figure 7A Let's take ) as an example for introduction.

[0180] like Figure 10Taking the display device 120 offset to the right by a distance N1 relative to the corresponding optical device 140, and the display device 110 offset to the left by a distance N2 relative to the corresponding optical device 130 as examples, where N1 is less than N2, when a user wears VR glasses, the center point W1 of the left eyeball cannot be aligned with the center point S1 of the display device 120, but is aligned with point S1' on the display device 120, with point S1' located to the left of point S1 at a distance N1. The center point W2 of the right eyeball cannot be aligned with the center point S2 of the display device 110, but is aligned with point S2' on the display device, with point S2' located to the right of point S2 at a distance N2. Thus, the display devices 110 and 120 are asymmetrical with respect to the central plane (or the center line of the face).

[0181] by Figure 10 Taking the VR glasses displaying environment 400 to the user as an example, the VR glasses generate two images, namely image 1000 and image 1100. Image 1000 is displayed in full screen on display device 120, and image 1100 is displayed in full screen on display device 110. That is, no display area needs to be reserved on the display device, allowing for a relatively small display device size, saving costs, and facilitating the trend towards lightweight and compact device design. When display device 120 displays image 1000, the center point S1 of display device 120 is aligned with the center point R11 of image 1000, meaning S1 and R11 are on the same straight line K11. When display device 110 displays image 1100, the center point S2 of display device 110 is aligned with the center point R12 of image 1100, meaning S2 and R12 are on the same straight line K12.

[0182] Images 1000 and 1100 have overlapping areas. The overlapping area on image 1000 is region 1010. In some embodiments, region 1030 can also be an overlapping area, overlapping with a portion of the image to the right of region 1110. In some embodiments, to save power, region 1030 may not display any content. Figure 10 In this example, region 1030 may also be left blank. The overlapping region on image 1100 is region 1110. In some embodiments, region 1130 may also be an overlapping region, overlapping with a portion of the image to the left of region 1010. In some embodiments, to save power, region 1130 may also be left blank. Figure 10 In the example below, we will use area 1130 as an example where no content is displayed.

[0183] The following explanation uses overlapping regions 1010 and 1110 as examples. The distance from the center point P11 of overlapping region 1010 to the center point R11 of image 1000 is L11, and the first direction from P11 to R11 is left. The distance from the center point P12 of overlapping region 1110 to R12 of image 1100 is L12, and the second direction from P12 to R12 is right. Since N1 is not equal to N2, distance L11 is not equal to distance L12. Taking N1 being less than N2 as an example, distance L11 is greater than distance L12. The difference between distance L11 and distance L12 is equal to the difference between distance N1 and N2. In some other embodiments, the direction from P11 to R11 is different from the direction from P12 to R12, for example, the directions are opposite. Among them, center point P11 and center point P12 are symmetrical with respect to the middle plane (or the center line of the face).

[0184] like Figure 10 When the user wears VR glasses, the center point W1 of the left eyeball aligns with point S1' of the display device 120. Point S1' aligns with point R11' on image 1000, and point R11' is located at a distance N1 to the left of the center point R11. The left eye captures image 1200. Image 1200 includes a center point T11. Point T11 is the image point corresponding to point R11' on image 1000. That is, points T11, W1, S1', and R11' are on the same straight line K11'. The straight line K11' is different from the straight line K11, and the distance between them is N1.

[0185] like Figure 10 The center W2 of the right eyeball is aligned with point S2' on display device 110. Point S2' is aligned with point R12' on image 1100, and point R12' is located to the right of the center point R12 at a distance of N2. The right eye acquires image 1300. Image 1300 includes center point T12. Point T12 is the image point corresponding to point R12' on image 1100. That is, points T12, W2, S2', and R12' are on the same straight line K12'. Among them, straight line K12' is different from straight line K12, and the distance between them is N2.

[0186] In the image 1200 acquired by the left eye, the overlapping region 1210 includes a center point P11'. Point P11' is the image point corresponding to point P11 on image 1000. The distance from point P11' to line K11' is L11'. In the image 1300 acquired by the right eye, the overlapping region 1310 includes a center point P12'. Point P12' is the image point corresponding to point P12 on image 1100. The distance from point P12' to line K12' is L12'. The distance L11' is equal to L12'. Moreover, the direction from point P11' to line K11' is opposite to the direction from point P12' to line K12', and the center points P11' and P12' are symmetrical with respect to the intermediate plane (or the center line of the face).

[0187] In other words, although Figure 10 The display device 120 is offset to the right by a distance N1 relative to the corresponding optical device 140, but the overlapping region 1010 is shifted to the left by a distance N1, compensating for (or offsetting) the offset of the display device 110. Similarly, the display device 110 is offset to the left by a distance N2 relative to the corresponding optical device 130, but the overlapping region 1110 is shifted to the right by a distance N2, compensating for (or offsetting) the offset of the display device 110. Therefore, the overlapping regions 1210 and 1310 can be merged.

[0188] See also Figure 10 The non-overlapping areas on image 1000 include two regions, namely region 1030 and region 1040. The overlapping region 1010 is located between region 1030 and region 1040. Region 1030 may display a second color; the type of the second color is not limited, for example, black, white, etc., or it could also be the background color of image 1000. The non-overlapping areas on image 1100 include two regions, namely region 1130 and region 1140. The overlapping region 1110 is located between region 1130 and region 1140. Region 1130 may display a first color; the type of the first color is not limited, for example, black, white, etc., or it could also be the background color of image 1100. The first color and the second color can be the same or different.

[0189] It should be noted that since N1 is not equal to N2, the area (or width) of the non-overlapping region 1030 on image 1000 is different from the area (or width) of the non-overlapping region 1130 on image 1100. Taking N1 less than N2 as an example, the width of region 1030 is less than the width of region 1130. Similarly, the area (or width) of the non-overlapping region 1040 on image 1000 is different from the area (or width) of the non-overlapping region 1140 on image 1100. Taking N1 less than N2 as an example, the width of region 1140 is less than the width of region 1040.

[0190] In some embodiments, images 1000 and 1100 are image blocks within different regions of the same panoramic image. For example, image 1000 is an image block located within a first display region of the panoramic image; image 1100 is an image block located within a second display region of the panoramic image. The overlapping region is the area where the first display region and the second display region overlap. Region 1030 may be a region located to the right of region 1010 in the panoramic image. Region 1130 may be a region located to the left of region 1110 in the panoramic image.

[0191] Understandably, as the display device 110 shifts left by a distance N2, region 1130 will move out of the right eye's field of vision. Therefore, region 1130 may not display any content, such as any color. It should be noted that when region 1130 moves out of the right eye's field of vision, since the right eye's field of view remains unchanged, the image 1300 captured by the right eye includes the right-side region 1330 (the shadow area). Region 1330 is not the image of image 1100; for example, it is a black area, representing that no object image is displayed in this part. Similarly, as the display device 120 shifts right by a distance N1 relative to the corresponding optical device 140, region 1030 will move out of the left eye's field of vision. Therefore, region 1030 may also not display any content, such as any color. It should be noted that when region 1030 moves out of the left eye's field of vision, since the size of the left eye's field of vision remains unchanged, the image 1200 captured by the left eye includes the left region 1230 (the shadow area). Region 1230 is not the image of image 1000. For example, it is a black area, which means that there is no image of any object displayed in this part.

[0192] It is understood that the positions of display device 110 and / or display device 120 can change dynamically. Figure 10 For example, in some embodiments, the position of the center point P11 of the overlapping region 1010 can move as the display device 120 moves, wherein the direction of movement of the center point P11 is opposite to the direction of movement of the display device 120, in order to compensate for or counteract the positional movement of the display device 120. Similarly, the position of the center point P12 of the overlapping region 1110 can move as the display device 110 moves, wherein the direction of movement of the center point P12 is opposite to the direction of movement of the display device 110, in order to compensate for or counteract the positional movement of the display device 110.

[0193] As mentioned earlier, VR glasses with assembly deviations still exhibit assembly deviations during IPD adjustment. Figure 10For example, suppose the VR glasses undergo IPD adjustment (i.e., display devices 110 and 120 move the same distance but in opposite directions, for example, display devices 110 and 120 move to the left and right respectively, or display devices 110 and 120 move to the right and left respectively). Before and after IPD adjustment, the relative positional relationship of the assembly deviation of display devices 110 and 120 remains unchanged (i.e., the difference in offset remains unchanged). Therefore, the positional movement relationship of the two images displayed by display devices 110 and 120 remains unchanged, ensuring that binocular fusion can be achieved before and after IPD adjustment. For example, if the VR glasses undergo IPD adjustment before displaying the first and second images, compared to not undergoing IPD adjustment, the distance difference between distance L11 and distance L12 remains unchanged, and the relative relationship between the first and second directions remains unchanged compared to before IPD adjustment. In some embodiments, when display device 110 and display device 120 move to the left and to the right respectively, or when display device 110 and display device 120 move to the right and to the left respectively, the first direction and the second direction remain unchanged compared to before IPD adjustment.

[0194] In some embodiments, the VR glasses perform IPD adjustment before displaying the first image and the second image. Before IPD adjustment, when display devices 110 and 120 display images respectively, the distance from the center point of the overlapping area on the image displayed by display device 120 to the center point of the image displayed by display device 120 is a third distance, and the distance from the center point of the overlapping area on the image displayed by display device 110 to the center point of the image displayed by display device 110 is a fourth distance. The difference between the third distance and the fourth distance is a first distance difference. After IPD adjustment, display devices 110 and 120 display the first image and the second image respectively. The distance from the center point of the overlapping area on the first image to the center point of the first image is distance L11, and the distance from the center point of the overlapping area on the second image to the center point of the second image is distance L12. The difference between distance L11 and distance L12 is a second distance difference, wherein the first distance difference is equal to the second distance difference.

[0195] In some embodiments, the VR glasses perform IPD adjustment after displaying the first image and the second image. Before IPD adjustment, display device 110 and display device 120 display the first image and the second image respectively. The distance from the center point of the overlapping area on the first image to the center point of the first image is distance L11, and the distance from the center point of the overlapping area on the second image to the center point of the second image is distance L12. The difference between distance L11 and distance L12 is the second distance difference. After IPD adjustment, when display device 110 and display device 120 display images respectively, the distance from the center point of the overlapping area on the image displayed by display device 120 to the center point of the image displayed by display device 120 is the fifth distance, and the distance from the center point of the overlapping area on the image displayed by display device 110 to the center point of the image displayed by display device 110 is the sixth distance. The difference between the fifth distance and the sixth distance is the third distance difference. The third distance difference is equal to the second distance difference.

[0196] For example, Figure 11 This is a schematic flowchart of a display method provided in one embodiment of this specification. This method can be applied to any of the above-described display methods, for example, it can be applied to... Figure 9 or Figure 10 The display method is shown. For example... Figure 11 As shown, the process includes:

[0197] S1101, VR glasses acquire 3D image data.

[0198] The 3D image data includes both 2D image information and depth information. The depth information includes the depth corresponding to each pixel in the 2D image information. The 3D image data can be generated by VR applications, such as VR games, VR educational applications, VR movie viewing applications, VR driving applications, and so on.

[0199] S1102, the VR glasses acquire a first coordinate transformation matrix and a second coordinate transformation matrix. The first coordinate transformation matrix is ​​used to convert 3D image data into a first planar image, and the second coordinate transformation matrix is ​​used to convert 3D image data into a second planar image. The first planar image corresponds to a first display device, and the second planar image corresponds to a second display device.

[0200] The first display device can be display device 120 mentioned above, corresponding to the left eye, and the second display device can be display device 110 mentioned above, corresponding to the right eye.

[0201] The first coordinate transformation matrix is ​​used to transform the 3D image data from the first coordinate system to the second coordinate system. The first coordinate system is the coordinate system where the 3D image data is located, and the second coordinate system is the coordinate system corresponding to the first display device or the left eye. The coordinate system corresponding to the left eye can be the coordinate system of the first virtual camera. The first virtual camera can be understood as a virtual camera created to simulate the left eye. Because the image acquisition principle of the human eye is similar to the image shooting principle of a camera, a virtual camera can be created to simulate the image acquisition process of the human eye. The first virtual camera simulates the left eye. For example, the position of the first virtual camera is the same as the position of the left eye, and / or, the field of view of the first virtual camera is the same as the field of view of the left eye. For example, generally, the human eye's field of view is 110 degrees vertically and 110 degrees horizontally, so the field of view of the first virtual camera is also 110 degrees vertically and 110 degrees horizontally. For another example, VR glasses can determine the position of the left eye, and the first virtual camera is set at the position of the left eye. There are several ways to determine the position of the left eye. For example, method 1: first determine the position of the first display device, and then add the distance A to the position of the first display device to estimate the position of the left eye. This method determines the left eye position relatively accurately. Here, spacing A is the distance between the display device and the viewer's eye, which can be pre-stored. Method 2: The left eye position is equal to the position of the first display device. This method ignores the distance between the viewer's eye and the display device and is easier to implement.

[0202] The second coordinate transformation matrix is ​​used to transform the 3D image data from the first coordinate system to the third coordinate system. The first coordinate system is the coordinate system where the 3D image data is located, and the third coordinate system is the coordinate system corresponding to the second display device or the right eye. The coordinate system corresponding to the right eye can be the coordinate system of the second virtual camera. This second virtual camera can be understood as a camera created to simulate the user's right eye. For example, the position of the second virtual camera is the same as the position of the right eye, and / or, the field of view of the second virtual camera is the same as the field of view of the right eye.

[0203] For example, the first and second coordinate transformation matrices can be pre-stored in the VR glasses. For instance, they can be stored in a register, and the VR glasses read the first and second coordinate transformation matrices from the register.

[0204] S1103, the VR glasses process the three-dimensional image data according to the first coordinate transformation matrix to obtain the first planar image, and process the three-dimensional image data according to the second coordinate transformation matrix to obtain the second planar image.

[0205] As mentioned earlier, both the first and second planar images are derived from 3D image data through coordinate transformation. This coordinate transformation process can be understood as using a virtual camera to capture 3D images, completing the conversion from 3D to 2D. For example, capturing 3D image data using a first virtual camera yields the first planar image, and capturing 3D image data using a second virtual camera yields the second planar image. For an example using the first virtual camera, please refer to [link to example]. Figure 12A This is a schematic diagram of the first virtual camera. The first virtual camera includes four parameters: field of view (FOV) angle, aspect ratio of the actual shooting window, near clipping plane, and far clipping plane. The aspect ratio of the actual shooting window can be the aspect ratio of the far clipping plane. The far clipping plane can be understood as the farthest range that the first virtual camera can capture, and the near clipping plane can be understood as the closest range that the first virtual camera can capture. It can be understood that objects in the 3D image data that are within the FOV and between the near and far clipping planes can be captured by the first virtual camera. For example, the 3D image data includes multiple objects, such as sphere 1400, sphere 1401, and sphere 1402. Sphere 1400 is outside the FOV and cannot be captured, while spheres 1401 and 1402 are within the FOV and between the near and far clipping planes, and can be captured. Therefore, the image captured by the first virtual camera includes spheres 1401 and 1402. The image captured by the first camera can be understood as a projection of 3D image data onto the near-cropping plane. The coordinates of each pixel on this projection image can be determined. For example, please refer to... Figure 12B A three-dimensional coordinate system O-XYZ is established with the center of the first virtual camera. The XY plane includes four quadrants, or four regions: the upper left region, the upper right region, the lower left region, and the lower right region. Taking the edge points G1, G2, G3, and G4 of the object's projection onto the near-cropping plane in the three-dimensional image data as an example, the coordinates of edge points G1 to G4 are (l, r, t, b). Here, l is left, t is top, r is right, and b is bottom. For example, the coordinates of edge point G1 are (3, 0, 3, 0), the coordinates of edge point G2 are (0, 3, 3, 0), the coordinates of edge point G3 are (3, 0, 0, 3), and the coordinates of edge point G4 are (0, 3, 0, 3). The depth of each of the four edge points is n.

[0206] Therefore, taking the first planar image as an example, the first planar image satisfies the formula: A*H=K; where A is the matrix corresponding to the three-dimensional image data, H is the first coordinate transformation matrix, and K is the matrix corresponding to the first planar image. For example, A is (l, r, t, b), and the first coordinate transformation matrix H satisfies the following:

[0207]

[0208] Where l represents left, r represents right, t represents top or bottom, and n represents the depth of the near clipping plane in the z-axis direction. f represents the depth of the far clipping plane in the z-axis direction. For pixels in the 3D image data, the first coordinate transformation matrix H described above can be used to offset them in the four directions of up, down, left, and right to obtain their positions in the first planar image. Since A is a row of four columns and H is a row of four columns, the resulting K is a row of four columns, meaning that the position of the pixel in the first planar image is described by parameters in the four directions of up, down, left, and right.

[0209] The principle for obtaining the second plane image is the same as that for obtaining the first plane image, and will not be repeated here.

[0210] S1104, the VR glasses acquire a first offset of the first display device and / or a second offset of the second display device.

[0211] For example, the first offset and the second offset can be the same or different. For instance, with... Figure 10 For example, the first offset of display device 110 is a leftward offset distance N2 relative to the corresponding optical device 130, and the second offset of display device 120 is a rightward offset distance N1 relative to the corresponding optical device 140. In some embodiments, if the distance and direction of the first and second offsets are equal, for example, if the first offset of display device 110 is a leftward offset distance N3 relative to the corresponding optical device 130, and the second offset of display device 120 is a leftward offset distance N3 relative to the corresponding optical device 140, then the first image and the second image do not need to undergo coordinate transformation, and the user's eyes can be fused. In this case, coordinate transformation of the first image and the second image is not required. Optionally, the coordinates of the first image and the second image can be shifted to the right by a distance N3, so that the center of the image appears directly in front of the user's eyes, avoiding strabismus.

[0212] In some implementations, the VR glasses pre-store a first offset and a second offset. For example, they are stored in a register. The VR glasses read the first offset and the second offset from the register. For instance, the first offset and the second offset may be calibrated and stored in the VR glasses before they leave the factory.

[0213] One feasible calibration method involves using a binocular fusion detection device to calibrate the positional offset of the display device on the VR glasses after assembly. For example, such as... Figure 12CThe binocular fusion detection device includes a camera 1401 and an optical system 1402. Images displayed by the two display devices of the VR glasses are captured by the camera 1401 after a series of reflections through the optical system 1402. For example, display device 120 of the VR glasses displays image 1, and display device 110 displays image 2. Both images 1 and 2 have crosshairs (the crosshairs on image 1 are dashed lines, and the crosshairs on image 2 are solid lines). Furthermore, the distance L01 from the intersection point O1 of the crosshairs on image 1 to the center point O2 of image 1 is equal to the distance L02 from the intersection point O3 of the crosshairs on image 2 to the center point O4 of image 2. Moreover, the direction from intersection point O1 to center point O2 is opposite to the direction from intersection point O3 to center point O4. After reflection through the optical system 1402, the camera 1401 captures image 3. Image 3 includes two crosshairs. One crosshair's intersection point is O1', which is the image point corresponding to intersection point O1 on image 1, and the other crosshair's intersection point O3' is the image point corresponding to intersection point O3 on image 2. It should be noted that if there were no assembly displacement deviation, since the crosses in Images 1 and 2 are symmetrical (i.e., distance L01 equals distance L02, and the direction from intersection point O1 to center point O2 is opposite to the direction from intersection point O3 to center point O4), there should be only one cross in Image 3 (the crosses of Images 1 and 2 are merged). However, due to positional offset between the two display devices (e.g., assembly deviation), the crosses in Images 1 and 2 cannot be merged, resulting in two crosses in Image 3. For example, the interval between the two crosses in Image 3 includes: an interval of x1 in the X direction and an interval of y1 in the Y direction. The interval between the two crosses can be used to determine the first offset of the first display device and the second offset of the second display device.

[0214] In some embodiments, the first offset of the first display device is (x1 / 2, y1 / 2), and the second offset of the second display device is (-x1 / 2, -y1 / 2). Alternatively, the first offset of the first display device is (x1, y1), and the second offset of the second display device is 0. Or, the first offset of the first display device is (x1 / 3, y1 / 3), and the second offset of the second display device is (-2x1 / 3, -2y1 / 3). In short, the sum of the displacements of the first and second display devices in the X-axis direction is x1, and the sum of the displacements in the Y-axis direction is y1. It should be noted that, taking the first offset of the first display device as (x1 / 2, y1 / 2) and the second offset of the second display device as (-x1 / 2, -y1 / 2) as an example, represents the first display device's offset in the positive X-axis and positive Y-axis directions because the first offset is a positive number; the second display device's offset is in the opposite X-axis and opposite Y-axis directions because the second offset is a negative number.

[0215] In some embodiments, the offset of an object in an image is proportional to the offset of the display device. To avoid excessive offset of an object in an image, the offset of the display device should not be too large, or the difference between the offsets of the two display devices should not be too large. For example, if the first offset of the first display device is (x1, y1) and the second offset of the second display device is 0, it will cause the offset of the object in the image on the first display device to be too large, or the difference between the offsets of the objects in the images displayed by the two display devices to be too large. Therefore, the offsets of the two display devices can be used to compensate for the total translation amount. For example, the first offset of the first display device is (x1 / 2, y1 / 2) and the second offset of the second display device is (-x1 / 2, -y1 / 2) to ensure that the offset of the object in the image is not too large or the difference between the physical offsets in the images displayed by the two display devices is not too large.

[0216] S1105, the VR glasses process the first planar image based on the first offset to obtain the third planar image, and / or process the second planar image based on the second translation to obtain the fourth planar image.

[0217] Continue with Figure 12C For example, the first planar image is image 1, and the second planar image is image 2. Assume the first offset of display device 120 is (x1 / 2, y1 / 2), and the second offset of display device 110 is (-x1 / 2, -y1 / 2). Then, as... Figure 13 In image 1, the position of the crosshair (A1, B1) is shifted from (x1 / 2, y1 / 2) to (A1', B1') to compensate for the assembly position shift, where A1' = A1 + x1 / 2, B1' = B1 + y1 / 2. Similarly, in image 2, the position of the crosshair (A2, B2) is shifted from (-x1 / 2, -y1 / 2) to (A2', B2') to compensate for the assembly position shift, where A2' = A2 - x1 / 2, B2' = B2 - y1 / 2. When display device 120 displays the processed image 1 and display device 110 displays the processed image 2, the image 3 captured by the camera includes a crosshair, meaning the crosshairs from the two images 1 are merged.

[0218] The process of processing the first planar image based on the first offset to obtain the third planar image may include at least one of method 1 or method 2.

[0219] Method 1 involves translating each pixel in the first planar image using a first offset to obtain the third planar image. In other words, the first planar image is moved as a whole (including overlapping and non-overlapping areas).

[0220] Method 2 involves translating the pixels within the overlapping region of the first planar image using a first offset to obtain the third planar image. This overlapping region is the region where the first and second planar images overlap, and includes at least one identical object. Method 2 considers that pixels in non-overlapping regions do not need to be fused, so it only translates pixels within the overlapping region to ensure binocular fusion. This method reduces workload and improves efficiency.

[0221] For example, taking a first offset of (x1 / 2, y1 / 2) as an example, the first point in the overlapping region of the first planar image is located at (X1, Y1), and the corresponding position of this first point on the third planar image is (X2, Y2), where X2 = X1 + x1 / 2 and Y2 = Y1 + y1 / 2. The first point can be any point within the overlapping region, such as the center point or an edge point.

[0222] The process of processing the second planar image based on the second translation amount to obtain the fourth planar image can include at least one of method 1 or method 2. Method 1 involves translating each pixel in the second planar image using the second offset to obtain the fourth planar image. In other words, the second planar image is moved as a whole. Method 2 involves translating pixels within the overlapping region of the second planar image using the second offset to obtain the fourth planar image. This overlapping region is the region where the first and second planar images overlap, and includes at least one identical object. For example, if the second offset is (-x1 / 2, -y1 / 2), the second point within the overlapping region of the second planar image is located at (X3, Y3), and the corresponding position of this second point in the fourth planar image is (X4, Y4), where X4 = X3 - x1 / 2, Y4 = Y3 - y1 / 2. The second point can be any point within the overlapping region, such as the center point, edge point, etc.

[0223] In some embodiments, the overlapping region may include multiple objects, such as a first object and a second object. The offsets of the first object and the second object may be different.

[0224] For example, let's take the first feature point of the first object and the second feature point of the second object as examples. On the first planar image, the coordinates of the first feature point of the first object are (X5, Y5), and the coordinates of the second feature point of the second object are (X6, Y6). The first feature point of the first object can be the center point of the first object or a vertex of the first object, and the second feature point of the second object can be the center point of the second object or a vertex of the second object, etc. Taking the first offset of the first feature point as (x1 / 2, y1 / 2) and the first offset of the second feature point as (x1 / 3, y1 / 3) as an example, the first offset of the first feature point is greater than that of the second feature point. Thus, the third planar image obtained after processing the first planar image is used for display on the display device 120. In some embodiments, the coordinates of the first feature point of the first object on the third planar image are (X7, Y7), where X7 = X5 + x1 / 2 and Y7 = Y5 + y1 / 2. In some embodiments, the coordinates of the second feature point of the second object on the third planar image are (X8, Y8), where X8 = X6 + x1 / 3 and Y8 = Y6 + y1 / 3. The coordinates (X5, Y5), (X6, Y6), (X7, Y7), and (X8, Y8) are coordinates in the same coordinate system, all of which are coordinates on the display device 120.

[0225] For example, in the second planar image, the coordinates of the first feature point of the first object are (X9, Y9), and the coordinates of the second feature point of the second object are (X10, Y10). The second offset of the first feature point is (x1 / 4, y1 / 4), and the second offset of the second feature point is (x1 / 5, y1 / 5), meaning the second offset of the first feature point is greater than that of the second object. Thus, the fourth planar image obtained after processing the second planar image is used for display on the display device 110. In some embodiments, the coordinates of the first feature point of the first object in the fourth planar image are (X11, Y11), where X11 = X9 + x1 / 4, and Y11 = Y9 + y1 / 4. In some embodiments, the coordinates of the second feature point of the second object on the fourth planar image are (X12, Y12), where X12 = X10 + x1 / 5, Y12 = Y10 + y1 / 5, and the coordinates (X9, Y9), (X10, Y10), (X11, Y11), and (X12, Y12) are coordinates in the same coordinate system, all of which are coordinates on the display device 110.

[0226] Assuming the coordinates of the first feature point of the first object on the third plane image are (X7, Y7), and the coordinates of the first feature point of the first object on the fourth plane image are (X11, Y11), then the coordinate difference between (X7, Y7) and (X11, Y11) is (D1, D2), where D1 = X7 - X11 and D2 = Y7 - Y11. Assuming the coordinates of the second feature point of the second object on the third plane image are (X8, Y8), and the coordinates of the second feature point of the second object on the fourth plane image are (X12, Y12), then the coordinate difference between (X8, Y8) and (X12, Y12) is (D3, D4), where D3 = X8 - X12 and D4 = Y8 - Y12. The coordinate differences (D1, D2) and (D3, D4) are different, for example, D1 > D3 and / or D2 > D4, because the offset of the first object is greater than the offset of the second object. The offset of the second object can be 0, that is, the second object can be not offset.

[0227] In the example above, taking the case where the offset of the first object is greater than that of the second object as an example, in some embodiments, the offset of the first object is greater than that of the second object, or only the first object is offset, when at least one of the following conditions is met:

[0228] Condition 1: The first object is located within the user's gaze point area, while the second object is not located within the user's gaze point area. For example, the user's gaze point can be obtained based on information acquired by the eye-tracking module 105. The gaze point area can be a circular or square area centered on the gaze point. Generally, users pay more attention to objects within the gaze point area and less attention to objects outside the gaze point area. Therefore, objects outside the user's gaze point have small or no offset, which has little impact on the user experience and saves computational effort. When the user's gaze point changes, the offset of different objects can change accordingly to match the change in gaze point and achieve a better visual effect.

[0229] Condition 2: Both the first object and the second object are located within the user's gaze point area, and the second object is closer to the edge of the user's gaze point area than the first object. Generally, users pay more attention to objects in the middle of the gaze point area and less attention to objects at the edges. Therefore, a smaller offset for objects at the edges will not significantly affect the user experience and can save workload.

[0230] Condition 3: The distance between the first object and the center of the first image is greater than the distance between the second object and the center of the second image. In the fused image, the first object is closer to the center than the second object. Generally, users pay more attention to objects in the center of an image and less to objects at the edges. Therefore, a smaller offset for objects at the edges of the image saves workload and has less impact on user experience. For example, when playing a video on an electronic device, users focus more on the content in the center of the screen; thus, the second object located at the top edge of the image can be offset less or not at all.

[0231] Condition 4: The number of user interactions with the first object is greater than the number of user interactions with the second object. Generally, objects with more interactions are those that users are interested in, while those with fewer interactions are not. Therefore, objects with fewer interactions have smaller offsets, saving workload and minimizing impact on user experience. For example, electronic devices can record the number of interactions between the user and each object. If the number of interactions with the first object is greater than a first threshold, the first object can be continuously offset. If the number of interactions with the second object is less than a second threshold, the second object is only offset, or offset more significantly, when it is located within the gaze area.

[0232] Condition 5: The first object is a user-specified object, while the second object is not. For example, if the user is more concerned with the first object, the user can choose to offset only the first object or offset the first object more significantly, according to their needs.

[0233] S1106, the first display device displays a third planar image, and the second display device displays a fourth planar image.

[0234] The third and fourth plane images are processed images. When the first display device displays the third plane image and the second display device displays the fourth plane image, the user will not experience a situation where binocular fusion cannot be achieved, and can see the virtual environment clearly and comfortably.

[0235] It should be noted that, in Figure 11 In the illustrated embodiment, a coordinate transformation matrix is ​​first used to process the 3D image data to obtain a first planar image and a second planar image, and then the first planar image and / or the second planar image are translated. In other embodiments, the coordinate transformation matrix can be adjusted first, and then the adjusted coordinate transformation matrix is ​​used to process the 3D image data to obtain a planar image. The planar image obtained in this way does not need to be translated again because the coordinate transformation matrix has already been adjusted, so the obtained planar image is a translated image.

[0236] For example, such as Figure 14 This is another schematic flowchart illustrating the display method provided in an embodiment of this application. The process includes:

[0237] S1501, acquire 3D image data.

[0238] S1502, obtain the first coordinate transformation matrix and the second coordinate transformation matrix. The first coordinate transformation matrix is ​​used to convert the 3D image data into a first planar image, and the second coordinate transformation matrix is ​​used to convert the 3D image data into a second planar image. The first planar image is displayed on the first display device, and the second planar image is displayed on the second display device.

[0239] S1503, obtain the first offset of the first display device on the VR glasses, and / or the second translation of the second display device.

[0240] For the implementation principles of S1501 to S1503, please refer to [link / reference needed]. Figure 11 The implementation principles of S1101 to S1103 will not be repeated here.

[0241] S1504, process the first coordinate transformation matrix according to the first offset to obtain the third coordinate transformation matrix, and / or process the second coordinate transformation matrix according to the second translation to obtain the fourth coordinate transformation matrix.

[0242] For example, let's take the first coordinate transformation matrix as follows:

[0243]

[0244] Assuming the first offset is (-x / 2, y / 2), update l, r, t, b in the first coordinate transformation matrix to lx / 2, rx / 2, t+y / 2, b+y / 2, and substitute them into the first coordinate transformation matrix to obtain the third coordinate transformation matrix, as follows:

[0245]

[0246] The above example uses the first coordinate transformation matrix to obtain the third coordinate transformation matrix. The principle of obtaining the fourth coordinate transformation matrix from the second coordinate transformation matrix is ​​the same, and will not be repeated.

[0247] S1505, the three-dimensional image data is processed according to the third coordinate system transformation matrix to obtain the first planar image, and / or, the three-dimensional image data is processed according to the fourth coordinate system transformation matrix to obtain the second planar image.

[0248] The first and second planar images do not require translation.

[0249] S1506, displaying a first planar image on a first display device, and / or displaying a second planar image on a second display device.

[0250] In various embodiments of this application, distance can be represented by the number of pixels. For example, in Figure 9 In the diagram, distances L9 and L10 can be represented in pixels. Distance L9 can be represented as: there are M1 pixels between the center point P9 of the overlapping region 9110 and the center point R9 of the image 9100; distance L10 can be represented as: there are M2 pixels between the center point P10 of the overlapping region 9210 and the center point R1 of the image 9200; if M1 is not equal to M2, then distance L9 is not equal to distance L10.

[0251] It should be noted that the preceding embodiments, assuming that the horizontal movement of display device 110 or display device 120 causes the center point W1 of the user's left eyeball to become misaligned with the center point S1 of display device 120, or the center point W2 of the user's right eyeball to become misaligned with the center point S2, are examples. In other embodiments, display device 110 or display device 120 may also move in other directions, such as vertically, causing the center point W1 of the user's left eyeball to become misaligned with the center point S1 of display device 120, or the center point W2 of the user's right eyeball to become misaligned with the center point S2. Alternatively, please refer to [link to relevant documentation]. Figure 15 The rotation angle of display device 110 In this case, the image displayed on the display device 110 has a projected image in the horizontal direction, and the center point of the projected image can be aligned with the center W2 of the right eyeball.

[0252] The above embodiments primarily use VR scenarios as examples, where the display method described in this specification is executed by a VR head-mounted display device. For AR scenarios, the display method described in this specification can be executed by an AR head-mounted display device. The AR head-mounted display device can be AR glasses, etc. For example, AR glasses include an optical engine for projecting image light. The light emitted by the optical engine can be guided onto the coupling grating of the lens of the AR glasses and exited into the human eye at the coupling grating position, thus allowing the user to see a virtual image corresponding to the image. The display method provided in this specification can be used for AR glasses with assembly deviations between the optical engine and the coupling grating.

[0253] AR glasses include monocular AR glasses and binocular AR glasses. In monocular AR glasses, at least a portion of one of the two lenses employs an optical waveguide structure; in binocular AR glasses, at least a portion of both lenses employ optical waveguide structures. An optical waveguide is a medium device that guides light waves to propagate within it; it is also called a dielectric optical waveguide. In some embodiments, an optical waveguide refers to an optical element that uses the principle of total internal reflection to guide light waves to propagate through total internal reflection. Common waveguide substrates can be guiding structures for transmitting optical frequency electromagnetic waves made of optically transparent media (such as quartz glass).

[0254] Although the description in this specification is presented in conjunction with some embodiments, this does not mean that the features of this application are limited to this embodiment. On the contrary, the purpose of describing the application in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of this specification. To provide a thorough understanding of this specification, many specific details will be included in the following description. This specification may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this specification, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features described in this specification can be combined with each other.

[0255] Figure 16 The image shows an electronic device 1600 provided in this application. This electronic device 1600 can be the VR head-mounted display device mentioned above. Figure 16 As shown, the electronic device 1600 may include: one or more processors 1601; one or more memories 1602; a communication interface 1603; and one or more computer programs 1604. These devices can be connected via one or more communication buses 1605. The one or more computer programs 1604 are stored in the memory 1602 and configured to be executed by the one or more processors 1601. The one or more computer programs 1604 include instructions that can be used to perform relevant steps of the mobile phone as described in the corresponding embodiments above. The communication interface 1603 is used to enable communication with other devices; for example, the communication interface may be a transceiver.

[0256] The methods provided in the embodiments of this application above are described from the perspective of an electronic device (e.g., a VR head-mounted display device) as the executing entity. To implement the functions of the methods provided in the embodiments of this application above, the electronic device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0257] In the above embodiments, the terms "when..." or "after..." can be interpreted, depending on the context, as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, the phrases "when..." or "if (the stated condition or event) is detected" can be interpreted, depending on the context, as meaning "if...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)". Furthermore, in the above embodiments, relational terms such as "first" and "second" are used to distinguish one entity from another, without limiting any actual relationship or order between these entities.

[0258] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0259] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)). Where there is no conflict, the solutions in the above embodiments can be used in combination.

[0260] It should be noted that a portion of this patent application contains copyrighted material. The copyright holder retains all rights except for making copies of the contents of patent documents or records from the patent office.

Claims

1. A display method, characterized in that, Applied to an electronic device, the electronic device including a first display screen and a second display screen, the method includes: The first image is displayed on the first display screen, which corresponds to the user's first glance. A second image is displayed on the second display screen, which corresponds to the user's second eye, wherein: There is an overlapping area between the first image and the second image, and the overlapping area includes at least one identical object; In the first image, the center point of the overlapping region is located at a first position; In the second image, the center point of the overlapping region is located at the second position; The distance from the first position to the center point of the first image is the first distance, and the direction is the first direction. The first distance and the first direction are related to the first offset of the first display screen relative to the first optical device. The first optical device is located between the first display screen and the first eye. The distance from the second position to the center point of the second image is the second distance, and the direction is the second direction. The second distance and the second direction are related to the second offset of the second display screen relative to the second optical device, and the second optical device is located between the second display screen and the second eye. If the first offset is not the same as the second offset, the first distance is not equal to the second distance, and / or the first direction is different from the second direction; When the first offset and the second offset are the same, the first distance is equal to the second distance, and the first direction is opposite to the second direction.

2. The method according to claim 1, characterized in that, The electronic device further includes a first optical device and a second optical device, wherein the first optical device corresponds to the first display screen and the second optical device corresponds to the second display screen, and the first optical device and the second optical device are symmetrical with respect to the intermediate plane; The first position and the second position are symmetrical with respect to the intermediate plane.

3. The method according to claim 2, characterized in that, The first display screen and the second display screen are asymmetrical with respect to the intermediate plane.

4. The method according to claim 2, characterized in that, The electronic device is a head-mounted display device. When the electronic device is worn by the user, the first display screen is located on the side of the first optical device that is away from the first eye, and the second display screen is located on the side of the second optical device that is away from the second eye.

5. The method according to claim 1, characterized in that, The electronic device is a head-mounted display device. When the electronic device is worn by a user, the first position and the second position are symmetrical with respect to the center line of the user's face.

6. The method according to claim 5, characterized in that, The first display screen and the second display screen are asymmetrical with respect to the center line of the face.

7. The method according to any one of claims 1-6, characterized in that, The first position changes as the position of the first display screen changes.

8. The method according to claim 7, characterized in that, When the first display screen moves in a third direction, the overlapping area on the first image moves in the opposite direction to the third direction.

9. The method according to any one of claims 1-6, characterized in that, The second position changes as the position of the second display screen changes.

10. The method according to claim 9, characterized in that, When the second display screen moves in the fourth direction, the overlapping area on the second image moves in the opposite direction to the fourth direction.

11. The method according to any one of claims 1-6, characterized in that, Before displaying the first image on the first display screen and the second image on the second display screen, the method further includes adjusting the interpupillary distance of the first display screen and the second display screen. The interpupillary distance adjustment includes: moving the first display screen a certain distance along a fifth direction, and moving the second display screen the same distance along a sixth direction opposite to the fifth direction; wherein, the fifth direction is the direction in which the first display screen moves away from the second display screen, or the fifth direction is the direction in which the first display screen moves closer to the second display screen. The distance difference between the first distance and the second distance remains unchanged compared to before the interpupillary distance adjustment, and the relative relationship between the first direction and the second direction remains unchanged compared to before the interpupillary distance adjustment.

12. The method according to any one of claims 1-6, characterized in that, The at least one identical object includes a first object and a second object; In the first image, the first feature point of the first object is located at the first coordinate, and the second feature point of the second object is located at the second coordinate; In the second image, the first feature point of the first object is located at the third coordinate, and the second feature point of the second object is located at the fourth coordinate. The coordinate difference between the first coordinate and the third coordinate is different from the coordinate difference between the second coordinate and the fourth coordinate.

13. The method according to claim 12, characterized in that, The method further includes: The coordinate difference between the first coordinate and the third coordinate is greater than the coordinate difference between the second coordinate and the fourth coordinate when at least one of the following conditions is met, said conditions include: The first object is located within the area where the user's gaze point is located, and the second object is located outside the area where the user's gaze point is located. Both the first object and the second object are located within the area where the user gaze point is located, and the second object is closer to the edge of the area where the user gaze point is located than the first object; The distance between the first object and the center point of the first image is less than the distance between the second object and the center of the second image; The number of user interactions corresponding to the first object is greater than the number of user interactions corresponding to the second object; or, The first object is a user-specified object, while the second object is not a user-specified object.

14. The method according to claim 1, characterized in that, The method further includes: Acquire 3D image data; Obtain a first coordinate transformation matrix and a second coordinate transformation matrix, wherein the first coordinate transformation matrix corresponds to the first optical device and the second coordinate transformation matrix corresponds to the second optical device; Obtain the first offset and the second offset; Based on the first coordinate transformation matrix and the first offset, the three-dimensional image data is processed into the first image; Based on the second coordinate transformation matrix and the second offset, the three-dimensional image data is processed into the second image.

15. The method according to claim 14, characterized in that, When the position of the first display module changes, the first coordinate transformation matrix changes; or... When the position of the second display module changes, the second coordinate transformation matrix changes.

16. An electronic device, characterized in that, include: Processor, memory, and one or more programs; The one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-15.

18. A computer program product, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-15.

Images