External display system, head-mounted display device, control method and storage medium

Abstract

The invention provides an external display system of head-mounted display equipment, the head-mounted display equipment, a control method of the head-mounted display equipment and a computer readable storage medium. The external display system of the head-mounted display device comprises a face image generation module, a display image generation module, a layer aliasing module and an external display screen. The facial image generation module acquires facial data of a user to generate a facial image. And the display image generation module acquires internal display data of the head-mounted display equipment to generate a corresponding external display image. The layer aliasing module performs layer aliasing on the face image and the external display image according to depth and transparency to generate an aliasing image. And the external display screen is positioned on the outer side of the head-mounted display equipment and is back to the face of the user so as to display the aliasing image outwards.

Description

Technical Field

[0001] The present invention relates to the field of extended reality technology, and in particular to an external display system of a head-mounted display device, a head-mounted display device, a control method for the head-mounted display device, and a computer-readable storage medium. Background Art

[0002] In recent years, new virtual reality (VR) and mixed reality (MR) head-mounted display devices have been continuously appearing on the market. However, when these existing head-mounted display devices are used, they generally cannot allow outside individuals to understand the images that the user sees, which is not conducive to the user communicating with outside individuals about the content of the images. In addition, since the opaque and closed appearance commonly used in existing head-mounted display devices will largely block the user's face, it will not only affect the appearance of the user when wearing it, but also affect the outside individuals from observing the user's expression, thus causing inconvenience for the user to communicate with outside individuals.

[0003] In order to overcome the above-mentioned defects of the prior art, there is an urgent need in the art for an improved head-mounted display device for displaying an aliased image of the user's face externally and the content internally, so as to facilitate normal communication and interaction between external individuals and the user. Summary of the invention

[0004] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides an external display system of a head-mounted display device, a head-mounted display device, a control method of the head-mounted display device, and a computer-readable storage medium, which can display an aliased image of the user's face externally and the content displayed internally, so that an external individual can see the user's facial expression and what the user sees at the same time, thereby facilitating normal communication and interaction between the external individual and the user.

[0006] Specifically, the external display system of the head-mounted display device provided according to the first aspect of the present invention includes a facial image generation module, a display image generation module, a layer aliasing module and an external display screen. The facial image generation module obtains the facial data of the user to generate a facial image. The display image generation module obtains the internal display data of the head-mounted display device to generate a corresponding external display image. The layer aliasing module performs layer aliasing on the facial image and the external display image according to the depth and transparency to generate an aliased image. The external display screen is located on the outside of the head-mounted display device and faces away from the user's face to display the aliased image externally.

[0007] Further, in some embodiments of the present invention, the facial image generation module is connected to at least one image sensor and at least one depth sensor. The image sensor faces the user's face to collect image data of the user's face. The depth sensor faces the user's face to collect first depth data of the user's face. The facial image generation module generates a 3D facial image with depth information based on the image data of the user's face and the first depth data.

[0008] Furthermore, in some embodiments of the present invention, a first interpolation table is further configured in the facial image generation module. The facial image generation module is further configured to: interpolate the 3D facial image under a single viewing angle according to the first interpolation table to obtain 3D facial images under multiple viewing angles; and transmit the 3D facial images under multiple viewing angles to the layer aliasing module to perform layer aliasing with the external display images under the corresponding viewing angles.

[0009] Further, in some embodiments of the present invention, the first interpolation table is divided in the form of a triangular mesh, and stores the texture coordinates, spatial coordinates and mesh information of each mesh vertex therein. The step of interpolating the 3D facial image under a single viewing angle according to the first interpolation table to obtain 3D facial images under multiple viewing angles includes: determining the triangular meshes corresponding to each first pixel according to the pixel indexes of the first pixels in the 3D facial image under a single viewing angle and the spatial coordinates and mesh information of each mesh vertex stored in the first interpolation table; performing triangular interpolation according to the texture coordinates of the mesh vertices of each triangular mesh and the relative positions of each first pixel and the corresponding mesh vertex to determine the texture coordinates of the second pixels in each triangular mesh under the corresponding viewing angle; and determining the 3D facial images under the corresponding viewing angle according to the texture coordinates of the first pixels and the second pixels under multiple viewing angles.

[0010] Furthermore, in some embodiments of the present invention, the display image generation module is further configured to: generate a virtual object image to be displayed, and determine the corresponding second depth data; and generate a 3D virtual object image with depth information according to the virtual object image and the second depth data. The layer aliasing module is further configured to: perform layer aliasing on the 3D facial image, the external display image and the 3D virtual object image according to the depth and transparency to generate an aliased image.

[0011] Furthermore, in some embodiments of the present invention, the display image generation module is further configured with a second interpolation table corresponding to the 3D virtual object image. The display image generation module is further configured to: interpolate the 3D virtual object image under a single viewing angle according to the second interpolation table to obtain 3D virtual object images under multiple viewing angles.

[0012] Furthermore, in some embodiments of the present invention, the layer aliasing module is configured with a first lookup table corresponding to a first depth of the 3D facial image, and a second lookup table corresponding to a second depth of the 3D virtual object image, and is configured to: perform depth-based layer aliasing on the 3D facial image and the 3D virtual object image according to the first lookup table and the second lookup table, adjust the display content and display position of the 3D facial image and / or the 3D virtual object image, so as to generate a 2D aliasing image with 3D visual effects.

[0013] Furthermore, in some embodiments of the present invention, the external display screen is configured with a microlens array composed of multiple microlens units, and is configured to: obtain 3D facial images of multiple perspectives and 3D virtual object images of multiple perspectives; and through the microlens array, according to the perspectives corresponding to each of the 3D facial images and each of the 3D virtual object images, control the light emitted by each pixel unit in the external display screen in the corresponding direction, so as to display the corresponding aliased image in the direction of each of the perspectives, respectively.

[0014] Furthermore, in some embodiments of the present invention, the microlens unit is columnar, the long side direction of which is different from the column direction of the pixel unit of the external display screen, and each of the microlens units corresponds to a non-integer number of pixel units.

[0015] Furthermore, in some embodiments of the present invention, the external display system further includes a feedback unit for sensing a viewer position of at least one external viewer and adjusting a display method of the aliased image according to the viewer position of the at least one external viewer.

[0016] Further, in some embodiments of the present invention, the external display image includes a mirrored 2D user interface image. The step of performing layer aliasing on the 3D facial image, the external display image, and the 3D virtual object image according to the depth and transparency to generate the aliased image includes: obtaining the 3D virtual object image and its corresponding second depth and second transparency; obtaining the mirrored 2D user interface image and its corresponding third transparency, and the third depth corresponding to the external display screen; and performing layer aliasing on the facial image, the virtual object image, and the mirrored user interface image according to the first depth, the second depth, the second transparency, the third depth, and the third transparency corresponding to the facial image to generate the aliased image.

[0017] Furthermore, in some embodiments of the present invention, the layer blending module is also connected to an ambient light sensor and is further configured to: determine the second brightness of the second layer where the 3D virtual object image is located; determine the third brightness of the third layer where the mirrored 2D user interface image is located; obtain an ambient light signal via the ambient light sensor; determine the first brightness of the first layer where the facial image is located based on the ambient light signal, adjust the second brightness to determine the second corrected brightness, and adjust the third brightness to determine the third corrected brightness; and perform layer blending on the facial image, the 3D virtual object image and the mirrored 2D user interface image based on the first brightness, the second corrected brightness and the third corrected brightness to generate the aliased image.

[0018] Further, in some embodiments of the present invention, the layer aliasing module is further configured to: determine a first occlusion attenuation coefficient of the third layer to the first layer according to a third transparency of the third layer; determine a second occlusion attenuation coefficient of the second layer to the first layer and a third occlusion attenuation coefficient of the second layer to the third layer according to a second transparency of the second layer; determine a first adjustment value of the first layer, a second adjustment value of the second layer and a third adjustment value of the third layer according to the ambient light signal, the first occlusion attenuation coefficient, the second occlusion attenuation coefficient and the third occlusion attenuation coefficient; in response to generating the aliased image, determine whether the maximum brightness of the aliased image is greater than the maximum display brightness of the external display screen; and in response to the judgment result that the maximum brightness of the aliased image is greater than the maximum display brightness of the external display screen, determine a tone mapping value according to the ratio of the maximum brightness of the aliased image to the maximum display brightness of the external display screen, and perform tone mapping on the RGB grayscale value of each pixel in the aliased image according to the tone mapping value, the first adjustment value, the second adjustment value and the third adjustment value, so that the maximum brightness is less than or equal to the maximum display brightness of the external display screen.

[0019] In addition, the head-mounted display device provided according to the second aspect of the present invention includes an internal display system and an external display system provided according to the first aspect of the present invention. The internal display system includes an internal display screen for displaying a virtual reality image generated by the display system to a first user wearing the head-mounted display device. The external display system includes an external display screen for displaying a facial image of the first user and an aliased image of the internal display image displayed by the internal display screen to a second user facing the first user.

[0020] In addition, the control method of the above-mentioned head-mounted display device provided according to the third aspect of the present invention includes the following steps: obtaining the user's facial data to generate a facial image; obtaining the internal display data of the head-mounted display device to generate a corresponding external display image; performing layer aliasing on the facial image and the external display image according to the depth and transparency to generate an aliased image; and displaying the aliased image externally via an external display screen located on the outside of the head-mounted display device and facing away from the user's face.

[0021] In addition, the computer-readable storage medium provided in the fourth aspect of the present invention stores computer instructions, which, when executed by a processor, implement the control method of the head mounted display device provided in the third aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0023] Figure 1 A schematic diagram of the architecture of a head mounted display device provided according to some embodiments of the present invention is shown.

[0024] Figure 2 A schematic flow chart of a control method for a head mounted display device provided according to some embodiments of the present invention is shown.

[0025] Figure 3 A schematic diagram of facial depth provided according to some embodiments of the present invention is shown.

[0026] Figure 4 A schematic diagram of a facial image divided in the form of a triangular mesh according to some embodiments of the present invention is shown.

[0027] Figure 5 A schematic diagram of generating a mirrored external display image provided according to some embodiments of the present invention is shown.

[0028] Fig. 6A and Figure 6B A schematic diagram of a three-dimensional object provided according to some embodiments of the present invention is shown.

[0029] Fig. 7A up Figure 7C A schematic diagram of an aliased image provided according to some embodiments of the present invention is shown.

[0030] Figure 8 A schematic diagram of pixel-level layer aliasing provided according to some embodiments of the present invention is shown.

[0031] Fig. 9 A schematic diagram of an external display screen for multi-viewing angle display provided according to some embodiments of the present invention is shown.

[0032] Fig.10 Schematic diagrams of microarray units provided according to some embodiments of the present invention are shown.

[0033] Fig.11 A schematic diagram of a multi-viewing angle display provided according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0034] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0035] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0037] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0038] As mentioned above, when using existing virtual reality (VR) and mixed reality (MR) head-mounted display devices, it is generally impossible for outside individuals to understand the images seen by the user, which is not conducive to the user communicating with outside individuals about the content of the images. In addition, since the opaque and closed appearance commonly used in existing head-mounted display devices will largely block the user's face, it will not only affect the appearance of the user when wearing it, but also affect the outside individuals from observing the user's expression, thus causing inconvenience for the user to communicate with outside individuals.

[0039] In order to overcome the above-mentioned defects of the prior art, the present invention provides an external display system of a head-mounted display device, a head-mounted display device, a control method of the head-mounted display device, and a computer-readable storage medium, which can display an aliased image of the user's face externally and the content displayed internally, so that an external individual can see the user's facial expression and what the user sees at the same time, thereby facilitating normal communication and interaction between the external individual and the user.

[0040] In some non-limiting embodiments, the control method of the head mounted display device provided in the third aspect of the present invention may be implemented via the head mounted display device provided in the second aspect of the present invention. Figure 1 , Figure 1 A schematic diagram of the architecture of a head mounted display device provided according to some embodiments of the present invention is shown.

[0041] like Figure 1 As shown, the head mounted display device 10 provided in the second aspect of the present invention may be configured with an internal display system 11 and the external display system 12 provided in the first aspect of the present invention. The internal display system 11 may include an internal display screen 11. The internal display screen 11 is located on the inner side of the head mounted display device 10 and faces the first user wearing the head mounted display device 10 to show the virtual reality image generated by the internal display system 11 to the user. In addition, the external display system 12 may include a facial image generation module 121, a display image generation module 122, a layer aliasing module 123 and an external display screen 124. The facial image generation module 121 is used to obtain facial data of the first user wearing the head mounted display device 10 to generate a facial image. Here, the facial image may be a static facial image or a dynamic facial expression image. The display image generation module 122 is used to obtain the internal display data of the head mounted display device 10 from the internal display system 11 and generate a corresponding external display image. The layer aliasing module 123 is used to perform layer aliasing on the facial image of the first user and the external display image according to the depth and transparency to generate an aliased image. The external display screen 124 is located outside the head mounted display device 10 and faces away from the face of the first user to display the aliased image to the external second user. Here, the external display screen 124 can be a flat screen or a curved screen.

[0042] In addition, in some non-limiting embodiments, the external display system 12 provided in the first aspect of the present invention may also be configured with a memory and a processor. The memory includes but is not limited to the computer-readable storage medium provided in the fourth aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored in the memory to implement the control method of the head-mounted display device provided in the third aspect of the present invention.

[0043] The following will describe the working principles of the head-mounted display device 10 and the external display system 12 in conjunction with some embodiments of the control method of the head-mounted display device 10. Those skilled in the art will understand that these embodiments of the control method of the head-mounted display device 10 are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the head-mounted display device 10 and the external display system 12. Similarly, the head-mounted display device 10 and the external display system 12 are also only a non-limiting implementation method provided by the present invention, and do not constitute a limitation on the execution subject or execution order of each step in these control methods.

[0044] Please refer to Figure 1 and Figure 2 , Figure 2 A schematic flow chart of a control method for a head mounted display device provided according to some embodiments of the present invention is shown.

[0045] like Figure 1 and Figure 2 As shown, in the process of externally displaying the aliased image, the external display system can first obtain the user's facial data via the facial image generation module 121 to generate a facial image, and obtain the internal display data of the head-mounted display device 10 from the internal display system 11 via the image generation module 122 to generate a corresponding external display image.

[0046] Please refer to Figure 3 , Figure 3 A schematic diagram of facial depth provided according to some embodiments of the present invention is shown.

[0047] exist Figure 3In the illustrated embodiment, the facial image generation module 121 is connected to at least one image sensor and at least one depth sensor. In the process of generating a facial image, the facial image generation module 121 can first obtain image data of the first user's face and first depth data from various positions of the first user's face to the external display screen 124 through the image sensor and the depth sensor facing the first user's face, and then generate a 3D facial image with depth information based on the image data and the first depth data of the first user's face.

[0048] In some embodiments, the above-mentioned depth sensor can be arranged on the inner side of the head-mounted display device 10 to collect first depth data of the first user's face in real time, and update the facial image of the first user by combining the first depth data collected in real time and the facial image data collected in real time.

[0049] Optionally, in some other embodiments, the depth sensor may be an external device such as a mobile phone that is disposed on the outside of the head mounted display device 10, thereby pre-modeling the first user's face by pre-collecting and storing first depth data of various positions of the first user's face to the external display screen 124, and then tracking the user's facial expressions in combination with the facial image data collected in real time to update the facial image of the first user.

[0050] Furthermore, in some embodiments, the facial image generation module 121 may also be preferably configured with a first interpolation table for compressing and storing the 3D facial image information of the first user to reduce the storage space required. Thus, after obtaining the 3D facial image of the first user in a single viewing angle, the facial image generation module 121 may also interpolate the 3D facial image in a single viewing angle according to the first interpolation table to obtain 3D facial images in multiple viewing angles.

[0051] Please refer to Figure 4 , Figure 4 A schematic diagram of a facial image divided in the form of a triangular mesh according to some embodiments of the present invention is shown.

[0052] exist Figure 4In the illustrated embodiment, the first interpolation table is divided in the form of a triangular mesh, which stores the texture coordinates, spatial coordinates and mesh information of each mesh vertex in the 3D facial image. When interpolating the 3D facial image under a single viewing angle, the facial image generation module 121 can first determine the triangular mesh corresponding to each first pixel according to the pixel index of the multiple first pixels in the 3D facial image under a single viewing angle, and the spatial coordinates and mesh information of each mesh vertex stored in the first interpolation table, and then perform triangular interpolation according to the texture coordinates of the mesh vertices of each triangular mesh and the relative positions of each first pixel and the corresponding mesh vertex, so as to respectively determine the texture coordinates of multiple second pixels in each triangular mesh under the corresponding viewing angle. Afterwards, the facial image generation module 121 can determine the 3D facial images under each corresponding viewing angle according to the texture coordinates of the first pixel and the multiple second pixels under multiple viewing angles. Afterwards, the facial image generation module 121 can transmit the 3D facial images under multiple viewing angles to the layer aliasing module 123, and perform layer aliasing with the external display images under the corresponding viewing angles to generate aliased images under multiple viewing angles.

[0053] Thus, by using triangular meshes to divide the face, the present invention can achieve naked-eye 3D visual effects by presenting the texture of each triangular mesh at a corresponding angle in each light-emitting direction. In addition, by compressing the information of the entire face into information related to mesh vertices, the present invention can greatly compress the storage space required for multi-view 3D facial images, thereby reducing the hardware cost of the head-mounted display device 10 and improving the display accuracy it supports.

[0054] Those skilled in the art will appreciate that the above-mentioned embodiments of interpolating a 3D facial image under a single viewing angle based on the first interpolation table to generate 3D facial images under multiple viewing angles are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concepts of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit the scope of protection of the present invention.

[0055] Optionally, in other embodiments, those skilled in the art may also configure multiple groups of image sensors and depth sensors to achieve the same technical effect of generating 3D facial images from multiple perspectives.

[0056] In addition, please refer to Figure 5 . Figure 5 A schematic diagram of generating a mirrored external display image provided according to some embodiments of the present invention is shown.

[0057] like Figure 5As shown, in the process of generating an external display image, the display image generation module 122 can obtain the internal display data such as the 2D user interface image displayed to the first user wearing the head-mounted display device 10 from the internal display system 11 of the head-mounted display device 10 as described above, and perform mirroring on it to generate a mirrored external display image, so as to present the field of view displayed by the first user in the form of a mirror-reflected image through the curved or flat external display screen 124, so as to give the external second user a visual perception of the external image reflected through the goggles.

[0058] In addition, please refer to Fig. 6A and Figure 6B . Fig. 6A and Figure 6B A schematic diagram of a three-dimensional object provided according to some embodiments of the present invention is shown.

[0059] like Fig. 6A and Figure 6B As shown, the external display image may also preferably include a 3D virtual object image with depth information. Specifically, in the process of generating a 3D virtual object image, the display image generation module 122 may first generate a virtual object image to be displayed and determine its corresponding second depth data. Here, the virtual object image includes but is not limited to three-dimensional geometric images such as spheres, polyhedrons, and polyhedrons, as well as simulated or cartoon three-dimensional images of virtual objects such as glasses, toys, tables and chairs. The second depth may be less than the standard depth corresponding to the external display screen 124 (for example: 0) to highlight the external display screen 124, or it may be greater than the standard depth corresponding to the external display screen 124 to sink into the external display screen 124. Afterwards, the display image generation module 122 may generate a 3D virtual object image with depth information based on the virtual object image and its corresponding second depth data.

[0060] Further, for the above-mentioned 3D virtual object image with depth information, a second interpolation table corresponding to the 3D virtual object image may also be preferably configured in the display image generation module 122. In response to generating the 3D virtual object image under a single viewing angle, the display image generation module may also preferably interpolate the 3D virtual object image under a single viewing angle according to the second interpolation table to obtain 3D virtual object images under multiple viewing angles.

[0061] Furthermore, the second interpolation table is divided in the form of a triangular mesh, which stores the texture coordinates, spatial coordinates and mesh information of each mesh vertex in the 3D virtual object image. When interpolating the 3D virtual object image under a single perspective, the display image generation module 122 can first determine the triangular mesh corresponding to each first pixel according to the pixel index of the multiple first pixels in the 3D virtual object image under a single perspective, and the spatial coordinates and mesh information of each mesh vertex stored in the second interpolation table, and then perform triangular interpolation according to the texture coordinates of the mesh vertices of each triangular mesh and the relative position of each first pixel and the corresponding mesh vertex, so as to determine the texture coordinates of multiple second pixels in each triangular mesh under the corresponding perspective. After that, the display image generation module 122 can determine the 3D virtual object image under the corresponding perspective according to the texture coordinates of the first pixel and multiple second pixels under multiple perspectives. Afterwards, the display image generation module 122 may transmit the 3D virtual object images under the multiple viewing angles to the layer aliasing module 123 to perform layer aliasing with the facial images under the corresponding viewing angles, so as to generate aliased images under the multiple viewing angles.

[0062] Thus, by using triangular meshes to divide virtual objects, the present invention can achieve naked-eye 3D visual effects by presenting the texture of each triangular mesh at a corresponding angle in each light-emitting direction. In addition, by compressing the information of the entire virtual object into information related to mesh vertices, the present invention can greatly compress the storage space required for multi-view 3D virtual object images, thereby reducing the hardware cost of the head-mounted display device 10 and improving the display accuracy it supports.

[0063] Those skilled in the art will appreciate that, although the above describes an embodiment in which the display image generation module 122 generates a 3D virtual object image by itself, these are merely some non-limiting implementations provided by the present invention, which are intended to clearly demonstrate the main concepts of the present invention and to provide some specific solutions that are convenient for the public to implement, rather than to limit the scope of protection of the present invention.

[0064] Optionally, in other embodiments, the internal display system 11 of the head mounted display device 10 with a mixed reality (MR) display function may also generate a 3D virtual object image, and display the 3D virtual object image to a first user wearing the head mounted display device 10 via the internal display screen 111. In this regard, the display image generation module 122 may directly obtain the displayed 3D virtual object image and its corresponding second depth data from the internal display system 11, and generate a 3D virtual object image for an external second user to observe the virtual object based on the obtained image data and the second depth data, thereby further reducing the data processing load on the external display system 12 by multiplexing the image data and the second depth data of the virtual object.

[0065] Please refer to FIG. 7A to FIG. 7C , FIG. 7A to FIG. 7C A schematic diagram of an aliased image provided according to some embodiments of the present invention is shown.

[0066] like FIG. 7A to FIG. 7C As shown, after generating the facial image and the external display image, the layer aliasing module 123 can perform layer aliasing on at least two images of the 2D or 3D facial image, the 2D external display image, and the 3D virtual object image according to the depth and transparency to generate an aliased image.

[0067] For example, in Fig. 7A In the illustrated embodiment, the layer aliasing module 123 may perform layer aliasing on the 2D or 3D facial image and the 3D virtual earth image to obtain a corresponding first aliased image.

[0068] For example, in Figure 7B In the illustrated embodiment, the layer aliasing module 123 may perform layer aliasing on the 2D or 3D facial image and the mirrored 2D user interface image to obtain a corresponding second aliased image.

[0069] For example, in Figure 7C In the illustrated embodiment, the layer aliasing module 123 may perform layer aliasing on the 2D or 3D facial image, the mirrored 2D user interface image, and the 3D virtual earth image to obtain a corresponding third aliased image.

[0070] Furthermore, for Fig. 7A and Figure 7C In the specific application of the aliased 3D facial image and 3D virtual object image shown, the layer aliasing module 123 may preferably be configured with a first lookup table corresponding to the first depth of the 3D facial image and a second lookup table corresponding to the second depth of the 3D virtual object image. When performing layer aliasing on the 3D facial image and the 3D virtual object image, the layer aliasing module 123 may perform depth-based layer aliasing on the 3D facial image and the 3D virtual object image according to the first lookup table and the second lookup table, and adjust the display content and display position of the 3D facial image and / or the 3D virtual object image to generate a 2D aliased image with a 3D visual effect.

[0071] Specifically, in the process of layer aliasing, the layer aliasing module 123 may first determine the covering relationship between the layers of the 3D facial image, the 3D virtual object image and / or the 2D user interface image according to the first depth of the 3D facial image, the second depth of the 3D virtual object image and / or the third depth of the 2D user interface image. Afterwards, the layer aliasing module 123 may determine the covering area of ​​the upper layer image on the lower layer image under multiple viewing angles according to the spatial position of at least one non-transparent object in the upper layer image compared to the lower layer image. Afterwards, the layer aliasing module 123 may query the first lookup table of the 3D facial image and the second lookup table of the 3D virtual object image to adjust the display position of each object in the 3D facial image and / or the 3D virtual object image, cover the covering area of ​​the lower layer image under the corresponding viewing angle with the non-transparent object of the upper layer image, and display or supplement the display content of the exposed area of ​​the lower layer image under the corresponding viewing angle to generate a 2D aliasing image with 3D visual effect.

[0072] Furthermore, in Figure 7C In the layer aliasing application involving the mirrored 2D user interface image shown, the layer aliasing module 123 may first obtain the 3D virtual object image and its corresponding second depth and second transparency, and obtain the mirrored 2D user interface image and its corresponding third transparency, and the third depth corresponding to the external display screen 124. Afterwards, the layer aliasing module 123 may perform layer aliasing on the facial image, the virtual object image, and the mirrored user interface image according to the first depth corresponding to the facial image, the second depth and second transparency corresponding to the 3D virtual object image, and the third depth and third transparency corresponding to the mirrored 2D user interface image, to generate an aliased image.

[0073] In addition, please refer to Figure 8 . Figure 8 A schematic diagram of pixel-level layer aliasing provided according to some embodiments of the present invention is shown.

[0074] exist Figure 8 In the illustrated embodiment, the layer aliasing module 123 may also be preferably connected to an ambient light sensor. In the process of performing pixel-level layer aliasing on the 3D facial image, the external display image, and the 3D virtual object image, for the pixel point Pixel_out (x, y), the layer aliasing module 123 may first determine the second brightness C0 (x, y) of the second layer where the 3D virtual object image is located according to the virtual object layer effect requirements, determine the third brightness B0 (x, y) of the third layer where the mirrored 2D user interface image is located according to the artistic effect requirements, and obtain the ambient light signal via the ambient light sensor.

[0075] Then, the layer aliasing module 123 can determine the first brightness adjustment coefficient L according to the ambient light signal and the reflection coefficient of human skin.a , combined with the image brightness A0(x, y) collected by the virtual human facial expression camera, and the first brightness A1(x, y) of the first layer where the facial image is located is determined through a multiplication operation.

[0076] In addition, the layer aliasing module 123 can also obtain an ambient light signal according to the ambient light sensor to determine the third brightness adjustment coefficient L b , and then adjust the third brightness B0(x, y) accordingly, and determine the third corrected brightness B1(x, y) through a multiplication operation.

[0077] In addition, the layer aliasing module 123 can also obtain an ambient light signal according to the ambient light sensor to determine the second brightness adjustment coefficient L c , and then adjust the above-mentioned second brightness C0(x, y) accordingly, and determine the second corrected brightness C1(x, y) through multiplication operation.

[0078] Thereafter, the layer blending module 123 may perform layer blending on the 3D facial image, 3D virtual object image and mirrored 2D user interface image according to the first brightness A1(x, y), the third corrected brightness B1(x, y) and the second corrected brightness C1(x, y) to generate a blended image of corresponding brightness.

[0079] Furthermore, in the process of performing pixel-level layer aliasing on the 3D facial image, the external display image and the 3D virtual object image, the layer aliasing module 123 can also preferably determine the first occlusion attenuation coefficient BA(x, y) of the third layer to the first layer according to the third transparency design of the UI layer display image of the third layer, and determine the second occlusion attenuation coefficient CA(x, y) of the second layer to the first layer and the third occlusion attenuation coefficient CB(x, y) of the second layer to the third layer according to the second transparency design of the virtual object of the second layer.

[0080] Then, the layer aliasing module 123 can adjust the first brightness coefficient L according to the ambient light signal. a , the first occlusion attenuation coefficient BA(x, y) and the second occlusion attenuation coefficient CA(x, y), and then determine the first adjustment value alphaA(x, y) of the first layer through a product operation, and according to the third brightness adjustment coefficient L related to the ambient light signal b and the third shading attenuation coefficient CB(x, y), and then determine the third adjustment value alphaB(x, y) of the third layer through a product operation, and according to the second brightness adjustment coefficient L related to the ambient light signal c , determine the second adjustment value alphaC(x, y) of the second layer.

[0081] Then, the layer aliasing module 123 can calculate the grayscale value of the pixel point (x, y) corresponding to the generated aliased image according to the grayscale values ​​Pixel_A, Pixel_B, Pixel_C of the pixel point (x, y) corresponding to each layer and the corresponding first adjustment value alphaA, third adjustment value alphaB and second adjustment value alphaC:

[0082] Pixel_out=Pixel_A×alphaA+Pixel_B×alphaB+Pixel_C×alphaC

[0083] Preferably, the maximum brightness L of the generated aliased image is determined according to the grayscale value of each pixel of each layer and the occlusion attenuation coefficient BA(x, y), CA(x, y), and CB(x, y) of each upper layer on its lower layer. max . For example, L max The calculation method is as follows:

[0084] L max =max(A1×BA×CA+B1×CB+C1)

[0085] Afterwards, whenever a new aliased image is generated, the layer aliasing module 123 can determine the maximum brightness L of the aliased image. max Is it greater than the maximum display brightness L of the external display screen 124? pmax . In response to the maximum brightness L of the aliased image max Greater than the maximum display brightness L of the external display screen 124 pmax The layer aliasing module 123 may further determine the maximum brightness L of the aliased image according to the judgment result. max The maximum display brightness L of the external display screen 124 pmax The ratio of , determines the tone mapping value TM[GL0(x, y)] of each layer, and performs tone mapping (Tone Mapping, TM) on the RGB grayscale value of each pixel in the aliased image according to the tone mapping value TM[GL0(x, y)], the first adjustment value alphaA(x, y), the second adjustment value alphaC(x, y) and the third adjustment value alphaB(x, y), so that the maximum brightness of the aliased image is less than or equal to the maximum display brightness of the external display screen, that is, L max ≤L pmax For example, the original RGB grayscale value of each pixel (x, y) in the aliased image can be calculated using the following formula:

[0086] GL0(x, y) = GL A (x, y)×alphaA(x, y)+GL B(x, y)×alphaB(x, y)+GL C (x, y)×alphaC(x, y)

[0087] The grayscale value of each pixel (x, y) in the aliased image after tone mapping can be expressed as:

[0088] GL1(x,y)=GL0(x,y)×TM[GL0(x,y)].

[0089] So, in FIG. 7A to FIG. 7C In the illustrated embodiment, due to the different transparency of different regions of the layer, when performing aliasing images, the layer aliasing module 123 can overlay and display a 2D user interface image with uneven transparency and a partially transparent 3D virtual object layer on the 3D facial image layer, thereby using the low-transparency edge of the 2D user interface and the opaque 3D virtual object to partially or completely block the 3D facial image below. By performing pixel-level layer aliasing display, the present invention can more flexibly implement the layer overlay function, thereby displaying the incomplete occlusion between layers to optimize the user experience.

[0090] Please continue to refer to Figure 1 and Figure 2 After generating the aliasing image, the external display system 12 can display the aliasing image to the external second user via the external display screen 124 located on the outside of the head-mounted display device 10 and facing away from the face of the first user.

[0091] Further, please refer to Figures 9 to 11 . Fig. 9 A schematic diagram of an external display screen for multi-viewing angle display provided according to some embodiments of the present invention is shown. Fig.10 Schematic diagrams of microarray units provided according to some embodiments of the present invention are shown. Fig.11 A schematic diagram of a multi-viewing angle display provided according to some embodiments of the present invention is shown.

[0092] exist Fig. 9 In the illustrated embodiment, the external display screen 124 may preferably be configured with a microlens array 91 composed of a plurality of microlens units. When displaying the aliased image to the external second user, the external display screen 124 may obtain the 3D facial images of the plurality of viewing angles, the 3D virtual object images of the plurality of viewing angles, and / or the aliased images of the plurality of viewing angles, and then, through the microlens array 91, according to the viewing angles corresponding to the 3D facial images, the 3D virtual object images, and / or the aliased images, control the light emitted in the corresponding directions by each pixel unit in the external display screen 124, so as to display the corresponding aliased images in the directions of the respective viewing angles.

[0093] Further, in Fig.10 In the illustrated embodiment, the present invention can preferably arrange each microlens unit 101 to be cylindrical, the long side direction of which is different from the column direction of the pixel unit facing the external display screen 124, and each microlens unit 101 corresponds to a non-integer number of pixel units, so that the angle direction of refraction of each pixel point is densely and evenly distributed in space, so as to eliminate the discontinuity phenomenon when the user switches the viewing angle.

[0094] Furthermore, in some embodiments, the external display system 12 provided in the first aspect of the present invention may also preferably include a feedback unit. When displaying the aliased image externally, the external display system 12 may also sense the viewer position of at least one external viewer via the feedback unit, and adjust the display mode of the aliased image according to the average viewer position of the at least one external viewer.

[0095] So, like Fig.11 As shown, the optical mirror on the external display screen 124 can limit the light emitted by each pixel point to a specific direction, and display a light signal of a corresponding angle to the external second user according to the different relative positions between the mirror and the external user, so as to achieve the effect of enabling users at different viewing angles to see different images respectively. For example, the second user located on the left side of the first user 110 can observe the left face of the first user 110 from the external display screen 124, the second user located in front of the first user 110 can simultaneously observe the front face of the first user 110 from the external display screen 124, and the second user located on the right side of the first user 110 can simultaneously observe the right face of the first user 110 from the external display screen 124.

[0096] In summary, the above-mentioned head-mounted display device 10 provided by the present invention not only has a glassy, ​​frameless visual effect, which can greatly improve the attractiveness of the product appearance, but its supporting external display system 12, the control method of the head-mounted display device, and the computer-readable storage medium can also display the face of the first user wearing the head-mounted display device 10 and the aliased image of the internal display content seen by the first user to the external second user, so that the external individual can see the facial expression and what he sees of the first user at the same time, thereby facilitating normal communication and interaction between the external individual and the first user.

[0097] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0098] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0099] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0100] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An external display system of a head mounted display device, characterized in that: include: A facial image generation module, which obtains the user's facial data to generate a facial image; A display image generation module, which obtains internal display data of the head mounted display device to generate a corresponding external display image; A layer aliasing module, performing layer aliasing on the facial image and the external display image according to the depth and transparency, so as to generate an aliased image; as well as The external display screen is located on the outside of the head mounted display device and faces away from the user's face to display the aliased image outwardly.

2. The external display system according to claim 1, characterized in that: The facial image generation module is connected to at least one image sensor and at least one depth sensor, wherein: The image sensor is directed toward the user's face to collect image data of the user's face, The depth sensor faces the user's face to collect first depth data of the user's face. The facial image generation module generates a 3D facial image with depth information according to the image data of the user's face and the first depth data.

3. The external display system according to claim 2, characterized in that: The facial image generation module is also configured with a first interpolation table, and the facial image generation module is further configured as follows: interpolating the 3D facial image under a single viewing angle according to the first interpolation table to obtain 3D facial images under multiple viewing angles; as well as The 3D facial images under the multiple viewing angles are transmitted to the layer blending module to be layer blended with the external display images under the corresponding viewing angles respectively.

4. The external display system according to claim 3, characterized in that: The first interpolation table is divided in the form of a triangular mesh and stores texture coordinates, spatial coordinates and mesh information of each mesh vertex. The step of interpolating the 3D facial image under a single viewing angle according to the first interpolation table to obtain 3D facial images under multiple viewing angles includes: Determine the triangular mesh corresponding to each of the first pixel points according to the pixel indexes of the plurality of first pixel points in the 3D facial image under a single viewing angle, and the spatial coordinates and the mesh information of each mesh vertex stored in the first interpolation table; Performing triangular interpolation according to the texture coordinates of the mesh vertices of each of the triangular meshes and the relative positions of each of the first pixel points and the corresponding mesh vertices to respectively determine the texture coordinates of a plurality of second pixel points in each of the triangular meshes at corresponding viewing angles; and According to the texture coordinates of the plurality of first pixel points and the plurality of second pixel points at a plurality of viewing angles, 3D facial images at corresponding viewing angles are respectively determined.

5. The external display system according to claim 2, characterized in that: The display image generation module is further configured to: generate a virtual object image to be displayed and determine the second depth data corresponding thereto; and generate a 3D virtual object image with depth information according to the virtual object image and the second depth data. The layer aliasing module is further configured to perform layer aliasing on the 3D facial image, the external display image and the 3D virtual object image according to the depth and transparency to generate an aliased image.

6. The external display system according to claim 5, characterized in that: The display image generation module is also configured with a second interpolation table corresponding to the 3D virtual object image, and the display image generation module is further configured as follows: The 3D virtual object image under a single viewing angle is interpolated according to the second interpolation table to obtain 3D virtual object images under multiple viewing angles.

7. The external display system according to claim 6, characterized in that: The layer aliasing module is configured with a first lookup table corresponding to a first depth of the 3D facial image and a second lookup table corresponding to a second depth of the 3D virtual object image, and is configured as follows: According to the first lookup table and the second lookup table, depth-based layer aliasing is performed on the 3D facial image and the 3D virtual object image, and the display content and display position of the 3D facial image and / or the 3D virtual object image are adjusted to generate a 2D aliased image with 3D visual effects.

8. The external display system according to claim 7, characterized in that: The external display screen is provided with a micro lens array composed of a plurality of micro lens units, and is configured as follows: Acquire 3D facial images from multiple perspectives and 3D virtual object images from multiple perspectives; and Through the microlens array, according to the viewing angles corresponding to the 3D facial images and the 3D virtual object images, the light emitted by each pixel unit in the external display screen in the corresponding direction is controlled to display the corresponding aliased image in the direction of each viewing angle.

9. The external display system according to claim 8, characterized in that: The microlens unit is columnar, the long side direction of which is different from the column direction of the pixel unit facing the external display screen, and each of the microlens units corresponds to a non-integer number of pixel units.

10. The external display system according to claim 8, characterized in that: The external display system further includes a feedback unit for sensing a viewer position of at least one external viewer and adjusting a display mode of the aliased image according to the viewer position of the at least one external viewer.

11. The external display system according to claim 5, characterized in that: The external display image includes a mirrored 2D user interface image, and the step of performing layer aliasing on the 3D facial image, the external display image, and the 3D virtual object image according to the depth and transparency to generate an aliased image includes: Acquire the 3D virtual object image and its corresponding second depth and second transparency; Acquire the mirrored 2D user interface image and its corresponding third transparency, and the third depth corresponding to the external display screen; and According to the first depth, the second depth, the second transparency, the third depth and the third transparency corresponding to the facial image, layer aliasing is performed on the facial image, the virtual object image and the mirrored user interface image to generate the aliased image.

12. The external display system according to claim 11, characterized in that: The layer aliasing module is also connected to the ambient light sensor and is further configured as follows: Determine a second brightness of a second layer where the 3D virtual object image is located; Determine a third brightness of a third layer where the mirrored 2D user interface image is located; Acquiring an ambient light signal via the ambient light sensor; Determine, according to the ambient light signal, a first brightness of a first layer where the facial image is located, adjust the second brightness to determine a second corrected brightness, and adjust the third brightness to determine a third corrected brightness; as well as According to the first brightness, the second corrected brightness and the third corrected brightness, layer aliasing is performed on the facial image, the 3D virtual object image and the mirrored 2D user interface image to generate the aliased image.

13. The external display system according to claim 12, characterized in that: The layer aliasing module is further configured to: Determining a first occlusion attenuation coefficient of the third layer to the first layer according to a third transparency of the third layer; Determine, according to the second transparency of the second layer, a second occlusion attenuation coefficient of the second layer to the first layer, and a third occlusion attenuation coefficient of the second layer to the third layer; Determine a first adjustment value of the first layer, a second adjustment value of the second layer, and a third adjustment value of the third layer according to the ambient light signal, the first occlusion attenuation coefficient, the second occlusion attenuation coefficient, and the third occlusion attenuation coefficient; In response to generating the aliased image, determining whether a maximum brightness of the aliased image is greater than a maximum display brightness of the external display screen; as well as In response to a judgment result that the maximum brightness of the aliased image is greater than the maximum display brightness of the external display screen, a tone mapping value is determined according to a ratio of the maximum brightness of the aliased image to the maximum display brightness of the external display screen, and tone mapping is performed on the RGB grayscale value of each pixel in the aliased image according to the tone mapping value, the first adjustment value, the second adjustment value, and the third adjustment value, so that the maximum brightness is less than or equal to the maximum display brightness of the external display screen.

14. A head mounted display device, characterized in that: include: An internal display system, comprising an internal display screen, for displaying a virtual reality image generated by the display system to a first user wearing the head mounted display device; as well as The external display system according to any one of claims 1 to 13, comprising an external display screen for displaying a mixed image of the facial image of the first user and the internal display image displayed by the internal display screen to a second user facing the first user.

15. A method for controlling a head mounted display device, characterized in that: The following steps are involved: Obtaining user's facial data to generate a facial image; Acquire internal display data of the head mounted display device to generate a corresponding external display image; Performing layer aliasing on the facial image and the externally displayed image according to the depth and transparency to generate an aliased image; as well as The aliased image is displayed externally via an external display screen located on the outside of the head mounted display device and facing away from the user's face.

16. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by the processor, the control method of the head mounted display device as claimed in claim 15 is implemented.

Images