Head-mounted display device, control method thereof, and non-transitory computer-readable storage medium

Abstract

A head-mounted display device includes a camera, a processor, and a display. The camera is configured to capture a specific image of a specific object in an environment. The processor is coupled to the camera and configured to perform operations including obtaining a plurality of feature points of the specific image, each of the plurality of feature points including a world coordinate, generating a viewpoint coordinate based on the world coordinate of each of the plurality of feature points, and generating a projection coordinate of the environment based on the viewpoint coordinate of each of the plurality of feature points. The display is coupled to the processor and configured to display a virtual image including a converted specific image. The control techniques provided by the present disclosure allow a user to see a plurality of real objects in real size by displaying the virtual image.

Description

Technical Field

[0001] This disclosure relates to a head-mounted display device (HMD device), its control method, and a non-transitory computer-readable storage medium. Specifically, the present invention relates to a head-mounted display device for displaying real-world objects in a virtual reality (VR) environment, its control method, and a non-transitory computer-readable storage medium. Background Technology

[0002] In the virtual reality industry, one of the most challenging issues is how to interact with the real-world environment while wearing a virtual reality device. Pass-through technology is a well-known solution, allowing users to see real-world objects without removing their headsets. Almost all virtual reality manufacturers on the market today support this technology. However, ensuring that real-world objects appear to have the same size (proportion) in the virtual environment as they do in the real world is one of the biggest challenges in developing pass-through technology.

[0003] The difficulty lies in the fact that depth information in the environment is unknown. To date, no high-performance algorithm has been proposed for 3D reconstruction using stereoscopic images. Among the proposed algorithms, some produce images with visible distortion artifacts. Furthermore, another drawback of existing perspective techniques is that users cannot select the view they want to see, and the resulting image occupies the entire field of view. This limits how users can interact with the real world in virtual reality. Summary of the Invention

[0004] This disclosure provides a head-mounted display device including a camera, a processor, and a display. The camera is used to capture a specific image of a specific object in an environment. The processor is coupled to the camera and is used to perform the following operations: acquiring a plurality of feature points of the specific image, each of the plurality of feature points including a world coordinate; generating a viewpoint coordinate based on the world coordinates of the plurality of feature points; and generating a projected coordinate of the environment based on the viewpoint coordinate. The display is coupled to the processor and is used to display a virtual image including a transformed specific image, wherein the transformed specific image is derived from the specific image based on the projected coordinates of the plurality of feature points.

[0005] This disclosure also provides a control method for a head-mounted display device, the head-mounted display device including a camera, a processor, and a display. The control method includes the steps of: acquiring, with the processor, a plurality of feature points of a specific image of a specific object in an environment; acquiring, with the processor, a plurality of world coordinates of the plurality of feature points; generating, with the processor, a plurality of viewpoint coordinates of the plurality of world coordinates of the plurality of feature points; and generating, with the processor, a plurality of projection coordinates of the plurality of viewpoint coordinates of the environment, such that the display shows a virtual image including a transformed specific image, wherein the transformed specific image is derived from the specific image based on the plurality of projection coordinates.

[0006] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon for executing the aforementioned control method, the control method comprising the steps of: acquiring, with a processor, a plurality of feature points of a specific image of a specific object in an environment; acquiring, with the processor, a plurality of world coordinates of the plurality of feature points; generating, with the processor, a plurality of viewpoint coordinates of the plurality of world coordinates; and generating, with the processor, a plurality of projection coordinates of the plurality of viewpoint coordinates of the environment, so that a display shows a virtual image containing a transformed specific image, wherein the transformed specific image is derived from the specific image based on the plurality of projection coordinates.

[0007] In one embodiment of the invention, each of the plurality of feature points contains a local coordinate of a local coordinate system of the particular object, wherein the world coordinates are generated based on the local coordinates.

[0008] In one embodiment of the invention, the projected coordinates are generated according to the following formula:

[0009] c = M projection ×M view ×M model ×v

[0010] Where v is the local coordinate, M model Convert the local coordinates to the world coordinates, M view Convert the world coordinates to the viewpoint coordinates, M projection Convert the viewpoint coordinates to the projected coordinates, and let c be the projected coordinates.

[0011] In one embodiment of the invention, the camera further includes a field of view parameter, and the processor is further configured to generate the projection coordinates based on the viewpoint coordinates and the field of view parameter.

[0012] In one embodiment of the invention, the processor is further configured to obtain the plurality of feature points based on a mesh module of the particular object.

[0013] In one embodiment of the invention, the processor further converts the world coordinates to the viewpoint coordinates using a viewpoint transformation array; and converts the viewpoint coordinates to the projected coordinates using a projection transformation array.

[0014] In one embodiment of the present invention, the projection coordinates are projection points in a display screen shown by the display, projected from the viewpoint coordinates to a user's coordinates.

[0015] In one embodiment of the invention, the world coordinates are generated according to a world coordinate system, and the viewpoint coordinates are generated according to a user coordinate system.

[0016] It should be understood that the foregoing general description and the following specific description are merely exemplary and explanatory, and are intended to provide further explanation of the claimed disclosure. Attached Figure Description

[0017] To make the above and other objects, features, advantages and embodiments of this disclosure more apparent and understandable, the accompanying drawings are described below:

[0018] Figure 1 This is a schematic diagram of the environment in which a user operates a head-mounted display device of a virtual reality system in some embodiments of this disclosure;

[0019] Figure 2 This is a schematic diagram of a head-mounted display device in some embodiments of this disclosure;

[0020] Figure 3 This is a flowchart of the control method in some embodiments of this disclosure;

[0021] Figure 4 This is a partial embodiment of the present disclosure. Figure 1 A schematic diagram of a specific image of the environment shown;

[0022] Figure 5 This is a partial embodiment of the present disclosure. Figure 1 A schematic diagram of the mesh module for a specific object shown;

[0023] Figure 6 This is a partial embodiment of the present disclosure. Figure 1 A schematic diagram of the specific object shown;

[0024] Figure 7 This is a schematic diagram illustrating the conversion of viewpoint coordinates to projected coordinates in some embodiments of this disclosure; and

[0025] Figure 8 This is a schematic diagram of a virtual image in some embodiments of this disclosure.

[0026] Symbol explanation:

[0027] E: Environment

[0028] S: Specific object

[0029] U: User

[0030] 100: Head-mounted display device

[0031] 110: Camera

[0032] 130: Processor

[0033] 150: Monitor

[0034] 170: Storage

[0035] 300: Control Method

[0036] S310~S370: Steps

[0037] SP: Specific Images

[0038] a1~a4: Feature points

[0039] A1~A4: Feature points

[0040] B: Origin

[0041] B1~B4: Viewpoint Coordinates

[0042] C1~C4: Projected coordinates

[0043] 800: Virtual Image

[0044] ST: Specific image after conversion Detailed Implementation

[0045] To make the description of this disclosure more detailed and complete, reference may be made to the accompanying drawings and the various embodiments described below, in which the same numbers represent the same or similar elements.

[0046] Please refer to Figure 1 This is a schematic diagram of the environment E of a head-mounted display device of a virtual reality system operated by a user U in some embodiments of this disclosure. Figure 1 As shown, user U wears head-mounted display device 100. Environment E is an environment in which user U operates head-mounted display device 100. Environment E includes a specific object S, which is a real object that user U wishes to interact with. In one embodiment, the system of the head-mounted display device may be a mixed reality (MR) system or an augmented reality (AR) system, and this disclosure is not limited thereto.

[0047] In some embodiments, such as Figure 1 As shown, the specific object S is a computer screen, but the embodiments disclosed herein are not limited to this. In some other embodiments, the specific object S may be a mouse, a controller, a mobile phone, or any other real object that the user U wishes to interact with.

[0048] In some embodiments, the user U sets a real object that they wish to interact with through the virtual reality system or the controller (not shown in the figure).

[0049] Please refer to Figure 2 . Figure 2 This is a schematic diagram of a head-mounted display device 100 in some embodiments of this disclosure. Figure 2 As shown, the head-mounted display device 100 includes a camera 110, a processor 130, and a display 150. The connection relationship is that the camera 110 is coupled to the processor 130, and the display 150 is coupled to the processor 130.

[0050] Please refer to Figure 3 . Figure 3 This is a flowchart of a control method 300 in some embodiments of this disclosure. The control method 300 can be... Figure 2 The control method 300, performed by the head-mounted display device 100 shown, includes steps S310 to S370.

[0051] In step S310, a processor acquires multiple feature points of a specific image of a specific object. In some embodiments, step S310 is performed by... Figure 2 The processor 130 shown is the one that executes the commands.

[0052] Please refer to this as well. Figure 4 . Figure 4 This is a partial embodiment of the present disclosure. Figure 1 This is a schematic diagram of a specific image of environment E. In some embodiments, by Figure 2 The camera 110 shown is Figure 1 A specific image SP of a specific object S is captured in the environment E shown.

[0053] like Figure 4 As shown, attributed to Figure 2 The camera 110 shown is Figure 1 The orientation and angle between specific objects S shown are such that the specific image SP appears as a trapezoid rather than a rectangle.

[0054] Obtained from camera 110 Figure 4 After the specific image SP shown, Figure 2 The processor 130 shown acquires the plurality of feature points of a specific image SP.

[0055] In some embodiments, Figure 2The head-mounted display device 100 shown further includes a memory 170. In terms of connectivity, the memory 170 is coupled to... Figure 2 The processor 130 is shown. In some embodiments, the memory 170 includes... Figure 1 The example shown is a mesh module of a specific object S.

[0056] Please refer to Figure 5 . Figure 5 This is a partial embodiment of the present disclosure. Figure 1 A schematic diagram of the mesh module 500 for a specific object S is shown. Figure 5 As shown, Figure 1 The grid module of the specific object S shown is a computer screen, and is divided into two triangles and four feature points A1 to A4.

[0057] In some embodiments, memory 170 stores multiple mesh modules for multiple specific objects. For example, the multiple specific objects may be a mouse, a controller, and a mobile phone, and memory 170 stores a mesh module corresponding to each of the multiple specific objects. In some embodiments, the mesh module includes multiple triangles that approximate the shape of the specific object.

[0058] In some embodiments, the plurality of mesh modules are pre-stored in memory 170. In some embodiments, the plurality of mesh modules are generated by... Figure 1 The settings shown are configured by user U.

[0059] Please refer to this again. Figure 4 .like Figure 4 As shown, Figure 2 The processor 130 shown is based on Figure 5 The grid module 500 shown obtains feature points a1 to a4 of a specific image SP.

[0060] Please refer to this again. Figure 3 In step S330, the processor obtains multiple world coordinates of the plurality of feature points. In some embodiments, step S330 is performed by... Figure 2 The processor 130 shown is the one that executes the commands.

[0061] In some embodiments, Figure 4 The feature points a1 to a4 shown each contain a local coordinate of a local coordinate system for that particular object.

[0062] In some embodiments, Figure 1 The specific object S shown includes a tracker (not shown in the figure) that transmits the plurality of local coordinates of feature points a1 to a4 to... Figure 2 The head-mounted display device 100 shown is shown.

[0063] Please refer to this as well. Figure 6 . Figure 6 This is a partial embodiment of the present disclosure. Figure 1 The diagram shows a specific object S. Taking an origin B in this local coordinate system as an example, origin B is usually the center point of the specific object S, and also the origin (0,0,0) of the local coordinate system. In the three-dimensional space formed by the X-axis, Y-axis, and Z-axis, feature points a1 to a4 each contain a local coordinate. These multiple local coordinates are derived from the relative position and distance between feature points a1 to a4 and the origin B. For example, the local coordinates of feature point a1 are (X1,Y1,Z1), the local coordinates of feature point a2 are (X2,Y2,Z2), the local coordinates of feature point a3 are (X3,Y3,Z3), and the local coordinates of feature point a4 are (X4,Y4,Z4).

[0064] In some embodiments, Figure 2 The processor 130 shown generates multiple world coordinates for feature points a1 to a4 based on the multiple local coordinates of feature points a1 to a4. These multiple world coordinates are generated according to a world coordinate system, also known as world coordinate space or global coordinate space. The multiple world coordinates of feature points a1 to a4 are used to indicate the relative position and distance between feature points a1 to a4 and the origin B within this world coordinate space (not shown in the figure).

[0065] In some embodiments, Figure 2 The processor 130 shown uses a model conversion array M model The local coordinates are transformed into the world coordinates. In this disclosure, the transformation array M between the local coordinates and the world coordinates is... model It can also be achieved using a variety of different conversion arrays.

[0066] Please refer to this again. Figure 3 In step S350, the processor generates multiple viewpoint coordinates of the multiple world coordinates. In some embodiments, the multiple viewpoint coordinates are generated according to a user coordinate system corresponding to a user viewpoint. In some embodiments, step S350 is performed by... Figure 2 The processor 130 shown is the one that executes the commands.

[0067] In some embodiments, the user coordinate system corresponds to the user's viewpoint, also known as a viewpoint space, and the user coordinate system transforms the world coordinates into... Figure 1 The results show the coordinates of multiple coordinates in front of the user's U-shaped field of view.

[0068] In some embodiments, Figure 2 The processor 130 shown uses a viewpoint array M viewThe world coordinates are transformed into the viewpoint coordinates. In this disclosure, the transformation array M between the world coordinates and the viewpoint coordinates... view It can also be achieved using a variety of different conversion arrays.

[0069] Please refer to this again. Figure 3 In step S370, the processor generates multiple projection coordinates of the multiple viewpoint coordinates to cause the display to show a virtual image containing a transformed specific image. In some embodiments, the transformed specific image is derived from the specific image based on the multiple projection coordinates.

[0070] In some embodiments, step S370 is performed by Figure 2 The processor 130 shown executes the commands. In some embodiments, Figure 2 The camera 110 shown includes a field of view parameter, and the projected coordinates are generated based on the plurality of viewpoint coordinates and the field of view parameter. In some embodiments, the processor 130 uses a projection transformation array M projection Convert the multiple viewpoint coordinates into the projected coordinates.

[0071] Please refer to Figure 7 and Figure 8 . Figure 7 This is a schematic diagram illustrating the conversion of the viewpoint coordinates to the projected coordinates in some embodiments of this disclosure. Figure 8 This is a schematic diagram of a virtual image 800 in some embodiments of this disclosure. Figure 7 As shown, this field-of-view parameter of camera 110 determines one of the camera 110's viewpoints. The virtual image 800 is... Figure 2 As shown on monitor 150. Figure 7 As shown, projection coordinates C1 to C4 represent the projections from viewpoint coordinates B1 to B4 onto a display screen shown on monitor 150. Figure 1 The diagram shows multiple projection points of a user coordinate for user U (which can also be the viewpoint coordinates of camera 110 since user U is wearing a head-mounted display device 100 containing camera 110). It should be noted that viewpoint coordinates B1 to B4 are multiple virtual coordinates that do not exist in the real world.

[0072] After projection, such as Figure 7 As shown, viewpoint coordinates B1 are converted to projected coordinates C1, viewpoint coordinates B2 are converted to projected coordinates C2, viewpoint coordinates B3 are converted to projected coordinates C3, and viewpoint coordinates B4 are converted to projected coordinates C4.

[0073] In some embodiments, the user coordinates of user U are the origin of the user coordinate system. In some embodiments, the virtual image 800 is displayed on... Figure 2The display screen shown is on monitor 150. For example... Figure 8 As shown, the transformed specific image ST is generated based on projection coordinates C1 to C4, and the transformed specific image ST is... Figure 4 The specific image SP shown is derived from this. The dashed area in the figure represents... Figure 2 The virtual image is displayed on the monitor 150 shown. That is, the monitor 150 displays a virtual image 800 containing a specific converted image ST.

[0074] As described above, the plurality of projected coordinates are generated according to the following equation:

[0075] [Formula 1]

[0076] c = M projection ×M view ×M model ×v

[0077] Where v is the local coordinate, M model Convert the local coordinates to the world coordinates, M view Convert the world coordinates to the viewpoint coordinates, M projection Convert the viewpoint coordinates to the projected coordinates, and let c be the projected coordinates.

[0078] It is important to note that Figure 2 The head-mounted display device 100 shown is merely an example, and the embodiments disclosed herein are not limited thereto. In some other embodiments, the head-mounted display device 100 includes two or more cameras, and each of the plurality of cameras generates its own plurality of projected coordinates in accordance with the methods described in the embodiments of this disclosure.

[0079] In some embodiments, in generating Figure 8 The virtual image shown is 800 before, Figure 2 The processor 130 shown executes a rendering algorithm to integrate the virtual image and the transformed specific image ST to generate the virtual image 800.

[0080] The control method 300 described above for controlling a head-mounted display device, including a camera, a processor, and a display, can be implemented by a computer program stored on a computer-readable storage medium. After a computer reads the computer-readable storage medium, the computer can execute the control method 300. The computer-readable storage medium can be a read-only memory, a flash memory, a disk, a hard disk, an optical disk, a USB flash drive, a network database, or other computer-readable storage media readily conceived by those skilled in the art and capable of achieving the same function. The control method 300 described above for controlling a head-mounted display device, including a camera, a processor, and a display, can also be implemented by the computer program in conjunction with at least one processor or chip.

[0081] In embodiments of this disclosure, a surface of the specific object can be simplified into a large number of planar polygons, and "polygon triangulation" is used to cut the specific object into a large number of triangles to obtain a mesh of the specific object. Through the multiple mesh modules, multiple feature points in multiple endpoints of the multiple triangles can be obtained, and the multiple local coordinates of the multiple endpoints of the multiple triangles are converted into multiple projected coordinates. The multiple projected coordinates constitute the shape of the transformed specific image. Then, the processor maps the texture of the specific object onto the mesh of the transformed specific image.

[0082] Embodiments of this disclosure provide a head-mounted display device, its control method, and a non-transitory computer-readable storage medium. Multiple feature points can be acquired through multiple grid modules, and the local coordinates of each feature point are converted into projected coordinates. The multiple projected coordinates constitute the shape of the converted specific image. In embodiments of this disclosure, the user can see multiple real objects that they wish to view, and these real objects are presented at their actual size.

[0083] In some embodiments, Figure 2 The processor 130 shown includes, but is not limited to, a single processor or a combination of multiple microprocessors, such as a central processing unit (CPU) or a graphics processing unit (GPU). The processor 130 is electrically coupled to memory 170 to access at least one instruction. This at least one instruction can be used to execute the control method described above.

[0084] In some embodiments, Figure 2The memory 170 shown may include a flash memory, a hard disk drive (HDD), a solid-state drive (SSD), a dynamic random access memory (DRAM), or a static random access memory (SRAM). In some embodiments, the memory 170 may be a non-transitory computer-readable storage medium storing at least one instruction relating to a control method. The processor 130 may access and execute the at least one instruction.

[0085] In some embodiments, Figure 2 The camera 110 shown may be an image capturing circuit or other unit or circuit with the same or similar function. In some embodiments, Figure 2 The display 150 shown may be a display circuit or other unit or circuit with the same or similar functions for displaying images.

[0086] Although several embodiments have been described in detail above as examples, the head-mounted display device, its control method, and the non-transitory computer-readable storage medium proposed in this disclosure can also be implemented using other systems, hard disks, floppy disks, storage media, or combinations thereof. Therefore, the scope of protection of this disclosure should not be limited to the specific implementations described in the embodiments of this disclosure, but should be determined by the scope defined in the appended claims.

[0087] It will be apparent to those skilled in the art to which this disclosure pertains that various modifications and variations can be made to the structure of this disclosure without departing from its scope or spirit. In view of the foregoing, the scope of protection of this disclosure also covers modifications and variations made within the scope of the appended claims.

Claims

1. A head-mounted display device, characterized in that, Include: A camera is used to capture a specific image of a specific object in an environment. A processor, coupled to the camera, is used to perform the following operations: The specific image is cut into multiple triangles to obtain a grid module of the specific object; Based on the multiple endpoints of multiple triangles in the mesh module, multiple feature points of the specific image are obtained, wherein each of the multiple feature points contains a world coordinate. A viewpoint coordinate is generated based on the world coordinates of each of the multiple feature points; as well as Based on the viewpoint coordinates of each of the plurality of feature points, a projected coordinate of the environment is generated; and A display, coupled to the processor, is used to display a virtual image comprising a transformed specific image, wherein the transformed specific image is derived from the specific image based on the projected coordinates of the respective plurality of feature points. Each of the plurality of feature points contains a local coordinate of a local coordinate system of the particular object, the world coordinates are generated based on the plurality of local coordinates, the particular object includes a tracker, and the head-mounted display device receives the plurality of local coordinates of the plurality of feature points from the tracker; Wherein, an origin of the local coordinate system is a center point of the specific object. The origin is set according to the multiple local coordinates of the multiple feature points, and the origin is a point (0,0,0) in the local coordinate system, and the origin is not any of the multiple endpoints of the multiple triangles.

2. The head-mounted display device as claimed in claim 1, characterized in that, The projected coordinates are generated according to the following formula: in For this local coordinate, Convert the local coordinates to the world coordinates. Convert the world coordinates to the viewpoint coordinates. Convert the viewpoint coordinates to the projected coordinates, and These are the projected coordinates.

3. The head-mounted display device as claimed in claim 1, characterized in that, The camera further includes a field of view parameter, and the processor further generates the projected coordinates based on the viewpoint coordinates and the field of view parameter.

4. The head-mounted display device as claimed in claim 1, characterized in that, The processor is further used for: Transform the world coordinates to the viewpoint coordinates using a viewpoint transformation array; and The viewpoint coordinates are converted to the projected coordinates using a projection transformation array.

5. The head-mounted display device as claimed in claim 1, characterized in that, The projection coordinates are a projection point in a display screen that is projected from the viewpoint coordinates to a user's coordinates.

6. The head-mounted display device as claimed in claim 1, characterized in that, The world coordinates are generated according to a world coordinate system, and the viewpoint coordinates are generated according to a user coordinate system.

7. A control method, characterized in that, For a head-mounted display device, the head-mounted display device including a camera, a processor and a display, the control method includes the following steps: The processor cuts a specific image of a specific object in an environment into multiple triangles to obtain a mesh module of the specific object. Using this processor, multiple feature points in the specific image are obtained based on multiple endpoints of multiple triangles in the mesh module; The processor obtains multiple world coordinates of the multiple feature points; The processor generates multiple viewpoint coordinates for the multiple world coordinates; as well as The processor generates multiple projected coordinates of the multiple viewpoint coordinates of the environment, so that the display shows a virtual image containing a transformed specific image, wherein the transformed specific image is derived from the specific image based on the multiple projected coordinates. Wherein, each of the plurality of feature points contains a local coordinate in a local coordinate system of the specific object, the specific object includes a tracker, and the control method further includes: The world coordinates are generated based on the multiple local coordinates; and Receive the multiple local coordinates of the multiple feature points from the tracker; Wherein, an origin of the local coordinate system is a center point of the specific object. The origin is set according to the multiple local coordinates of the multiple feature points, and the origin is a point (0,0,0) in the local coordinate system, and the origin is not any of the multiple endpoints of the multiple triangles.

8. A non-transitory computer-readable storage medium, characterized in that, It has a computer program stored thereon, which executes a control method comprising the following steps: The processor cuts a specific image of a specific object in an environment into multiple triangles to obtain a mesh module of the specific object. Using this processor, multiple feature points of the specific image are obtained based on multiple endpoints of multiple triangles in the mesh module; The processor obtains multiple world coordinates of the multiple feature points; The processor generates multiple viewpoint coordinates for the multiple world coordinates; as well as The processor generates multiple projected coordinates of the multiple viewpoint coordinates of the environment, so that a display shows a virtual image containing a transformed specific image, wherein the transformed specific image is derived from the specific image based on the multiple projected coordinates. Wherein, each of the plurality of feature points contains a local coordinate in a local coordinate system of the specific object, the specific object includes a tracker, and the control method further includes: The world coordinates are generated based on the multiple local coordinates; and Receive the multiple local coordinates of the multiple feature points from the tracker; Wherein, an origin of the local coordinate system is a center point of the specific object. The origin is set according to the multiple local coordinates of the multiple feature points, and the origin is a point (0,0,0) in the local coordinate system, and the origin is not any of the multiple endpoints of the multiple triangles.

Images