Head-mounted display and three-dimensional image generating method

Abstract

To provide a head-mounted display capable of generating a face image showing a user's real-time facial expression with little sense of incongruity.SOLUTION: A housing 10 of a head-mounted display 100 forms a shielded area by covering a partial area of a face including both eyes of a user 70. A display unit 11 is provided inside the housing 10 and displays an image that the user 70 sees with both eyes. A visible light sensor 13 detects the color of visible light in an environment in which the user 70 is located. A shielded area partial image generating unit 24 generates a three-dimensional partial image obtained by imaging a partial area of the face in the shielded area in a state where a color image which is detected by the visible sensor 13 and shows a color corresponding to the color of visible light in the environment in which the user 70 is located is displayed on the display unit 11.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a head mounted display and a three-dimensional image generating method. [Background technology]

[0002] A plurality of users may wear head-mounted displays and view virtual space images. In this case, three-dimensional images of the users wearing the head-mounted displays taken by a plurality of cameras may be used as avatars existing in the virtual space images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 3464754 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, if a three-dimensional image of a user wearing a head mounted display is taken as an avatar, the avatar's face will be partially covered by the head mounted display. Patent Document 1 describes the following: A face image of a user not wearing a head mounted display is taken in advance. A region hidden by the head mounted display is cut out from the face image of the user not wearing the display. The head mounted display region of the face image wearing the head mounted display is replaced with an image of the cut out region.

[0005] By adopting the configuration described in Patent Document 1, it is possible to create an avatar in a state where a part of the face is not covered by the head-mounted display. However, since it is not possible to display the state of the user's eyes in real time, other users cannot read the facial expression from the state of their eyes. Furthermore, if the environment when a user wearing a head-mounted display is photographed is different from the environment when a user not wearing the display is photographed, the composite image obtained by combining the two may be an unnatural image.

[0006] An object of the present invention is to provide a head mounted display and a three-dimensional image generating method capable of generating a facial image showing a user's real-time facial expression with less sense of discomfort. [Means for solving the problem]

[0007] The present invention provides a head mounted display comprising a housing that forms a shielded area by covering a partial area of ​​a user's face including both eyes, a display unit provided within the housing that displays an image the user sees with both eyes, a visible light sensor that detects the color of visible light in the environment in which the user is located, and a shielded area partial image generation unit that generates a three-dimensional partial image of the partial area of ​​the face in the shielded area while a color image indicating a color corresponding to the color of visible light in the environment in which the user is located detected by the visible light sensor is displayed on the display unit.

[0008] The present invention provides a three-dimensional image generation method, which generates a three-dimensional image of a user by photographing the user wearing a head-mounted display with multiple cameras, detects the color of visible light in the environment in which the user is located with a visible light sensor, displays a color image on a display unit of the head-mounted display, indicating the color corresponding to the color of visible light in the environment in which the user is located detected by the visible light sensor, generates a three-dimensional partial image by photographing a partial area of ​​the user's face including both of their eyes in an obstructed area covered by the head-mounted display while the color image is displayed on the display unit, removes the obstructed area from the three-dimensional image, and generates a composite image by combining the three-dimensional image from which the obstructed area has been removed with the partial image. Effect of the Invention

[0009] According to the head mounted display and three-dimensional image generating method of the present invention, a facial image showing the user's real-time facial expression can be generated with less sense of discomfort. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a head mounted display according to an embodiment. [Diagram 2] FIG. 2 is a conceptual diagram showing the operation of the head mounted display according to an embodiment and the processing executed by the three-dimensional image generating method according to an embodiment. [Diagram 3] FIG. 3 is a flowchart showing the operation of the head mounted display according to an embodiment and a three-dimensional image generating method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a head mounted display and a three-dimensional image generating method according to an embodiment will be described with reference to the accompanying drawings. In Fig. 1, a user 70 wears a housing 10 of a head mounted display 100 on his face. The housing 10 forms a shielded area by covering a partial area of ​​the face of the user 70 including both eyes. A display unit 11 that displays an image viewed by the user 70 with both eyes is provided inside the housing 10. The display unit 11 can be configured as a flat panel such as a liquid crystal panel or an organic EL panel.

[0012] The user 70 is located in an environment such as a room illuminated by a light 80, and the cameras 90a and 90b photograph the user 70. The light 80 includes sunlight. Although two cameras 90a and 90b are illustrated in FIG. 1, the number of cameras may be three or more. As described later, in order to generate a three-dimensional image showing the user 70 in a stereoscopic manner, it is preferable to use a larger number of cameras. Any camera including the cameras 90a and 90b and a camera not illustrated will be referred to as a camera 90. The multiple cameras 90 are arranged to surround the user 70.

[0013] A sensor 12 is provided inside the housing 10 for capturing an image of a partial area of ​​the face of the user 70 in the blocked area. A visible light sensor 13 is attached to the outside of the housing 10 for detecting the color of visible light (illumination light) in the environment in which the user 70 is located. A value indicating the color of the visible light detected by the visible light sensor 13 is supplied to an illumination equivalent image generating unit 42 in the image processing unit 40. The value indicating the color of the visible light may be, for example, a value indicating a color temperature.

[0014] The communication unit 30 receives a virtual space image provided from the virtual space image providing server 60 via the network 50. Typically, the network 50 is the Internet. The virtual space image processing unit 41 in the image processing unit 40 processes the virtual space image received by the communication unit 30 and provides it to the display unit 11 via the image switching unit 43. The user 70 views the virtual space image displayed on the display unit 11. Note that sound-related processing is omitted from FIG. 1. The virtual space image provided by the virtual space image providing server 60 is a three-dimensional viewing image seen by the user 70.

[0015] The illumination-corresponding image generating unit 42 generates an illumination-corresponding image, which is a color image showing a color corresponding to the color of visible light in the environment in which the user 70 is located, detected by the visible light sensor 13. The image switching unit 43 periodically supplies the illumination-corresponding image to the display unit 11 in place of the virtual space image, for example, at approximately 1 / 60 seconds. The image switching unit 43 may supply the illumination-corresponding image to the display unit 11 at predetermined time intervals.

[0016] The sensor 12 may be composed of a pair of small cameras. Each small camera includes an RGB sensor such as a CMOS sensor. The sensor 12 may be composed of a distance sensor for measuring the three-dimensional shape of a partial area of ​​the face of the user 70 in the blocked area, and one RGB sensor. The sensor 12 captures an image of the partial area of ​​the face of the user 70 in the blocked area while the image switching unit 43 supplies an image equivalent to the illumination to the display unit 11 and displays the image equivalent to the illumination on the display unit 11.

[0017] The imaging data (or distance data and imaging data) output from the sensor 12 is supplied to a masked area partial image generating unit 24 in the transmission image generating unit 20. The masked area partial image generating unit 24 generates a three-dimensional partial image of a partial area of ​​the face of the user 70 in the masked area.

[0018] The transmission image generating unit 20 is provided to generate a three-dimensional image of the user 70 based on images of the user 70 captured by a plurality of cameras 90, and transmit the image to the virtual space image providing server 60. The three-dimensional image generating unit 21 generates a three-dimensional image 70a of the user 70, as shown in Fig. 2(a), based on the images of the user 70 captured by the plurality of cameras 90. The three-dimensional image 70a of the user 70 is an image showing the entire body of the user 70, although only the upper body is shown in Fig. 2.

[0019] The three-dimensional image generator 21 can generate a three-dimensional image 70a using volumetrics technology. The three-dimensional image 70a shows a user 70 with the housing 10 covering a partial area of ​​the user's 70 face, including both eyes.

[0020] The three-dimensional image generating unit 21 supplies the three-dimensional image 70a to the masked area image deleting unit 22. The masked area image deleting unit 22 deletes the masked area covered by the housing 10 from the three-dimensional image 70a to generate a three-dimensional image 70b from which the masked area has been deleted, as shown in (b) of FIG. 2. The masked area image deleting unit 22 supplies the three-dimensional image 70b to the image synthesis unit 23. As described above, the masked area partial image generating unit 24 generates a three-dimensional partial image 70c of a partial area of ​​the face of the user 70 in the masked area, as shown in (c) of FIG. 2. The masked area partial image generating unit 24 supplies the partial image 70c to the image synthesis unit 23.

[0021] The image synthesis unit 23 synthesizes the three-dimensional image 70b and the partial image 70c to generate a three-dimensional image 70d (synthetic image) of the user 70 including both eyes of the user 70 with the housing 10 removed, as shown in (d) of FIG. 2. The image synthesis unit 23 supplies the three-dimensional image 70d to the communication unit 30. The three-dimensional image 70a shown in (a) of FIG. 2 and the partial image 70c shown in (c) of FIG. 2 are photographed and formed in the same environment under substantially the same illumination light. Therefore, by synthesizing the two, it is possible to generate the three-dimensional image 70d shown in (d) of FIG. 2, which has little sense of incongruity.

[0022] 1, the image processing unit 40 supplies the following information to the transmission image generating unit 20. The illumination-corresponding image generating unit 42 may not be able to accurately reproduce an illumination-corresponding image corresponding to the color of visible light in the environment in which the user 70 is located. In this case, the image processing unit 40 supplies a value indicating the color of the visible light detected by the visible light sensor 13 and information on image processing in the virtual space image processing unit 41 to the transmission image generating unit 20. The masked area partial image generating unit 24 may generate a three-dimensional partial image 70c based on the input value indicating the color of the visible light and the image processing information.

[0023] The communication unit 30 supplies the three-dimensional image 70d to the virtual space image providing server 60 via the network 50. The virtual space image providing server 60 is also supplied with three-dimensional images, similarly with the housings removed and including both of the users' eyes, transmitted from head-mounted displays worn by other users. The virtual space image providing server 60 generates a virtual space image including avatars based on the three-dimensional image 70d of the user 70 and the three-dimensional images of the other users.

[0024] Therefore, since the avatar in the virtual space image transmitted from virtual space image providing server 60 and displayed on display unit 11 includes both eyes in a real-time state, it is possible to read the facial expression from the state of the eyes of other avatars (other users), making it easier for avatars (other users) to communicate with each other. Note that even if display unit 11 periodically displays an image equivalent to illumination instead of a virtual space image for only 1 / 60 seconds, it is possible to make both eyes of the avatar substantially in a real-time state.

[0025] The operation of the head mounted display 100 and the three-dimensional image generating method will be described with reference to the flowchart shown in Fig. 3. When processing is started by turning on the power of the head mounted display 100, the transmission image generating unit 20 acquires captured images generated by the multiple cameras 90 capturing images of the user 70 under illumination light in step S1. The three-dimensional image generating unit 21 generates a three-dimensional image 70a of the user 70 in step S2. The masked area image deleting unit 22 deletes the masked area from the three-dimensional image 70a in step S3, and the processing proceeds to step S4.

[0026] In parallel with step S1, the visible light sensor 13 detects the color of the illumination light in step S6. Also, in parallel with step S1, the communication unit 30 receives the virtual space image transmitted from the virtual space image providing server 60 in step S10, and the process proceeds to step S7.

[0027] In step S7, image switching unit 43 determines whether it is time to switch from the virtual space image to the illumination equivalent image. If it is not time to switch from the virtual space image to the illumination equivalent image (NO), image switching unit 43 supplies the virtual space image to display unit 11, and display unit 11 displays the virtual space image in step S11. After step S11, the process proceeds to step S12.

[0028] If it is time to switch from the virtual space image to the illumination equivalent image in step S7 (YES), the image switching unit 43 supplies the illumination equivalent image to the display unit 11 instead of the virtual space image, and the display unit 11 displays the illumination equivalent image in step S8. In step S9, the masked area partial image generating unit 24 generates a partial image 70c of the face of the user 70 in the masked area. After step S9, the process proceeds to step S4.

[0029] In step S4, the image synthesis unit 23 synthesizes the three-dimensional image 70b from which the obstructed area has been deleted with the partial image 70c. In step S5, the communication unit 30 transmits the synthesized image (three-dimensional image 70d) to the virtual space image providing server 60. After step S5, the process proceeds to step S12.

[0030] In step S12, the head mounted display 100 determines whether or not to end the operation by turning off the power, etc. If the operation is not to be ended (NO), the head mounted display 100 returns the process to steps S1, S6, and S10. If the operation is to be ended (YES), the head mounted display 100 ends the process.

[0031] 1, the head mounted display 100 is configured to include a transmission image generating unit 20, a communication unit 30, and an image processing unit 40. An image transmitting / receiving device including the transmission image generating unit 20, the communication unit 30, and the image processing unit 40 may be an external configuration connected to the head mounted display 100.

[0032] The transmission image generating unit 20 and the image processing unit 40 can be configured with an image processing circuit or a microprocessor. The transmission image generating unit 20 or the image processing unit 40 may be configured with a combination of hardware and software. The software includes a computer program that causes the microprocessor to execute at least a part of the processes shown in FIG. 3.

[0033] The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]

[0034] 10. Chassis 11 Display section 12 Sensors 13 Visible Light Sensor 20 Transmission image generation unit 21 Three-dimensional image generation unit 22 Occluded area image removal unit 23 Image synthesis unit 24 Occluded area partial image generation unit 30 Communications Department 40 Image processing section 41 Virtual space image processing section 42 Lighting equivalent image generator 43 Image switching section 50 Network 60 Virtual space image providing server 70 users 80 Lighting 90a, 90b Camera 100 Head Mounted Display

Claims

1. A housing that forms a shielding area by covering a partial area of ​​a user's face including both eyes; a display unit provided in the housing and displaying an image viewed by the user with both eyes; a visible light sensor that detects the color of visible light in an environment in which the user is located; a masked area partial image generating unit that generates a three-dimensional partial image by capturing the partial area of ​​the face in the masked area while displaying a color image that indicates a color corresponding to a color of visible light in the environment where the user is located, the color image being detected by the visible light sensor, on the display unit; A head mounted display comprising:

2. An image switching unit that supplies the color image to the display unit at predetermined intervals while a three-dimensional viewing image viewed by the user is being supplied to the display unit, The masked area partial image generating unit generates the partial image when the image switching unit supplies the color image to the display unit. The head mounted display according to claim 1 .

3. a three-dimensional image generating unit that generates a three-dimensional image of the user based on captured images of the user wearing the head mounted display captured by a plurality of cameras; a masked area image deletion unit that deletes the masked area covered by the housing from the three-dimensional image; an image synthesis unit that synthesizes the three-dimensional image from which the occluded region has been deleted and the partial image to generate a synthetic image; The head mounted display according to claim 1 or 2, further comprising:

4. generating a three-dimensional image of a user wearing a head-mounted display by photographing the user with a plurality of cameras; Detecting the color of visible light in an environment where the user is located with a visible light sensor; displaying a color image showing colors corresponding to the colors of visible light in the environment where the user is located, detected by the visible light sensor, on a display unit of the head mounted display; generating a three-dimensional partial image by capturing an image of a partial area of ​​the face of the user, including both eyes, in a masked area covered by the head-mounted display while the color image is displayed on the display unit; removing the occluded region from the three-dimensional image; The 3D image from which the occluded region has been removed and the partial image are combined to generate a combined image. Three-dimensional image generation method.

Images