Display device, display method, and program

Abstract

To allow a real space image to be optimally displayed on at least a portion of a virtual space image.SOLUTION: Provided is a display device comprising a display unit for displaying a virtual space image, a detection unit for detecting a certain object in the real space, and a transmission control unit for changing the transmission state of an area of the display unit designed to allow the object detected by the detection unit be viewed through the display unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] In recent years, virtual reality (VR) spaces are increasingly being shared by multiple users for use in meetings, games, shopping, and other activities. Users wear a head-mounted display device (HMD) on their heads to participate in meetings and other activities using VR spaces. An example of such a display device is described in Patent Document 1 below. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-106587 Summary of the Invention [Problem to be solved by the invention]

[0004] When a user participates in a virtual conference, various materials are displayed on a display screen installed in the virtual space. However, if the resolution of the display device is low, the clarity of the materials displayed on the display screen will be low, making it difficult for the user to read the materials. Therefore, users often desire to view the materials on the display of their own personal computers.

[0005] The present invention has been made in view of the above, and has as its object to optimally display an image in real space on at least a part of an image in virtual space. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the display device of the present invention comprises a display unit that displays an image of a virtual space, a detection unit that detects an object in real space, and a transmission control unit that changes the transmission state of an area of ​​the display unit where the object detected by the detection unit can be seen through the display unit.

[0007] The display method of the present invention includes the steps of displaying an image of a virtual space on a display unit, detecting an object in real space, and changing the transparency state of an area of ​​the display unit where the detected object can be seen through the display unit.

[0008] The program of the present invention causes a computer operating as a display device to perform the following steps: displaying an image of a virtual space on a display unit; detecting an object in real space; and changing the transparency state of an area of ​​the display unit where the detected object can be seen through the display unit. [Effects of the Invention]

[0009] According to the present invention, it is possible to optimally display an image in real space on at least a part of an image in virtual space. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a specific configuration of a display device according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of the display device according to this embodiment. [Figure 3] FIG. 3 is a flowchart showing the display method according to this embodiment. [Figure 4] FIG. 4 is a schematic diagram showing an image of a virtual space. [Figure 5] FIG. 5 is a schematic diagram showing an image in real space. [Figure 6] FIG. 6 is a schematic diagram of an image in which an image in real space is superimposed on an image in virtual space. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a display device, a display method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0012] <Specific configuration of the display device> 1 is a schematic diagram showing a specific configuration of a display device according to this embodiment. In this embodiment, the display device is described as being applied to a head-mounted display device, but the display device is not limited to this configuration.

[0013] As shown in FIG. 1, the display device 10 includes a display unit 11 and a light-shielding unit 12.

[0014] The display unit 11 is supported by the exterior casing 20. The display unit 11 has a display panel 21, a half mirror 22, and a combiner mirror 23. The display panel 21 is arranged horizontally on the upper part of the exterior casing 20. The display panel 21 has a flat shape, and various display panels such as a liquid crystal panel, an organic EL panel, and a plasma panel are applicable. The display panel 21 has a display surface 21a on the bottom surface which can display an image of the virtual space image A. The display surface 21a can irradiate display light La downward, i.e., toward the inside of the exterior casing 20.

[0015] The half mirror 22 is disposed below the display panel 21 inside the exterior casing 20. The half mirror 22 is disposed at a predetermined angle with respect to the display panel 21. The half mirror 22 is provided with a reflective coating 22a on its upper surface and an anti-reflection coating 22b on its lower surface. The half mirror 22 reflects light from above and transmits light from the front. That is, the half mirror 22 reflects the display light La emitted from the display panel 21 toward the combiner mirror 23. The half mirror 22 also transmits the reflected light Lb reflected by the combiner mirror 23 backward.

[0016] The combiner mirror 23 is disposed inside the exterior casing 20 and in front of the half mirror 22. The combiner mirror 23 is disposed vertically in front of the exterior casing 20. The combiner mirror 23 has a concave shape. A reflective coating 23a is provided on the inner surface of the combiner mirror 23. The combiner mirror 23 reflects the display light La that is emitted from the display panel 21 and reflected by the half mirror 22, and irradiates the display light La toward the half mirror 22 as reflected light Lb.

[0017] The display unit 11 reflects the display light La emitted from the display panel 21 forward by the half mirror 22, reflects the display light La backward by the combiner mirror 23, and transmits the reflected light Lb through the half mirror 22 to be guided to the user's eyeball. Therefore, the user visually recognizes the virtual space image A displayed by the display unit 11 as being located in front of the display device 10.

[0018] The combiner mirror 23 also transmits and takes in real image light Lc that constitutes the real space image B from the outside toward the half mirror 22. The real space image B is an image that includes an object, which will be described later. The display unit 11 allows the real image light Lc that constitutes the real space image B to transmit through the combiner mirror 23 and the half mirror 22 and reach the left and right eyeballs of the user. Therefore, the user directly views the images of the objects that exist in the real space image B.

[0019] At this time, the reflected light Lb (display light La) that generates the virtual space image A and the real image light Lc that generates the real space image B reach the user's eyeball. As a result, the user sees a composite image in which the real space image B is superimposed on the virtual space image A.

[0020] The shading unit 12 has a shading panel 25. The shading panel 25 is supported vertically on the front part of the exterior casing 20. The shading panel 25 is arranged at a predetermined interval outside the combiner mirror 23. The shading panel 25 has a planar shape, and various display panels such as a liquid crystal panel, an organic EL panel, and a plasma panel can be used. The shading panel 25 has pixels arranged in a matrix, and each pixel can be adjusted and controlled to change from transparent to opaque. The transmittance of the shading panel 25 can be adjusted and controlled for the entire area or a specified part.

[0021] The shading panel 25 is configured, for example, by having transparent pixel electrodes arranged in an array on one surface and a transparent counter electrode arranged on the other surface, and a voltage corresponding to a shading control signal is applied to each electrode. The voltage of the shading control signal differs between pixels corresponding to the shading region and pixels corresponding to the transmitting region. Based on the shading control signal, each pixel of the shading panel 25 blocks or transmits real image light Lc that generates the real space image B. The transmittance of the shading panel 25 can be adjusted between 0% and 100% according to the shading control signal.

[0022] When the transmittance of shading panel 25 is 0%, shading panel 25 blocks external real image light Lc that generates real space image B, and the user sees only real space image B. On the other hand, when the transmittance of shading panel 25 is 100%, shading panel 25 transmits all of the external real image light Lc that generates real space image B, and the user sees a composite image in which real space image B is superimposed on virtual space image A. Furthermore, when the transmittance of shading panel 25 is adjusted between 0% and 100%, shading panel 25 adjusts the transmittance of external real image light Lc that generates real space image B, and the user sees a composite image in which real space image B, adjusted to a predetermined transmittance, is superimposed on virtual space image A.

[0023] [Display device processing configuration] FIG. 2 is a block diagram showing the configuration of the display device according to this embodiment.

[0024] As shown in FIG. 2, the display device 10 transmits and receives various information to and from the virtual space construction system 100. The virtual space construction system 100 generates information about a VR space. The virtual space construction system 100 is, for example, a server. The virtual space construction system 100 also generates information about the VR space based on three-dimensional models of avatars generated by the personal computers of multiple users. The virtual space construction system 100 outputs the generated information about the VR space to the display device 10.

[0025] The display device 10 displays an image of the VR space as seen by the user, based on the information about the VR space acquired from the virtual space construction system 100.

[0026] In addition to the display unit 11 and shading unit 12 described above, the display device 10 includes a virtual space image acquisition unit 31, an image processing unit 32, a camera 41, a real space image processing unit 42, a device detection unit 43, a transmission image generation unit 44, a shading control unit 45, and an image determination unit 46. The display device 10 also includes a user image generation unit 51, a signal synthesis unit 52, and a signal transmission unit 53.

[0027] Here, the image processing unit 32, real space image processing unit 42, device detection unit 43, transmission image generation unit, shading control unit 45, image determination unit 46, user image generation unit 51, and signal synthesis unit 52 are configured by, for example, at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a RAM (Random Access Memory), and a ROM (Read Only Memory).

[0028] The virtual space image acquisition unit 31 is connected to the virtual space construction system 100 and also to the image processing unit 32. The virtual space image acquisition unit 31 is, for example, a communication module, and is connected to the virtual space construction system 100 and the image processing unit 32 via a network (for example, the Internet). The virtual space image acquisition unit 31 acquires information about the VR space from the virtual space construction system 100. The virtual space image acquisition unit 31 outputs the information about the VR space acquired from the virtual space construction system 100 to the image processing unit 32.

[0029] The image processing unit 32 generates display image data based on the information of the VR space and outputs the display image data as a display signal to the display unit 11. The display unit 11 displays the virtual space image A based on the display signal input from the image processing unit 32.

[0030] The camera 41 is a camera for performing SLAM (Simultaneous Localization and Mapping). The camera 41 captures and acquires RGB images, and outputs the captured image data. The camera 41 is attached to, for example, an HMD serving as the display device 10. The camera 41 may be, for example, a monocular camera (wide-angle camera, fisheye camera, omnidirectional camera), a compound eye camera (stereo camera, multi-camera), an RGB-D camera (depth camera or ToF camera), or the like.

[0031] The camera 41 is connected to a real-space image processing unit 42. The real-space image processing unit 42 acquires captured image data captured by the camera 41. The real-space image processing unit 42 executes visual slant processing using the captured image data, and performs mapping of the real space and estimation of the user's position and head tracking.

[0032] The camera 41 is also connected to a device detection unit 43. The device detection unit 43 acquires captured image data captured by the camera 41. The device detection unit 43 uses the captured image data to detect various devices captured in the image of real space. Here, the devices include a personal computer display, keyboard, mouse, etc. The devices to be detected are set in advance, and training data is stored. The device detection unit 43 identifies the devices using the training data, for example, by machine learning.

[0033] The real space image processing unit 42 and the device detection unit 43 are connected to the transparent image generation unit 44. The real space mapping result and the self-position / head tracking result processed by the real space image processing unit 42, and the device detection result detected by the device detection unit 43 are input to the transparent image generation unit 44. The transparent image generation unit 44 generates transparent image data based on the real space mapping result, the self-position / head tracking result, and the device detection result. The transparent image data is image data that is displayed by transmitting part or all of the display unit 11.

[0034] The transmission image generation unit 44 is connected to the shading control unit 45. The shading control unit 45 sets the transmission area and transmittance of the shading unit 12 based on the transmission image data generated by the transmission image generation unit 44. The transmission area and transmittance of the shading unit 12 are determined by determining the transmission area and transmittance of the shading unit 12 when the real space image B is transmitted through the display unit 11. The area and clarity that reach the left and right eyes of the user. The transmission area and transmittance are set in advance and adjusted as necessary. For example, the initial transmission area is located at the center position in the left-right direction at the bottom of the display unit 11, and occupies 20% of the area of ​​the entire display unit. The initial transmittance is 70%. The shading control unit 45 outputs transmission image data consisting of the transmission area and transmittance as a transmission signal to the shading unit 12. The shading unit 12 transmits a predetermined area based on the shading signal input from the shading control unit 45.

[0035] The image determination unit 46 is connected to the image processing unit 32 and the transparent image generation unit 44. The image determination unit 46 receives the display image data generated by the image processing unit 32 and the transparent image data generated by the transparent image generation unit 44. The image determination unit 46 determines whether the degree of change (e.g., amount of change, rate of change) of the virtual space image A generated by the image processing unit 32 and displayed on the display unit 11 is less than a predetermined first threshold value. The image determination unit 46 also determines whether the degree of change (e.g., amount of change, rate of change) of the real space image B generated by the transparent image generation unit 44 and transmitted through the display unit 11 is less than a predetermined second threshold value. Here, the degree of change of the virtual space image A and the degree of change of the real space image B are the difference in the number of pixels between two frame rates that switch over time. In this case, they are expressed as the amount of change in the number of pixels or the rate of change in the number of pixels between the two frame rates. The two frame rates may be consecutive or may be separated by a predetermined number of frame rates.

[0036] The first threshold and the second threshold are set appropriately. The first threshold is, for example, the difference in the number of pixels between the frame rates of virtual space image A that changes when the image on the display screen is switched in a conference held in a VR space. The second threshold is, for example, the difference in the number of pixels between the frame rates of real space image B that changes when the image on the display of a personal computer is switched in real space.

[0037] Image determination unit 46 is connected to shading control unit 45. Image determination unit 46 outputs the determination result of the degree of change in virtual space image A and the determination result of the degree of change in real space image B to shading control unit 45. That is, when image determination unit 46 determines that the degree of change in virtual space image A is less than a first threshold value, it outputs an adjustment signal to shading control unit 45 to widen the transparent region of shading unit 12 (display unit 11) or increase the transmittance of shading unit 12 (display unit 11). Note that when image determination unit 46 determines that the degree of change in virtual space image A is less than the first threshold value and that the degree of change in real space image B is greater than a second threshold value, it outputs an adjustment signal to shading control unit 45 to widen the transparent region of shading unit 12 (display unit 11) or increase the transmittance of shading unit 12 (display unit 11).

[0038] The shading control unit 45 changes the transmission area of ​​the shading unit 12 and the transmittance of the shading unit 12 based on the determination result of the image determination unit 46. In this embodiment, the shading control unit 45 changes the transmission area and transmittance of the shading unit 12, thereby changing the transmission state of the area of ​​the display unit 11 that can be seen through the display unit 11. However, this configuration is not limited to this. For example, the display unit 11 may be configured to include the shading unit 12, and the shading control unit 45 may change the transmission area and transmittance of the display unit 11.

[0039] In addition, when the degree of change in virtual space image A is greater than the first threshold value or the degree of change in real space image B is less than the second threshold value, image determination unit 46 may output an adjustment signal to shading control unit 45 to narrow the transparent area of ​​shading unit 12 (display unit 11) or lower the transmittance of shading unit 12 (display unit 11).

[0040] Furthermore, the user image generation unit 51 generates avatar data (three-dimensional data) of the user. The user image generation unit 51 is connected to the signal synthesis unit 52. The signal synthesis unit 52 synthesizes the real space mapping result and the self-position / head tracking result processed by the real space image processing unit 42 with the user's avatar data generated by the user image generation unit 51, and generates VR space display image data.

[0041] The signal synthesis unit 52 is connected to the signal transmission unit 53. The signal transmission unit 53 transmits the VR space display image data generated by the signal synthesis unit to the virtual space construction system 100 as a stream signal.

[0042] The virtual space construction system 100 displays a three-dimensional image based on the VR space display image data, i.e., a user's avatar image, at a predetermined position in the virtual communication space. The above-mentioned virtual space image acquisition unit 31 then acquires, as a stream signal, information on the VR space from the virtual space construction system 100, i.e., an image signal of the virtual space visible in the real space seen from the direction of the user's face.

[0043] [Display method] FIG. 3 is a flowchart showing a display method according to this embodiment, FIG. 4 is a schematic diagram showing an image of a virtual space, FIG. 5 is a schematic diagram showing an image of a real space, and FIG. 6 is a schematic diagram of an image in which an image of a virtual space is superimposed on an image of a real space.

[0044] As shown in FIGS. 1 to 3, in step S11, the virtual space image acquisition unit 31 acquires information about the VR space from the virtual space construction system 100 and outputs it to the image processing unit 32. In step S12, the image processing unit 32 generates display image data based on the information about the VR space and outputs it to the display unit 11. In step S13, the display unit 11 displays a virtual space image A based on the display signal input from the image processing unit 32. For example, as shown in FIG. 4, the virtual space image A is an image of a meeting in a virtual space, and is an image in which the avatars of multiple users are displayed on a screen.

[0045] In step S14, the camera 41 acquires an image of the area the user is looking at, and outputs the captured image data to the real space image processing unit 42 and the device detection unit 43. In step S15, the real space image processing unit 42 executes visual slant processing using the captured image data of the camera 41, performs real space mapping, and estimates the user's position and head tracking, and outputs the results to the transparent image generation unit 44. Meanwhile, in step S16, the device detection unit 43 detects and identifies various devices (such as a personal computer display) using the captured image data of the camera 41, and outputs the results to the transparent image generation unit 44.

[0046] In step S17, the transparent image generation unit 44 generates transparent image data based on the processed image processed by the real space image processing unit 42 and the device detected by the device detection unit 43, and outputs the generated data to the shading control unit 45. For example, as shown in FIG. 5, a real space image B is an image in front of the user. The image shows a display, keyboard, mouse, and the like of a personal computer placed on a table. In step S18, the shading control unit 45 sets a transmission area and a transmittance of the shading unit 12 based on the transmission image data generated by the transparent image generation unit 44. In step S19, the shading unit 12 sets a predetermined transmission area and a predetermined transmittance, and the real space partial image B1 is input to the display unit 11 through the shading unit 12.

[0047] When the shading control unit 45 sets the transparent region of the shading unit 12 to the entire region, the user can see the virtual space image A displayed across the entire display unit 11, and can also see the partial real space image B1 displayed through the entire display unit 11. That is, the user can see the partial real space image B1 (FIG. 5) through the entire region of the virtual space image A (FIG. 4). When the shading control unit 45 sets the transparent region of the shading unit 12 to a portion, the user can see the virtual space image A displayed across the entire display unit 11, and can also see the partial real space image B1 displayed through a portion of the display unit 11. That is, as shown in FIG. 6, the bottom portion of the virtual space image A, which is an image of a meeting in a virtual space, is partially cut out, and the user can see the partial real space image B1, which is an image of a personal computer display, keyboard, and mouse, in the cut-out region.

[0048] In step S20, image determination unit 46 acquires the display image data generated by image processing unit 32 and the transmission image data generated by transmission image generation unit 44. In step S21, image determination unit 46 determines whether the degree of change in virtual space image A is less than a preset first threshold value. That is, image determination unit 46 determines whether the difference Pa in the number of pixels between the frame rates of virtual space image A is less than first threshold value Ps1. Here, if image determination unit 46 determines that the difference Pa in the number of pixels between the frame rates of virtual space image A is less than first threshold value Ps1 (Yes), the process proceeds to step S22.

[0049] In step S22, the image determination unit 46 determines whether the degree of change in the real space image B generated by the transmission image generation unit 44 and transmitted through the light blocking unit 12 (display unit 11) is greater than a second threshold value. That is, the image determination unit 46 determines whether the difference Pb in the number of pixels between the frame rates of the real space image B is greater than the second threshold value P2. Here, if the image determination unit 46 determines that the difference Pb in the number of pixels between the frame rates of the real space image B is greater than the second threshold value P2 (Yes), in step S23, the image determination unit 46 outputs an adjustment signal to the light blocking control unit 45 to widen the transmission area of ​​the light blocking unit 12 (display unit 11) or increase the transmittance of the light blocking unit 12 (display unit 11).

[0050] On the other hand, if the image determination unit 46 determines in step S21 that the difference in the number of pixels Pa between the frame rates of the virtual space image A is not less than the first threshold value P1 (No), it simply exits the routine. Also, if the image determination unit 46 determines in step S22 that the difference in the number of pixels Pb between the frame rates of the real space image B is not greater than the second threshold value P2 (No), it simply exits the routine.

[0051] In step S24, the light blocking control unit 45 changes the transmission area of ​​the light blocking unit 12 (display unit 11) and the transmittance of the light blocking unit 12 (display unit 11) based on the determination result of the image determination unit .

[0052] In addition, when the image determination unit 46 determines in step S21 that the difference in pixel count Pa between the frame rates of the virtual space image A is not less than the first threshold value P1 (No), or when the image determination unit 46 determines in step S22 that the difference in pixel count Pb between the frame rates of the real space image B is not more than the second threshold value P2 (No), an adjustment signal may be output to the shading control unit 45 to narrow the transparent area of ​​the shading unit 12 (display unit 11) or to lower the transmittance of the shading unit 12 (display unit 11).

[0053] [Effects of the embodiment] The display device of this embodiment includes a display unit 11 that displays a virtual space image A, an equipment detection unit (detection unit) 43 that detects an object in a real space image B, and a shading control unit (transmission control unit) 45 that changes the transmission state of the transmission area of ​​the display unit 11 where the object detected by the equipment detection unit 43 can be seen through the display unit 11.

[0054] Therefore, it is possible to display a virtual space image A on the display unit 11, and also possible to display an object in a real space image B through a partial or entire area of ​​the display unit 11, thereby optimally displaying the real space image on at least a portion of the virtual space image.

[0055] In the display device of this embodiment, when the degree of change in virtual space image A is less than a preset first threshold, shading control unit 45 widens the transmission area of ​​shading unit 12 (display unit 11) or increases the transmittance of shading unit 12 (display unit 11). As a result, objects in real space image B can be clearly displayed in accordance with the change in virtual space image A.

[0056] In the display device of this embodiment, when the degree of change in virtual space image A is less than a first threshold value and the degree of change in real space image B is greater than a preset second threshold value, shading control unit 45 widens the transparent area of ​​shading unit 12 (display unit 11) or increases the transmittance of shading unit 12 (display unit 11). Therefore, it is possible to clearly display an object in real space image B according to the change status of virtual space image A and the change status of real space image B.

[0057] The display device according to the present invention has been described above, but it may be embodied in various different forms other than the above-described embodiments.

[0058] The components of the illustrated display device are conceptual and functional, and do not necessarily have to be physically configured as shown. In other words, the specific form of each device is not limited to that shown, and all or part of each device may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0059] The configuration of the display device is realized, for example, as software, by a program loaded into memory. In the above embodiment, the functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms using only hardware, only software, or a combination of both.

[0060] The above-described components include those that can be easily imagined by a person skilled in the art and those that are substantially the same. Furthermore, the above-described configurations can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0061] 10 display device 11 Display section 12 Light blocking section 21 Display panel 22 Half Mirror 23 Combiner mirror 25 Blackout Panel 31 Virtual space image acquisition unit 32 Image processing section 41 Camera 42 Real-space image processing section 43 Device detection unit 44 Transparent image generation unit 45 Light blocking control section (transmission control section) 46 Image Judgment Unit 51 User image generation unit 52 Signal synthesis unit 53 Signal transmitter A Virtual Space Image B. Real space image La display light Lb reflected light Lc real image light

Claims

1. a display unit that displays an image of the virtual space; a detection unit that detects an object in real space; a transmission control unit that changes a transmission state of a region of the display unit where the object detected by the detection unit can be seen through the display unit; and Equipped with the transparency control unit widens the area or increases the transmittance of the display unit when a degree of change in the image in the virtual space is less than a predetermined first threshold value; Display device.

2. the transparency control unit widens the area or increases the transmittance of the display unit when a degree of change in the image in the virtual space is less than the first threshold value and a degree of change in the image in real space that includes the object and is visible through the display unit is greater than a preset second threshold value. The display device according to claim 1 .

3. displaying an image of the virtual space on a display unit; Detecting an object in real space; changing a transparency state of an area of ​​the display unit where the detected object can be seen through the display unit; widening the area or increasing the transmittance of the display unit when the degree of change in the image of the virtual space is less than a predetermined first threshold value; Display methods including.

4. displaying an image of the virtual space on a display unit; Detecting an object in real space; changing a transparency state of an area of ​​the display unit where the detected object can be seen through the display unit; widening the area or increasing the transmittance of the display unit when the degree of change in the image of the virtual space is less than a predetermined first threshold value; A program that causes a computer operating as a display device to execute the above.

Images