Display system, image processing apparatus, method for controlling the same, and program

Abstract

To prevent a reduction in the quality of images displayed by a head-mounted display device.SOLUTION: A display system comprises: a head-mounted display device (10); a first image processing apparatus (20) that includes first CG generation means (220) generating computer graphics, and outputs an image for display to the display device (10); and a second image processing apparatus (30) that includes second CG generation means (320) generating computer graphics, and can communicate with the first image processing apparatus (20). The first image processing apparatus (20) includes: abnormality detection means (270) that detects an abnormality in an image input from the second image processing apparatus (30); and image selection means (250) that, according to a result of detection performed by the abnormality detection means (270), selects, as the image for display, one of a first image using the computer graphics generated by the first CG generation means (220) and a second image using the computer graphics generated by the second CG generation means (320).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a display system, a video processing device, a control method thereof, and a program suitable for use in, for example, an MR system. [Background technology]

[0002] VR (Virtual Reality) technology is known as a technology that allows users to experience virtual space. Mixed reality (MR) technology is also known as a technology that seamlessly fuses real space and virtual space in real time. In an MR system that uses a video see-through HMD (head mounted display), a composite image is presented to the HMD wearer in which computer graphics (hereinafter referred to as CG) is superimposed on a real space image acquired by an imaging unit built into the HMD. These images are independent for each of the left and right eyes, and a three-dimensional MR space using stereoscopic moving images can be presented to the HMD wearer. In VR, virtual space images are used instead of real space images.

[0003] An MR system generally comprises an HMD that mainly captures real space and displays MR images, and a personal computer or workstation that mainly generates CG images, superimposes them on real space images, and performs various image processing. Image data and other data are transmitted between the two devices via wired connections such as metal or optical cables, or wireless connections such as wireless LANs (Local Area Networks).

[0004] In such an MR system, even if data transmission is interrupted due to an error or some other reason, a technique is known that compensates for the missing image data so as to minimize the discomfort felt by the HMD wearer. For example, Patent Document 1 discloses a configuration in which a CG estimated image corresponding to the CG image of the most recently received composite image is estimated based on changes in position and orientation of a CG extracted image extracted in the past, and an interpolation image is generated by superimposing the CG estimated image on the most recently generated captured image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-005778 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the conventional technology disclosed in Patent Document 1 estimates a CG estimated image from past information to generate an interpolation image, and it cannot be said that it can completely compensate for missing image data. As a result, the quality of the image displayed on the HMD may decrease, the HMD wearer may feel uncomfortable, and the immersive feeling of the MR experience may be lost.

[0007] The present invention has been made in consideration of the above-mentioned points, and has an object to prevent degradation in the quality of images displayed on a head-mounted display device. [Means for solving the problem]

[0008] The display system of the present invention comprises a head-mounted display device, a first image processing device comprising a first CG generation means for generating computer graphics and outputting an image for display to the display device, and a second image processing device comprising a second CG generation means for generating computer graphics and capable of communicating with the first image processing device, wherein the first image processing device comprises an abnormality detection means for detecting an abnormality in an image input from the second image processing device or an abnormality in a connection state with the second image processing device, and an image selection means for selecting, depending on a result of detection by the abnormality detection means, whether to use a first image using the computer graphics generated by the first CG generation means as the image for display, or a second image using the computer graphics generated by the second CG generation means as the image for display. Effect of the Invention

[0009] According to the present invention, it is possible to prevent degradation of the quality of an image displayed on a head-mounted display device. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a schematic configuration of a display system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a display system according to a first embodiment. [Diagram 3] 3 is a diagram for explaining a process executed by a video selection unit in the display system according to the first embodiment. FIG. [Figure 4] FIG. 11 is a block diagram showing the configuration of a display system according to a second embodiment. [Diagram 5] FIG. 11 is a block diagram showing a configuration of a display system according to a third embodiment. [Figure 6] FIG. 13 is a block diagram showing a configuration of a display system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. [First embodiment] Fig. 1 is a diagram showing a schematic configuration of a display system 1 according to the first embodiment, and Fig. 2 is a block diagram showing the configuration of the display system 1 according to the first embodiment. 1 and 2, the display system 1 includes an HMD (head mounted display) 10 which is a head mounted display device, a first image processing device 20, a second image processing device 30, and a cable 40 which connects the image processing devices 20 and 30. In this embodiment, the display system 1 functions as an MR system which presents a composite image in which computer graphics (hereinafter, referred to as CG) are superimposed on a real space image which is an image captured by an imaging unit of the HMD 10 to a wearer of the HMD 10.

[0012] The first image processing device 20 is an image processing device that directly controls the HMD 10 and outputs a display image to the HMD 10. The second image processing device 30 is an image processing device that can communicate with the first image processing device 20 and is, for example, an image processing device called a PCWS (PC workstation) or the like. The first image processing device 20 and the second image processing device 30 generate a composite image of a mixed reality space that combines a real space and a virtual space, and provide the composite image to the HMD 10. In this embodiment, the communication path between the image processing devices 20 and 30 is a wired communication path using Thunderbolt 3 or the like, but may be a wireless communication path.

[0013] The HMD 10 includes an imaging element 100 constituting an imaging section, and a display element 110 constituting a display section. The image sensor 100 is configured with a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, etc. The image sensor 100 acquires an imaged image of a real space via an optical system (not shown) in accordance with an exposure time, a sensor gain, an exposure start timing, etc. that are set based on a control signal from the first image processing device 20. The display element 110 is configured by an OLED (Organic Light-Emitting Diode), an LCD (Liquid Crystal Display), etc. The display element 110 presents a synthetic image to the wearer of the HMD 10 via an optical system (not shown).

[0014] The display system 1 handles a stereoscopic image using a right-eye image and a left-eye image, and the image sensor 100 and the display device 110 are paired for the right eye and the left eye, not shown. In addition, a background image serving as a base for a composite image and a position detection image for generating CG are generated from the image captured by the image sensor 100. In this embodiment, a method of cutting out the other image from the image captured by the same image sensor is assumed, but since the required angle of view, resolution, image processing, etc. are different between the background image and the position detection image, separate image sensors may be used. In addition, the HMD 10 is equipped with an IMU (Inertial Measurement Unit, not shown; a so-called position and orientation sensor that detects angular velocity and acceleration of three axes and can calculate the position and orientation of the device) that detects position and orientation information of the HMD 10. Note that the technology related to capturing and displaying stereoscopic images is publicly known, and detailed description thereof will be omitted.

[0015] The first video processing device 20 includes a pre-processing unit 200, a position detection unit 210, a CG generation unit 220, a memory unit 230, a video synthesis unit 240, a video selection unit 250, a post-processing unit 260, an abnormality detection unit 270, and a recording and playback unit 280. The second video processing device 30 also includes a position detection unit 310 , a CG generation unit 320 , a storage unit 330 , a video synthesis unit 340 , and a recording and playback unit 380 .

[0016] The pre-processing unit 200 develops the captured image of the image sensor 100 transmitted from the HMD 10 in a form suitable for the background and position detection of the composite image, and outputs the background image to the image composition unit 240 and the position detection image to the position detection unit 210. The pre-processing unit 200 also outputs the background image to the image composition unit 340 and the position detection image to the position detection unit 310.

[0017] The position detection unit 210 outputs information required for generating CG, such as the position where CG is generated and the line of sight angle, based on the position detection image and the position and orientation information of the HMD 10. Furthermore, the position detection unit 310 outputs information required for generating CG, such as the position where CG is generated and the line of sight angle, based on the position detection image and the position and orientation information of the HMD 10.

[0018] The CG generating unit 220 generates a predetermined CG according to the output of the position detecting unit 210. The CG is rendered based on CAD data stored in a FROM (Flash ROM) or HDD (Hard Disc Drive) in the storage unit 230. The CAD data is downloaded from an external device, for example, and stored in the storage unit 230. The CG generating unit 220 functions as a first CG generating means in the present invention. Moreover, the CG generating unit 320 generates a predetermined CG according to the output of the position detecting unit 310. The CG is rendered based on CAD data stored in a FROM or HDD in the storage unit 330. The CG generating unit 320 functions as a second CG generating means in the present invention.

[0019] The image synthesizing unit 240 generates a synthetic image by superimposing CG on a background image. The image synthesizing unit 340 generates a synthetic image by superimposing CG on a background image. The image synthesizing units 240 and 340 function as a synthesizing means according to the present invention.

[0020] Video selection unit 250 selects whether the composite image generated by video synthesis unit 240 or the composite image generated by video synthesis unit 340 is to be used as the display image according to the detection result by abnormality detection unit 270. Video selection unit 250 includes buffer 250a for absorbing the timing difference when switching between the composite image generated by video synthesis unit 240 and the composite image generated by video synthesis unit 340. Video selection unit 250 functions as the video selection means referred to in the present invention.

[0021] The post-processing unit 260 performs image processing such as distortion correction of an optical system (not shown) on the display image, and outputs the image to the HMD 10 .

[0022] The anomaly detection unit 270 detects anomalies in the image input from the second video processing device 30. In this embodiment, the image input from the second video processing device 30 is a composite image generated by the image synthesis unit 340, and the presence or absence of a loss in the image is detected. When the anomaly detection unit 270 detects a loss, it outputs the fact to the image selection unit 250. The anomaly detection unit 270 functions as an anomaly detection means as referred to in the present invention. Note that, although the image input from the second video processing device 30 is the object of monitoring, the connection state with the second video processing device 30 (whether wired or wireless) may also be the object of monitoring to detect anomalies therein.

[0023] The recording and reproducing unit 280 records and reproduces the output video of the first video processing device 20 using the storage unit 230. Moreover, the recording and reproducing unit 380 records and reproduces the output video of the second video processing device 30 using the storage unit 330. During reproduction by the recording and reproducing unit 280, the second video processing device 30 may not be connected. The recording and reproducing units 280 and 380 function as the recording and reproducing means referred to in the present invention.

[0024] In the display system 1 configured as described above, it is possible to provide an MR experience with the first image processing device 20 alone, even in a state where the second image processing device 30 is not connected. For example, when the second image processing device 30 is connected, high-precision position detection and high-quality rendering can be enabled, and when the first image processing device 20 is operating alone, simple quality can be provided. Although Figures 1 and 2 show one first image processing device 20 and one second image processing device 30, the configuration may also be such that, for example, multiple first image processing devices 20 are connected to the second image processing device 30.

[0025] Next, the process executed by video selection section 250 in display system 1 will be described with reference to FIG. "Video 1" indicated by diagonal lines represents the output of the video synthesis unit 240 of the first video processing device 20. "Video 2" indicated by white represents the output of the video synthesis unit 340 of the second video processing device 30. "Video output" represents the display video output by the video selection unit 250 to the HMD 10. ..., n-1, n, n+1, ... are frame numbers. In this example, Video 2 follows Video 1 by two frames. Frames indicated in black represent missing frames. Under normal circumstances (when no abnormality is detected by abnormality detection section 270), video 2 is output as the video.

[0026] FIG. 3(a) shows a situation in which a loss occurs in frame number n of video 2. Since there is no loss up to frame number n-1, video 2 is selected for video output. When the abnormality detection unit 270 detects a loss of frame number n, the video selection unit 250 switches the video output to video 1. At this time, the buffer 250a absorbs the timing difference between video 1 and video 2. Thereafter, video 1 is selected for video output until there is no loss of video 2. Here, only one frame is lost, and video 2 is again selected for video output from frame number n+1 onwards. Possible causes of such a single frame loss include, for example, a momentary interruption in communication between the first video processing device 20 and the second video processing device 30, or a momentary increase in the load on the second video processing device 30 side causing a drop in the frame rate. In this way, loss of the video 2 input from the second video processing device 30 is detected on a frame-by-frame basis, and the video is switched to the video 1 on a frame-by-frame basis.

[0027] FIG. 3(b) shows a situation in which loss continues from frame number n onwards in video 2. Since there are no missing frames up to frame number n-1, video 2 is selected for video output. When abnormality detection unit 270 detects a missing frame at frame number n, video selection unit 250 switches the video output to video 1. At this time, buffer 250a absorbs the timing difference between video 1 and video 2. Thereafter, video 1 is selected for video output until there are no missing frames in video 2. Here, if, for example, k+1 consecutive frames are lost in video 2, it is assumed that an abnormality has occurred in the second video processing device 30 or its connection, and the first video processing device 20 transitions to a standalone operation mode. When transitioning to the standalone operation mode, the timing difference between video 1 and video 2 is adjusted, and in the illustrated example, it is adjusted to jump from frame number n+k to n+k+3, and thereafter video 1 is output with little latency. Possible causes of such continued video loss include, for example, when communication between the first video processing device 20 and the second video processing device 30 is cut off, or when the second video processing device 30 hangs up. In this way, when the loss of the video 2 input from the second video processing device 30 continues for a predetermined period of time, it is determined that the video 2 has stopped, and the first video processing device 20 transitions to a standalone operation mode.

[0028] FIG. 3(c) shows a situation where the image 2 is restored while the first image processing device 20 is in the stand-alone operation mode. If the video output is switched to video 2 at the point where video 2 is restored at frame number n, the frame number will be reversed due to the timing difference between video 1 and video 2. Therefore, the frame number of the video output (frame number n+1 in the illustrated example) is held by buffer 250a, and the video output is switched to video 2 at the timing when video 2 with the same frame number as the held frame number is input. This makes it possible to restore to an operation mode in which video 2 is the video output without the frame number being reversed.

[0029] As described above, the presence or absence of loss in the image input from the second image processing device 30 (the composite image generated by the image synthesis unit 340) is detected, and depending on the detection result, it is selected whether to use the composite image generated by the image synthesis unit 240 as the image for display, or the composite image generated by the image synthesis unit 340 as the image for display. This makes it possible to maintain a high-quality MR experience without degrading the quality of the image displayed on the HMD 10, even when, for example, a change occurs in the state of communication between the first image processing device 20 and the second image processing device 30.

[0030] [Second embodiment] 4 is a block diagram showing the configuration of the display system 1 according to the second embodiment. In the following, the same components as those in the display system 1 according to the first embodiment are denoted by the same reference numerals, and the description thereof is omitted, and the description will be focused on the differences from the display system 1 according to the first embodiment. In the second embodiment, compared to the first embodiment, the second video processing device 30 does not include the video synthesis unit 340. In this case, the image input from the second image processing device 30 is CG generated by the CG generating section 320, and the abnormality detecting section 270 detects whether or not there is a loss therein. In the first video processing device 20, the video selection unit 250 is disposed before the video synthesis unit 240. Depending on the detection result in the abnormality detection unit 270, the video selection unit 250 selects whether to use the CG generated by the CG generation unit 220 or the CG generated by the CG generation unit 320 for the synthetic image generated by the video synthesis unit 240.

[0031] [Third embodiment] 5 is a block diagram showing the configuration of a display system 1 according to the third embodiment. In the following, the same components as those in the display system 1 according to the first embodiment are denoted by the same reference numerals, and the description thereof is omitted, and the description will be focused on the differences from the display system 1 according to the first embodiment. In the third embodiment, as compared with the first embodiment, the second video processing device 30 does not have a position detection unit 310. The CG generation unit 320 of the second video processing device 30 generates CG based on the output of the position detection unit 210 of the first video processing device 20.

[0032] [Fourth embodiment] 6 is a block diagram showing the configuration of a display system 1 according to a fourth embodiment. In the following, the same components as those in the display system 1 according to the first embodiment are denoted by the same reference numerals, and the description thereof is omitted, and the differences from the display system 1 according to the first embodiment are mainly described. In the fourth embodiment, compared to the first embodiment, the second video processing device 30 does not include the position detection unit 310 and the video synthesis unit 340. As in the second embodiment, in the first video processing device 20, the video selection unit 250 is disposed before the video synthesis unit 240. Depending on the detection result in the abnormality detection unit 270, the video selection unit 250 selects whether to use the CG generated by the CG generation unit 220 or the CG generated by the CG generation unit 320 for the synthetic image generated by the video synthesis unit 240. Moreover, similarly to the third embodiment, the CG generating unit 320 of the second video processing device 30 generates CG based on the output of the position detecting unit 210 of the first video processing device 20 .

[0033] 1 to 6, the configuration of the display system 1 is not limited to this. For example, the first image processing device 20 may be integrated into the HMD 10, that is, the first image processing device 20 may be configured within the housing of the HMD 10. Although the display system 1 functions as an MR system, it may function as a VR system using virtual space images. Various parameters may be appropriately designed according to the purpose of use of the display system 1. The abnormality detection unit 270 may detect partial loss of the image itself, rather than frame by frame, in the image input from the second image processing device 30. In this case, the image selection unit 250 uses the composite image generated by the image composition unit 240 for the image of the partially missing area. The functional configurations shown in FIGS. 2 and 4 to 6 may be implemented as hardware, or may be realized by an information processing device executing software.

[0034] Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative of the specific examples of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features. (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0035] The disclosure of this embodiment includes the following configuration. (Configuration 1) A head-mounted display device; a first image processing device including a first CG generating means for generating computer graphics and outputting an image to be displayed on the display device; a second CG generating means for generating computer graphics, and a second image processing device capable of communicating with the first image processing device; The first image processing device comprises: an abnormality detection means for detecting an abnormality in an image input from the second image processing device or an abnormality in a connection state with the second image processing device; and an image selection means for selecting, depending on a result of detection by the abnormality detection means, whether to use a first image using the computer graphics generated by the first CG generation means as the display image, or a second image using the computer graphics generated by the second CG generation means as the display image. (Configuration 2) the first image processing device includes a synthesis means for synthesizing a predetermined image and a computer graphic; the first image is an image obtained by superimposing the computer graphic generated by the first CG generating means on the predetermined image, 2. The display system according to configuration 1, wherein the second image is an image in which the computer graphic generated by the second CG generating means is superimposed on the predetermined image. (Configuration 3) 3. The display system according to configuration 1 or 2, wherein the abnormality detection means detects loss of the image input from the second image processing device. (Configuration 4) 4. The display system according to configuration 3, wherein the abnormality detection means detects loss of the image input from the second image processing device on a frame-by-frame basis. (Configuration 5) The display system according to configuration 3 or 4, wherein the abnormality detection means determines that the image input from the second image processing device has stopped when the image input from the second image processing device continues to be lost for a predetermined period of time. (Configuration 6) 6. The display system according to any one of configurations 1 to 5, wherein the image input from the second image processing device is the computer graphics generated by the second CG generating means. (Configuration 7) the second image processing device includes a synthesis means for synthesizing the predetermined image and computer graphics; The display system according to configuration 2, wherein the image input from the second image processing device is an image in which the computer graphic generated by the second CG generating means is superimposed on the specified image. (Configuration 8) The image selection means normally selects the second image as the display image, 8. The display system according to any one of configurations 1 to 7, wherein when an abnormality is detected by the abnormality detection means, the first image is used as the display image. (Configuration 9) 9. The display system according to any one of configurations 1 to 8, further comprising a buffer for absorbing a timing difference when the image selection means switches between the first image and the second image. (Configuration 10) 10. The display system according to any one of configurations 1 to 9, wherein the first image processing device and the second image processing device each include a recording and reproducing unit that records and reproduces images using a storage unit. (Configuration 11) The display device includes an imaging unit, The display system according to configuration 2 or 7, wherein the predetermined image is an image captured by the imaging unit. [Explanation of symbols]

[0036] 1: display system, 10: HMD, 20: first image processing device, 30: second image processing device, 40: cable, 100: imaging element, 110: display element, 200: pre-processing unit, 210, 310: position detection unit, 220, 320: CG generation unit, 230, 330: storage unit, 240, 340: image synthesis unit, 250: image selection unit, 250a: buffer, 260: post-processing unit, 270: abnormality detection unit, 280, 380: recording and playback unit

Claims

1. a head-mounted display device; a first image processing device including a first CG generating means for generating computer graphics and outputting an image to be displayed on the display device; a second CG generating means for generating computer graphics, and a second image processing device capable of communicating with the first image processing device; The first image processing device comprises: an abnormality detection means for detecting an abnormality in the image input from the second image processing device or an abnormality in the connection state with the second image processing device; and image selection means for selecting, depending on the result of detection by the abnormality detection means, whether to use a first image using the computer graphics generated by the first CG generation means as the image to be displayed, or a second image using the computer graphics generated by the second CG generation means as the image to be displayed.

2. the first image processing device includes a synthesizing means for synthesizing a predetermined image with computer graphics; the first image is an image obtained by superimposing the computer graphics generated by the first CG generating means on the predetermined image, 2. The display system according to claim 1, wherein the second image is an image in which the computer graphics generated by the second CG generating means are superimposed on the predetermined image.

3. 3. The display system according to claim 1, wherein the abnormality detection means detects a loss of the image input from the second image processing device.

4. 4. The display system according to claim 3, wherein the abnormality detection means detects loss of the image input from the second image processing device on a frame-by-frame basis.

5. The display system described in Claim 4, characterized in that the abnormality detection means analyzes the continuity between frames based on the frame number associated with each frame of the image input from the second image processing device, and detects a loss of the image in response to a break in the continuity between the frames.

6. 4. The display system according to claim 3, wherein said abnormality detection means determines that the video has stopped when a dropout of the video input from said second video processing device continues for a predetermined period of time.

7. 3. The display system according to claim 1, wherein the image input from the second image processing device is the computer graphics generated by the second CG generating means.

8. the second image processing device includes a synthesizing means for synthesizing the predetermined image with computer graphics, 3. The display system according to claim 2, wherein the image input from the second image processing device is an image in which the computer graphics generated by the second CG generating means is superimposed on the predetermined image.

9. the image selection means normally selects the second image as the display image, 3. The display system according to claim 1, wherein when the abnormality detection means detects an abnormality, the first image is used as the display image.

10. 3. The display system according to claim 1, further comprising a buffer for absorbing a timing difference when the image selection means switches between the first image and the second image.

11. 3. The display system according to claim 1, wherein the first and second video processing devices each include a recording / playback unit that records and plays back video using a storage unit.

12. the display device includes an imaging unit, 9. The display system according to claim 2, wherein the predetermined image is an image captured by the imaging unit.

13. A video processing device that outputs a display image to a head-mounted display device, a first CG generating means for generating computer graphics; an abnormality detection means for detecting an abnormality in an image input from another image processing device having a second CG generation means for generating computer graphics, or an abnormality in a connection state with the other image processing device; and image selection means for selecting, depending on the result of detection by said anomaly detection means, whether to use a first image using said computer graphics generated by said first CG generation means as said display image, or a second image using said computer graphics generated by said second CG generation means as said display image.

14. A control method for a video processing device that controls a video processing device that includes a first CG generation means for generating computer graphics and outputs a display image to a head-mounted display device, comprising: detecting an abnormality in an image input from another image processing device having a second CG generating means for generating computer graphics, or an abnormality in a connection state with the other image processing device; and selecting, depending on the result of the detection, whether to use a first image using the computer graphics generated by the first CG generation means as the image to be displayed, or a second image using the computer graphics generated by the second CG generation means as the image to be displayed.

15. A program for controlling an image processing device that includes a first CG generation means for generating computer graphics and outputs a display image to a head-mounted display device, a process of detecting an abnormality in an image input from another image processing device having a second CG generating means for generating computer graphics, or an abnormality in a connection state with the other image processing device; and a step of selecting, depending on the result of the detection, whether to use a first image using the computer graphics generated by the first CG generation means as the image for display, or a second image using the computer graphics generated by the second CG generation means as the image for display.

Images