Image processing device, method, and program

Abstract

To switch a method for composing real images and virtual images with a simple configuration.SOLUTION: An image processing apparatus comprises: acquisition means that acquires real images picked up by an imaging unit and outputs the real images; creation means that receives the real images output by the acquisition means, generates and outputs virtual images, and outputs the received real images or converted images obtained by converting the real images; composition means that composes the real images output by the acquisition means, the virtual images output by the creation means, and the real images or the converted images output by the creation means; and display control means that displays, on a display, composite images obtained by the composition means. The image processing apparatus changes the delay time of the real images in the images displayed on the display depending on if the creation means outputs the real images or outputs the converted images.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for synthesizing a captured image and a virtual image and displaying the synthesized image. [Background technology]

[0002] In recent years, technologies such as mixed reality (MR) and augmented reality (AR) have been proposed as a way to seamlessly combine real and virtual spaces in real time. One of these technologies is a technology that uses a video see-through head-mounted display (HMD). In this technology, an image that roughly matches the image observed from the HMD user's pupil position is captured by a video camera or the like, and computer graphics (CG) are superimposed on the captured image, which the HMD wearer can observe through an internal panel of the HMD.

[0003] In this case, delays in the images displayed on the HMD can be a problem, causing users to feel sick or uncomfortable. The causes of delays include image transmission and the processing required to render CG, and attempts have been made to solve each of these issues by improving transmission and processing methods. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4847192 Summary of the Invention [Problem to be solved by the invention]

[0005] In a typical video see-through method, real-world images are captured by the HMD and transmitted to an external device, and CG generated by signal processing is then superimposed on the real-world images, transmitted again to the HMD, and displayed on the HMD. This transmission and processing process causes delays in the displayed images.

[0006] In addition, there is a known technique in which real-world images are captured and displayed on an HMD with minimal processing, and then separately processed CG is composited onto the image. However, although this reduces the delay in the real-world images, it creates a new issue of a time difference with the CG delay, which is not desirable in some use cases.

[0007] Patent Document 1 presents a method of selecting between transmitting a real image to a CG rendering unit, overlaying it, and then transmitting it again to the HMD, or displaying the real image on the HMD via the shortest route and combining it with a separately generated CG within the HMD. However, this method requires route switching, status monitoring, and communication, making the configuration complex. [Means for solving the problem]

[0008] In order to solve the above problem, according to one aspect of the present invention, an image processing device comprises an acquisition means for acquiring a real image captured by an imaging unit and outputting the real image, a generation means for receiving the real image output by the acquisition means, generating and outputting a virtual image, and outputting a converted image obtained by converting the received real image or the real image, a synthesis means for synthesizing the real image output by the acquisition means, the virtual image output by the generation means, and the real image or the converted image output by the generation means, and a display control means for displaying the image synthesized by the synthesis means on a display unit, wherein the generation means changes the delay time of the real image in the image displayed on the display unit depending on whether it outputs the real image or the converted image. [Effects of the Invention]

[0009] According to the present invention, the method for combining real and virtual images can be changed with a simple configuration. [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 1 is a diagram showing the functional configuration of an image processing device relating to a method in which the difference between real images and CG is small. [Figure 2] FIG. 1 is a diagram illustrating a functional configuration of an image processing device relating to a system in which only real video has low delay. [Figure 3] 1 is a diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment. [Figure 4] FIG. 10 is an explanatory diagram of a layer structure in a conventional calculation unit in the first embodiment. [Figure 5] FIG. 10 is an explanatory diagram of an image synthesized by a processing unit of a conventional method in the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a layer structure in a low-delay calculation unit in the first embodiment. [Figure 7] FIG. 2 is an explanatory diagram of an image synthesized by a low-delay processing unit in the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating the functional configuration of an image processing device according to a second embodiment. [Figure 9] 1 is a diagram illustrating a hardware configuration of an image processing device according to a first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the drawings. Note that the configurations described in the following embodiments are representative examples, and the scope of the present invention is not limited to these specific configurations.

[0012] (First embodiment) 1 is a diagram showing the functional configuration of an image processing device relating to a method in which the difference between real images and CG is small. First, with reference to FIG. 1, the flow of signal processing in the conventional video see-through method will be described.

[0013] 1, reference numeral 111 denotes an objective optical system that captures a real image of the outside world as light, and 112 denotes an image sensor that converts the real image from an optical signal to an electrical signal, which constitute the imaging unit 11. In this case, the imaging unit 11 may be configured such that the objective optical system 111 and the image sensor 112 are separate structures, or may be an integrated camera module.

[0014] Reference numeral 121 denotes a display that converts an electrical image signal into an optical signal, and reference numeral 122 denotes an eyepiece optical system that delivers the image light to the eyes of the HMD user, and these together constitute the display unit 12. In this case, the display 121 is a flat-panel image display element such as an organic EL display or a liquid crystal display.

[0015] An image processing unit 131 performs processing to develop the RAW data acquired by the image sensor 112 and processing to adjust the image quality, and constitutes the processing unit 13. There is relatively little delay in transmission from the imaging unit 11 to the processing unit 1, and the image processing unit 131 performs processing with relatively little delay in the entire system. In this case, the processing unit 13 may be located inside the HMD together with the imaging unit 11 and the display unit 12, or may be located outside the HMD that has the imaging unit 11 and the display unit 12.

[0016] Numeral 141 denotes a CG superimposing unit, which calculates the position and orientation information of the HMD, then renders a virtual image (CG) based on the position and orientation information, receives the real image processed by the image processing unit 131, and generates an image in which the CG is superimposed on the real image, constituting the calculation unit 14. There are many methods for calculating the position and orientation information of the HMD, such as a method using the real image transmitted from the processing unit 33, a method using an image from a system separate from the display, or a method of calculation from various sensors such as acceleration and angular velocity, but details will be omitted here.

[0017] However, calculating the position and orientation mainly from the video requires a lot of processing, which requires a lot of system resources or execution time, and this affects the delay. Furthermore, the CG rendering process may also require a lot of processing depending on the amount of CG data and image quality, which also affects the delay. Therefore, delays occur in the CG superimposition unit 141 due to the large number of calculations, and transmission from the processing unit 13 to the calculation unit 14 has a relatively large delay, so the portion of the system that goes through the calculation unit 14 has a relatively large delay. Here, for the sake of convenience, the processing unit 13 and the calculation unit 14 are described separately, but it is also possible that the processing unit 13 and the calculation unit 14 are located in the same place. Alternatively, it is also possible that the calculation unit 14 is located on the cloud, which is far away from the location where the HMD is used, and the transmission delay is large.

[0018] Also, although it was mentioned that CG is rendered and overlaid, the CG can be 3D data handled by CAD, etc., or it can simply be an image that replicates the PC screen displayed on a regular display.

[0019] The imaging unit 11 converts the image of the outside world in front of the HMD user's line of sight from light into an electrical signal, the processing unit 13 processes the image to improve image quality, and the calculation unit 14 superimposes the CG image and the real image as seen from the position and posture of the HMD user. By performing display control to display the composite image on the display unit 12 inside the HMD, the HMD user can observe the CG image simultaneously with the real image of the outside world.

[0020] This conventional video see-through method has the advantage that it is possible to display CG with high positional accuracy relative to real images, and there is no difference in delay between the real images and CG. However, there is a problem in that the image observed by the HMD user has a large delay due to delays associated with transmission and processing on each path, especially delays via the calculation unit 14.

[0021] Fig. 2 is a diagram showing the functional configuration of an image processing device relating to a system in which only real images have low latency. Next, the flow of signal processing in a video see-through system in which only real images have low latency will be described with reference to the imaging and display device in Fig. 2.

[0022] In FIG. 2, the imaging unit 11 and the display unit 12 are the same as those in FIG. 1, and therefore their explanation will be omitted.

[0023] Reference numeral 231 denotes an image processing unit, which performs the same processing as the image processing unit 131, but outputs the real image not to the calculation unit 24 but to an image synthesis unit 232 also in the processing unit 23.

[0024] Reference numeral 241 denotes a CG generation unit, which is the same as the CG superimposition unit 141 in that it renders CG after calculating the position and orientation of the HMD, and constitutes the calculation unit 24. However, after rendering, it does not combine with the real image, but outputs the CG and chromakey information or alpha channel information for combining to the processing unit 23. Again, since the CG generation unit 241 requires many calculations, the processing in the calculation unit 24 takes time, and there is still a relatively large delay in the entire system. In this case, as mentioned above, the processing unit 23 and the calculation unit 24 may be in the same place or in different places.

[0025] The real image captured by the imaging unit 11 is reflected by the low-delay processing unit 23 and displayed on the display unit 12, so the real image observed by the HMD user can be displayed with as little delay as possible. However, since it takes time to calculate the position and orientation of the HMD to generate CG, and to render the CG, and further time is required for transmission, the delay in the synthesized CG image is not significantly different from that of the conventional video see-through method. As a result, a new issue arises: a difference in delay time between the real image displayed with low delay and the CG with large delay.

[0026] As explained above, the conventional video see-through method has the problem of large delays, although it does not cause a delay time difference between the real image and the CG, while the video see-through method, which only has low delay for the real image, has the problem of causing a delay time difference between the real image and the CG. Therefore, the convenience of HMDs will be improved if HMD users can choose either method depending on their use case and environment.

[0027] 3 is a diagram showing the functional configuration of the image processing device according to the first embodiment, with reference to which the flow of signal processing in the image capturing and display device will be described.

[0028] In FIG. 3, the imaging unit 11 and the display unit 12 are the same as those in FIG. 1, and therefore their explanation will be omitted.

[0029] 3, 331 is an image processing unit, and although the processing itself is the same as that of image processing unit 131, the image is output to both calculation unit 14 and image synthesis unit 232, which is also located within processing unit 33. In this case, the images output to image synthesis unit 232 and calculation unit 14 may be exactly the same, or may differ in some respects, such as resolution or compression, but the original image is a real image that constitutes part of the field of view of the HMD user, captured by imaging unit 11.

[0030] In the figure, 341 is a CG synthesis unit, which is no different from the CG superimposition unit 141 in that it renders CG after calculating the position and orientation of the HMD, and constitutes the calculation unit 34. However, the user selects whether to superimpose CG and real images after rendering, or whether to output CG and chromakey information or alpha channel information for compositing to the processing unit 33. The layer configuration in which real images and CG are superimposed will be described later using Figures 4 to 7. Again, the CG synthesis unit 341 requires many calculations, so it takes time to pass through the calculation unit 34, and there is a relatively large delay in the entire system. In this case, as mentioned above, the processing unit 33 and the calculation unit 34 may be located in the same place or in different places.

[0031] The real image captured by the imaging unit 11 can be reflected by the processing unit 33 with little delay and displayed on the display unit 12, or it can be displayed on the display unit 12 without any delay time difference with the CG via the calculation unit 34. This switching can be achieved by simply changing the real image synthesized by the CG synthesis unit 341 without changing the signal processing flow and configuration shown in Fig. 3. This simplifies the configuration of the imaging unit 11, display unit 12, and processing unit 33, and allows them to be used without switching operation modes, etc.

[0032] Next, the hardware configuration of the image processing device including the processing unit 33 and the calculation unit 34 will be described. Fig. 9 is a block diagram showing the hardware configuration of the image processing device in this embodiment. In the figure, a CPU 92 comprehensively controls each device connected via a bus 91. The CPU 92 also reads and executes instructions and programs stored in a read-only memory (ROM) 93. The operating system (OS), as well as each processing program, device driver, etc. according to this embodiment are stored in the ROM 93, temporarily stored in a random access memory (RAM) 94, and executed appropriately by the CPU 92.

[0033] The input I / F 95 receives an input signal from an external device such as the imaging unit 11 in a format that can be processed by the image processing device. The output I / F 96 outputs an output signal in a format that can be processed by an external device such as the display unit 12.

[0034] Next, the layer structure of the video synthesized by the video synthesis unit 232 of this embodiment will be described with reference to FIGS.

[0035] 4 shows the layer structure before composition in a conventional video see-through method. Reference numeral 41 denotes an image output to the image composition unit 232 after processing the real image captured by the imaging unit 11 in the image processing unit 331, and is placed at the bottom layer of the layer structure and used as a background image.

[0036] 42 is an image output to the calculation unit 34 after the real image acquired by the imaging unit 11 has been processed by the image processing unit 331, and is used as a background image when compositing with CG, and is positioned as an intermediate layer when viewed from the perspective of the entire system.

[0037] Here, if the images of a person's face contained in images 41 and 42 are moving from left to right in real time, the person's face will be at the right edge of the screen in image 41, but near the center of the screen in image 42. This is because there is little delay in transmission and processing of background image 41 in processing unit 33, so the person's face has already moved to the right side of the screen, and background image 42, which passes through calculation unit 34, handles a delayed image before the person's face has moved due to delays in transmission and processing. On the other hand, the clouds and sun, which are nearly stationary, will not differ in position even if there is a difference in delay time between images 41 and 42.

[0038] Reference numeral 43 denotes a CG image generated by the CG composition unit 341, and is located at the top of the layer structure. The CG image 43 generated by the calculation unit 34 is also a delayed image because the CG is rendered after calculating the position and orientation of the HMD. The parts other than the CG lightning image are made up of chromakey information for chromakey composition and alpha channel information for alpha composition.

[0039] What is output from the calculation unit 34 to the processing unit 33 is an image obtained by combining the background image 42 that has passed through the calculation unit 34 with the CG image 43 in the calculation unit 34. Because there is the background image 42 that has passed through the calculation unit 34, chromakey information or alpha channel information is not normally output, but as a means of image expression, it is conceivable to add this information to part of the image, or to control the transparency.

[0040] FIG. 5 shows the combined image in the conventional video see-through method.

[0041] Although all layers are synthesized in the image synthesis unit 232 in the processing unit 33, if the output from the calculation unit 34 does not include chroma information or alpha channel information, the real image presented to the HMD user will be the background image 42 that passes through the calculation unit 34 and includes a delay.

[0042] At this time, although the background video 41 in the processing section 33 with a small delay time is output from the video processing section 331 to the video synthesis section 232, it is not used as a result.

[0043] As a result of these processes, although there is no difference in delay time between the real image and the CG, the images displayed on the HMD both have a large delay, so the HMD user is presented with an image that is no different from that presented by the conventional video see-through method.In this explanation, the image 51 sent to the display unit using the conventional method is a CG lightning bolt superimposed on a real image of a person's face.

[0044] 6 shows the layer configuration before composition in a video see-through method in which only the real video has low delay. Video 41 and video 43 are the same as those in FIG. 4, so a description thereof will be omitted.

[0045] Reference numeral 62 denotes the converted image converted by the calculation unit 34, and as described above, it is an intermediate layer that renders the real image represented by the background image 42 that passes through the calculation unit 34. If no real image is acquired from the processing unit 33, or if real image is transmitted from the processing unit 33 to the calculation unit 34 but not used, the image 62 converted by the calculation unit 34 will be empty. It is also possible to transmit the real image from the processing unit 33 to the calculation unit 34, control the transparency of the real image, for example, by making part or all of the image transparent, and add this as alpha channel information. It is also possible to transmit the real image from the processing unit 33 to the calculation unit 34, convert part or all of the color information of the real image into colors representing chroma information, and add this as chroma information. By these means, the image 62 converted by the calculation unit 34 has part or all of the image converted, is composited with the CG image 43 in the calculation unit 34, and is then output from the calculation unit 34 to the processing unit 33.

[0046] Here, as described above, the image represented by the human face in image 41 moves from left to right in real time, while the clouds and sun remain stationary.

[0047] FIG. 7 shows the synthesized image in the video see-through method, which has little delay in the real image.

[0048] All layers are finally composited in the image composition unit 232 in the processing unit 33. If all of the images 62 modified by the calculation unit 34 are changed to be transparent, everything except the actual CG image is transparent, so the background image 41 in the processing unit 33 is composited as the background, and the actual image presented to the HMD user is the background image 41 in the processing unit 33 with little delay.

[0049] As a result of these processes, the delay time of the real image is small, but the image displayed on the HMD has a delay time difference between the real image and the CG, so the HMD user is presented with an image that is no different from that presented by a video see-through method in which only the real image has low delay.In this explanation, in the image 71 sent to the display unit using a method in which only the real image has low delay, the person's face, which is the real image, has little delay compared to real time and has already moved to the right edge of the screen, so the CG lightning is not superimposed on the person's face.

[0050] As explained above, in this embodiment, the image processing unit 331 outputs the real image to the image synthesis unit 232 and the CG synthesis unit 341, and the layer structure of the CG and real image is determined in advance. This makes it possible to control the delay time of the real image displayed on the HMD simply by changing the real image in the calculation unit 34, without affecting the configuration and control of the imaging unit 11, display unit 12, and processing unit 33.

[0051] (Second embodiment) 8 is a diagram showing the functional configuration of an image processing device according to the second embodiment. The flow of signal processing in image capture and display according to the second embodiment will be described with reference to FIG.

[0052] 8, the imaging unit 11, display unit 12, processing unit 33, and calculation unit 34 are the same as those in the first embodiment, and therefore their description will be omitted. In Fig. 8, in the control unit 85, a synthesis control unit 851 controls the synthesis method for the video synthesis unit 232 of the processing unit 33 and the CG synthesis unit 341 of the calculation unit 34.

[0053] In the first embodiment, it was possible to select between a conventional video see-through method in which there is no delay time difference between the real image and CG, or a method in which only the real image has low delay, simply by changing the real image in the calculation unit 34. In the second embodiment, by adding a control unit 85, it is possible to manually or automatically select either of the above methods.

[0054] If the HMD user wishes to view only low-latency real images, the synthesis control unit 851 issues an instruction to the image synthesis unit 232 to output only the background image 41 in the processing unit 33 shown in Figures 4 and 6 to the display unit 12. Alternatively, the image synthesis unit 232 may make images other than the background image 41 in the processing unit 33 transparent, or the CG synthesis unit 341 may make all images, including the CG, transparent.

[0055] Furthermore, if the HMD user wishes to view only the CG and fixed background image, the synthesis control unit 851 issues an instruction to the CG synthesis unit 341 to convert the image 62 changed by the calculation unit 34 into a fixed image, and only the CG and fixed background image are output to the display unit 12. Alternatively, even if the image synthesis unit 232 discards the background image 41 from the processing unit 33 without synthesizing it, only the CG and fixed background image are output to the display unit 12 in the same manner.

[0056] In this way, the control unit 85 can issue instructions to the processing unit 33 or the calculation unit 34, or the control unit 85 can issue instructions to the processing unit 33 and the calculation unit 34, depending on the user's wishes, thereby making it possible to control the images to be synthesized.

[0057] Furthermore, there may be cases where the state of the video signal communicated between the processing unit 33 and the calculation unit 34 becomes intermittent due to transmission or processing reasons, causing problems with comfortable viewing on the HMD. In this case, the synthesis control unit 851 issues an instruction to the video synthesis unit 232 to output only the background video 41 from the processing unit 33 to the display unit 12, making it possible to comfortably view only the low-latency real video.

[0058] Alternatively, background image 41 in processing unit 33 in Fig. 4 is compared with background image 42 that passes through calculation unit 34. As a result, if the delay time exceeds a certain set threshold, synthesis control unit 851 issues an instruction to CG synthesis unit 341 to change image 62 in calculation unit 34 as shown in Fig. 6, making it possible to view only real images using a video see-through method with low delay.

[0059] In this way, depending on the transmission and processing conditions, the control unit 85 can control the processing unit 33 or the calculation unit 34, or the control unit 85 can control the images to be synthesized by issuing instructions to the processing unit 33 and the calculation unit 34.

[0060] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0061] 11 Imaging unit 111 Objective optical system 112 Image Sensor 12 Display section 121 Display optical system 122 Display 33 Processing section 331 Video Processing Unit 232 Video Composition Unit 34 Arithmetic section 341 CG synthesis department

Claims

1. an acquisition means for acquiring a real image captured by the imaging unit and outputting the real image; a generating means for receiving the real image output by the acquiring means, generating and outputting a virtual image, and outputting the received real image or a converted image obtained by converting the real image; a synthesis means for synthesizing the real image output by the acquisition means, the virtual image output by the generation means, and the real image or the converted image output by the generation means; a display control means for displaying the image synthesized by the synthesis means on a display unit; An image processing device characterized in that the generation means changes a delay time of the real image in the image displayed on the display unit depending on whether the real image or the converted image is output.

2. The image processing device according to claim 1, characterized in that the synthesis means overlays and synthesizes, from above, the virtual image output by the generation means, the real image or converted image output by the generation means, and the real image output by the acquisition means.

3. The image processing device described in claim 1 or 2, characterized in that the generation means changes the delay time of the real image in the image displayed on the display unit depending on whether it outputs the real image or outputs an empty image without receiving or using the real image.

4. 3. The image processing apparatus according to claim 1, wherein the generating means outputs, as the converted image, an image obtained by converting the transparency of part or all of the received real image.

5. 5. The image processing device according to claim 4, wherein the generating means outputs, as the converted image, an image obtained by converting a part or all of the received real image into a transparent image.

6. 3. The image processing device according to claim 1, wherein the generating means outputs, as the converted image, an image obtained by converting the colors of part or all of the received real image.

7. 7. The image processing device according to claim 6, wherein the generating means outputs, as the converted image, an image in which the colors of part or all of the received real image are converted into colors representing chromakey information.

8. 8. The image processing device according to claim 1, wherein the generating means switches between outputting the real image and outputting the converted image based on an instruction from a user.

9. 8. The image processing device according to claim 1, further comprising a control means for controlling whether the generation means outputs the real image or the converted image based on the state of the image output from the generation means to the synthesis means.

10. 10. The image processing device according to claim 1, wherein the acquisition means adjusts the real image acquired from the imaging unit and outputs the adjusted image.

11. an acquisition step of acquiring a real image captured by the imaging unit and outputting the real image; a generating step of receiving the real image output in the acquiring step, generating and outputting a virtual image, and outputting the received real image or a converted image obtained by converting the real image; a synthesis step of synthesizing the real image output in the acquisition step, the virtual image output in the generation step, and the real image or the converted image output in the generation step; a display control step of displaying the image synthesized in the synthesis step on a display unit; An image processing method characterized in that, in the generation step, the delay time of the real image in the image displayed on the display unit is changed depending on whether the real image or the converted image is output.

12. 11. A program that causes a computer to function as each of the means of the image processing apparatus according to claim 1.

Images