Display terminal equipment

Abstract

A display terminal device, wherein a CPU determines the arrangement position of a virtual object in a real space through software processing, and outputs information indicating a first image and a position, the first image being an image of the virtual object; an imaging unit captures a second image, the second image being an image of the real space; a synthesizer combines the first image and the second image through hardware processing based on the position, thereby generating a synthesized image; and a display unit is directly connected to the synthesizer and displays the synthesized image.

Description

Technical Field

[0001] The present disclosure relates to a display terminal device. Background Art

[0002] Display terminal devices have been developed to implement services using augmented reality (AR) technology. Examples of display terminal devices include head mounted displays (HMDs). HMDs include, for example, optical see-through type HMDs and video see-through type HMDs.

[0003] In an optical see-through HMD, for example, a virtual image optical system using a half mirror or a transparent light guide plate is held in front of the user's eyes. An image is displayed inside the virtual image optical system. Therefore, a user wearing an optical see-through HMD can view the scenery around the user even when viewing an image displayed inside the virtual image optical system. Thus, an optical see-through HMD using AR technology is able to superimpose an image (hereinafter, may be referred to as a "virtual object") of a virtual object (hereinafter, may be referred to as a "virtual object image") on an optical image of an object existing in a real space in various modes (such as text, icons, and animations, etc.) according to the position and posture of the optical see-through HMD.

[0004] In contrast, a video see-through HMD is worn by a user to cover the user's eyes, and the display of the video see-through HMD is held in front of the user's eyes. In addition, the video see-through HMD includes a camera module for capturing a landscape image in front of the user, and the landscape image captured by the camera module is displayed on the display. Therefore, although it is difficult for a user wearing a video see-through HMD to directly view the landscape in front of the user, the user can see the landscape in front of the user using the image on the display. In addition, a video see-through HMD using AR technology can use an image of the landscape in front of the user as an image of the background in the real space (hereinafter referred to as the "background image") to superimpose a virtual object image on the background image according to the position and posture of the video see-through HMD. Hereinafter, the image obtained by superimposing the virtual object image on the background image may be referred to as a "synthetic image."

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: JP2018-517444A

[0008] Patent Document 2: JP2018-182511A Summary of the invention

[0009] Technical issues

[0010] Here, in the AR technology used in the video see-through HMD, the superposition of the virtual object image on the background image is performed by software processing, which relatively takes time, including analysis of the background image, etc. Therefore, in the video see-through HMD, the delay occurring between the time point when the background image has been captured and the time point when the composite image including the background image is displayed increases. In addition, the background image changes at any time as the video see-through HMD moves.

[0011] Thus, when the facial orientation of a user wearing a video see-through HMD changes, the updating speed of the background image on the display may sometimes fail to keep up with the changing speed of the user's facial orientation. Figure 1 As shown in FIG. 1 , when the facial orientation of the user wearing the video see-through HMD changes from orientation D1 to orientation D2, the background image BI captured when the orientation was D1 is sometimes displayed on the display even at the time point of the orientation D2. Therefore, at the time point when the facial orientation of the user reaches the orientation D2, the background image BI displayed on the display is different from the actual scenery FV in front of the user, thereby increasing the user's sense of discomfort.

[0012] In addition, among the background image and the virtual object image included in the composite image, the virtual object image is superimposed on the background image, and the background image changes as the video see-through HMD moves as described above. Therefore, when the video see-through HMD moves, it is difficult for the user to recognize the delay between the time point when the background image has been captured and the time point when the virtual object image to be superimposed on the background image is displayed or updated, while the user can easily recognize the delay in updating the background image. That is, the user is not sensitive to the display delay of the virtual object image, but is sensitive to the update delay of the background image. Thus, the increased update delay of the background image increases the user's sense of disharmony.

[0013] Therefore, the present disclosure proposes a technology capable of reducing the sense of discomfort of a user wearing a display terminal device such as a video see-through type HMD adopting AR technology.

[0014] Solution to the problem

[0015] According to the present disclosure, a display terminal device includes a CPU, an imaging unit, a synthesizer, and a display. The CPU determines the arrangement position of a virtual object in a real space through software processing, and outputs a first image and information indicating the arrangement position, the first image being an image of the virtual object. The imaging unit captures a second image, which is an image of the real space. The synthesizer combines the first image and the second image through hardware processing based on the arrangement position, thereby generating a synthesized image. The display is directly connected to the synthesizer and displays the synthesized image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The problem of the present disclosure is illustrated.

[0017] Figure 2 A configuration example of a display terminal device according to an embodiment of the present disclosure is illustrated.

[0018] Figure 3 An example of a processing procedure in a display terminal device according to an embodiment of the present disclosure is illustrated.

[0019] Figure 4 An image synthesis process according to an embodiment of the present disclosure is illustrated.

[0020] Figure 5 An image synthesis process according to an embodiment of the present disclosure is illustrated.

[0021] Figure 6 The effect of the technology of the present disclosure is illustrated. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. Note that in the following embodiments, the same reference numerals are attached to the same parts or the same processes to omit repeated descriptions.

[0023] Furthermore, the technology of the present disclosure will be described in the following order of items.

[0024] <Display terminal device configuration>

[0025] <Displaying the processing process in the terminal device>

[0026] <Image synthesis processing>

[0027] <Effects of the Disclosed Technology>

[0028] <Display terminal device configuration>

[0029] Figure 2 The figure shows an example of a configuration of a display terminal device according to an embodiment of the present disclosure. Figure 2 In the embodiment, the display terminal device 1 includes a camera module 10, a central processing unit (CPU) 20, a display 30, a sensor module 40, and a memory 50. The camera module 10 includes an imaging unit 11, a memory 12, and a synthesizer 13. The display terminal device 1 is worn by a user of the display terminal device 1 to cover the user's eyes. Examples of the display terminal device 1 include a video see-through HMD and smart devices such as a smartphone and a tablet terminal. When the display terminal device 1 is a smart device, the smart device is worn by the user of the smart device, and the smart device has a head-mounted instrument for the smart device to cover the user's eyes.

[0030] The camera module 10 includes lines L1, L2, L3, and L4. The imaging unit 11 is connected to the CPU 20 via the line L1, and is connected to the synthesizer 13 via the line L4. The memory 12 is connected to the CPU 20 via the line L3. The synthesizer 13 is connected to the display 30 via the line L2.

[0031] The imaging unit 11 includes a lens unit and an image sensor. The imaging unit 11 captures an image of the scenery in front of the user wearing the display terminal device 1 so that the user's eyes are covered by the display terminal device 1 as a background image. The imaging unit 11 outputs the captured background image to the synthesizer 13 and the CPU 20. The imaging unit 11 captures the background image at a predetermined frame rate. The imaging unit 11 outputs the same background image captured at the same time point to the synthesizer 13 via the line L4 on the one hand, and outputs the same background image to the CPU 20 via the line L1 on the other hand. That is, the camera module 10 includes the line L1, and the background image captured by the camera module 10 is output from the camera module 10 to the CPU 20 through the line L1.

[0032] The sensor module 40 detects the acceleration and angular velocity of the display terminal device 1 to detect changes in the position and posture of the display terminal device 1, and outputs information indicating the detected acceleration and angular velocity (hereinafter, may be referred to as "sensor information") to the CPU 20. Examples of the sensor module 40 include an inertial measurement unit (IMU).

[0033] The CPU 20 performs synchronous localization and mapping (SLAM) based on the background image and sensor information at a predetermined cycle. That is, the CPU 20 generates an environment map and a posture graph in SLAM based on the background image and sensor information. The CPU 20 uses the environment map to identify the real space where the display terminal device 1 exists. The CPU 20 uses the posture graph to identify the position and posture of the display terminal device 1 in the identified real space. In addition, the CPU 20 determines the layout position of the virtual object in the real space based on the generated environment map and posture graph, that is, the layout position of the virtual object image in the background image (hereinafter, may be referred to as "virtual object layout position"). The CPU 20 outputs information indicating the determined virtual object layout position to the memory 12 in association with the virtual object image (hereinafter, may be referred to as "layout position information"). The CPU 20 outputs the virtual object image and the layout position information to the memory 12 via the line L3.

[0034] The memory 50 stores applications executed by the CPU 20 and data used by the CPU 20. For example, the memory 50 stores data on virtual objects (e.g., data for reproducing the shape and color of virtual objects). The CPU 20 generates a virtual object image by using the data on virtual objects stored in the memory 50.

[0035] The memory 12 stores the virtual object image and arrangement position information input from the CPU 20 in a predetermined cycle for a predetermined time.

[0036] The synthesizer 13 generates a synthesized image by superimposing the virtual object image on the background image based on the virtual object image stored in the memory 12 and the latest virtual object image and arrangement position information among the plurality of arrangement position information. That is, the synthesizer 13 generates a synthesized image by superimposing the latest virtual object image on the latest background image input from the imaging unit 11 at the position indicated by the arrangement position information. The synthesizer 13 outputs the generated synthesized image to the display 30 via the line L2. That is, the camera module 10 includes the line L2, and the synthesized image generated by the camera module 10 is output from the camera module 10 to the display 30 through the line L2.

[0037] The synthesizer 13 is implemented as hardware and is implemented by, for example, an electronic circuit created by wired logic. That is, the synthesizer 13 generates a synthesized image by combining a background image and a virtual object image through hardware processing. In addition, the synthesizer 13 and the display 30 are directly connected to each other through hardware via line L2.

[0038] The display 30 displays the synthesized image input from the synthesizer 13. This causes a synthesized image obtained by superimposing a virtual object image on a background image to be displayed in front of the eyes of the user wearing the display terminal device 1.

[0039] Here, both the camera module 10 and the display 30 conform to the same interface standard, for example, the Mobile Industry Processor Interface (MIPI) standard. When both the camera module 10 and the display 30 conform to the MIPI standard, the background image captured by the imaging unit 11 is serially transmitted to the synthesizer 13 through a camera serial interface (CSI) according to the MIPI standard. The synthesized image generated by the synthesizer 13 is serially transmitted to the display 30 through a display serial interface (DSI) according to the MIPI standard.

[0040] <Displaying the processing process in the terminal device>

[0041] Figure 3 An example of a processing procedure in a display terminal device according to an embodiment of the present disclosure is illustrated.

[0042] Figure 3 The camera module driver, sensor module driver, SLAM application, and AR application in FIG. 5 are software stored in the memory 50 and executed by the CPU 20. In contrast, the camera module 10, the sensor module 40, and the display 30 are hardware. Figure 3 The camera module driver in is a driver of the camera module 10 . Figure 3 The sensor module driver in is the driver of the sensor module 40.

[0043] exist Figure 3 In the present invention, in step S101, the camera module 10 outputs a background image to the CPU 20. In step S103, the background image input to the CPU 20 is transferred to the SLAM application via the camera module driver.

[0044] Furthermore, in parallel with the processing in step S101, in step S105 the sensor module 40 outputs the sensor information to the CPU 20. In step S107, the sensor information input to the CPU 20 is passed to the SLAM application via the sensor module driver.

[0045] Then, in step S109 , the SLAM application performs SLAM based on the background image and the sensor information to generate an environment map and a pose graph in SLAM.

[0046] Then, in step S111 , the SLAM application transmits the environment map and the posture graph generated in step S109 to the AR application.

[0047] Then, in step S113 , the AR application determines the virtual object arrangement position based on the environment map and the gesture graph.

[0048] Then, in step S115, the AR application outputs the virtual object image and the arrangement position information to the camera module 10. The virtual object image and the arrangement position information input to the camera module 10 are associated with each other and stored in the memory 12.

[0049] In step S117 , the camera module 10 generates a synthesized image by superimposing the virtual object image on the background image based on the virtual object image stored in the memory 12 and the latest virtual object image and arrangement position information among the plurality of pieces of arrangement position information.

[0050] Then, in step S119 , the camera module 10 outputs the composite image generated in step S117 to the display 30 .

[0051] Then, in step S121 , the display 30 displays the synthesized image input in step S119 .

[0052] <Image synthesis processing>

[0053] Figure 4 and Figure 5 An image synthesis process according to an embodiment of the present disclosure is shown.

[0054] like Figure 4As shown, the synthesizer 13 generates the synthesized image CI by superimposing the virtual object image VI on the background image BI for each line in the horizontal direction (row direction) of the background image BI of each frame.

[0055] For example, the imaging unit 11, the synthesizer 13 and the display 30 are as follows: Figure 5 The operation is based on the vertical synchronization signal vsync and the horizontal synchronization signal hsync. Figure 5 In , "vsync+1" indicates a vertical synchronization signal inputted after the vertical synchronization signal vsync0, and "vsync-1" indicates a vertical synchronization signal inputted before the vertical synchronization signal vsync0. In addition, Figure 5 As an example, a case where five horizontal synchronization signals hsync are input and one vertical synchronization signal vsync is input is shown.

[0056] exist Figure 5 In the embodiment, the imaging unit 11 outputs the YUV data for each line of the background image BI (one line of YUV) to the synthesizer 13 according to the horizontal synchronization signal hsync.

[0057] The synthesizer 13 converts the YUV data input from the imaging unit 11 into RGB data. In addition, the synthesizer 13 superimposes the RGB data (VI RGB) of the virtual object image VI on the RGB data of the background image BI for each line according to the horizontal synchronization signal hsync and the arrangement position information. Thus, in the line where the virtual object image VI exists, the RGB data of the synthesized image (synthesized RGB) is output from the synthesizer 13 to the display 30 and displayed. In the line where the virtual object image VI does not exist (no image), the RGB data of the background image BI (one line of RGB) is output from the synthesizer 13 to the display 30 as it is and displayed.

[0058] The embodiments of the technology of the present disclosure have been described above.

[0059] Notice, Figure 2 The configuration of the display terminal device 1 is illustrated. In this configuration, the camera module 10 includes a memory 12 and a synthesizer 13. However, the display terminal device 1 may also adopt a configuration in which one or both of the memory 12 and the synthesizer 13 are provided outside the camera module 10.

[0060] <Effects of the Disclosed Technology>

[0061] As described above, the display terminal device according to the present disclosure (the display terminal device 1 according to the embodiment) includes a CPU (the CPU 20 according to the embodiment), an imaging unit (the imaging unit 11 according to the embodiment), a synthesizer (the synthesizer 13 according to the embodiment) and a display (the display 30 according to the embodiment). The CPU determines the arrangement position of the virtual object in the real space (the virtual object arrangement position according to the embodiment) through software processing, and outputs a first image (the virtual object image according to the embodiment) as an image of the virtual object and information indicating the arrangement position (arrangement position information according to the embodiment). The imaging unit captures a second image (the background image according to the embodiment), which is an image of the real space. The synthesizer generates a synthesized image by combining the first image and the second image through hardware processing based on the arrangement position. The display is directly connected to the synthesizer and displays the synthesized image.

[0062] For example, a camera module including an imaging unit and a synthesizer includes a first line (line L1 according to an embodiment) and a second line (line L2 according to an embodiment). A first image is output from the camera module to a CPU via the first line. A synthesized image is output from the camera module to a display via the second line.

[0063] Furthermore, for example, the synthesizer combines the first image and the second image for each line in the horizontal direction of the second image.

[0064] Furthermore, for example, both the camera module and the display comply with the MIPI standard.

[0065] Furthermore, for example, the CPU generates an environment map and a posture graph by performing SLAM based on the second image, and determines the arrangement position based on the environment map and the posture graph.

[0066] According to the above configuration, the background image captured by the imaging unit is output to a display directly connected to the synthesizer without undergoing software processing performed by the CPU, so that the background image is displayed on the display immediately after being captured by the imaging unit. Therefore, the delay that occurs between the point in time when the background image has been captured and the point in time when the synthesized image including the background image is displayed can be reduced. Therefore, when the facial orientation of a user wearing a display terminal device according to the present disclosure changes, the background image on the display can be updated to follow the change in the facial orientation of the user. Thus, for example, Figure 6As shown, when the facial orientation of the user wearing the display terminal device according to the present disclosure changes from orientation D1 to orientation D2, the background image BI captured when the user's facial orientation reaches orientation D2 is displayed on the display when it is in orientation D2. Therefore, at the time point when the user's facial orientation reaches orientation D2, the difference between the background image BI displayed on the display and the actual scenery FV in front of the user is reduced to a level that the user finds it difficult to recognize the difference. Thus, according to the above configuration, the sense of disharmony of the user wearing the display terminal device can be reduced.

[0067] Note that the effects described in the specification are merely examples and not limitations. Other effects may be exhibited.

[0068] Furthermore, the technology of the present disclosure may also adopt the following configurations.

[0069] (1) A display terminal device, comprising:

[0070] A CPU determines a layout position of a virtual object in a real space through software processing, and outputs a first image and information indicating the layout position, wherein the first image is an image of the virtual object;

[0071] an imaging unit that captures a second image, the second image being an image of a real space;

[0072] a synthesizer that combines the first image and the second image through hardware processing based on the arrangement position, thereby generating a synthesized image; and

[0073] A display directly connected to the synthesizer and displaying the synthesized image.

[0074] (2) The display terminal device according to (1), further comprising

[0075] a camera module comprising the imaging unit and the synthesizer,

[0076] The camera module includes: a first circuit, through which the first image is output from the camera module to the CPU; and a second circuit, through which the composite image is output from the camera module to the display.

[0077] (3) The display terminal device according to (1) or (2),

[0078] The synthesizer combines the first image and the second image for each line in the horizontal direction of the second image.

[0079] (4) The display terminal device according to (2),

[0080] Among them, both the camera module and the display comply with the MIPI standard.

[0081] (5) The display terminal device according to any one of (1) to (4),

[0082] The CPU generates an environment map and a posture graph by performing SLAM based on the second image, and determines the arrangement position based on the environment map and the posture graph.

[0083] Reference numerals list

[0084] 1Display terminal equipment

[0085] 10Camera Module

[0086] 11 Imaging Unit

[0087] 13 Synthesizer

[0088] 20CPU

[0089] 30 Displays

[0090] 40 sensor modules

Claims

1. A display terminal device, comprising: Memory; an imaging unit that captures a second image, the second image being an image of a real space; A CPU, which determines a layout position of a virtual object in a real space through software processing according to the second image, and outputs a first image and information indicating the layout position to the memory, wherein the first image is an image of the virtual object; a synthesizer that combines the first image and the latest second image through hardware processing based on the latest information stored in the memory, thereby generating a synthesized image; and a display directly connected to the synthesizer and displaying the synthesized image, wherein the imaging unit is connected to the CPU via a first line and to the synthesizer via a fourth line, and outputs the same second image captured at the same time point to the CPU via the first line and to the synthesizer via the fourth line, The synthesizer as hardware is directly connected to the display as hardware via a second line, the second image captured by the imaging unit is output to the display directly connected to the synthesizer via the second line without implementing software processing based on the CPU, The CPU generates an environment map and a posture graph by performing SLAM based on the second image, and determines the arrangement position based on the environment map and the posture graph.

2. The display terminal device according to claim 1, in, The synthesizer combines the first image and the second image for each line in the horizontal direction of the second image.

3. The display terminal device according to claim 1, in, Both the camera module and the display comply with the MIPI standard.

Images