Display terminal

Abstract

To provide a virtual object display terminal configured to realize high operability regardless of a display position of a virtual object, while maintaining a sense of reality.SOLUTION: An HMD (head-mounted display) 200, which is to be worn on the head of a user (wearer) 201, as a display terminal includes: a color image camera and a distance image camera for acquiring a color image and a distance image of a predetermined imaging range, respectively; and a display control unit. The display control unit includes: a space recognition unit which uses the color image and the distance image to generate a three-dimensional map of a structural object 300; a display data generation unit for generating display data in which a rear region of a virtual object 100 behind the structural object 300 in a line-of-sight direction is specified, based on the three-dimensional map and real-space placement position data of the virtual object 100; and a display correction unit for correcting the display data so as to display operation points 110 of the rear region, which accept an operation instruction on the virtual object 100.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for displaying virtual objects on a display terminal. [Background technology]

[0002] HMD (Head Mounted Display) It is an augmented reality technology that adds computer-generated information to real-world objects and displays them on display terminals such as Technology to improve visibility of Augmented Reality (AR) images For example, in Patent Document 1, "a predetermined image is superimposed on the external world that is visible through the display unit." a superimposed image display control unit that displays a predetermined image; and a predetermined image of the outside world that is visible through the superimposed image display control unit. an imaging unit that captures at least an image of the range, and a partial image specifying unit for specifying a partial image within a predetermined range whose position corresponds to the image; A predetermined image is displayed by the superimposed image display control unit according to the color information of the partial image. and a visibility correction unit that corrects the appearance of the display. (Summary excerpt) A head-mounted display device including a display unit is disclosed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-142887 Summary of the Invention [Problem to be solved by the invention]

[0004] Virtual objects displayed in AR images are displayed around the virtual objects in advance. By giving instructions to the operation points, operations such as movement, rotation, and transformation can be performed. However, depending on the display position of the virtual object, a part of the virtual object may be The hidden surface is removed and the necessary operation points are displayed. Therefore, high operability is not necessarily achieved. If the virtual object is displayed as it is, regardless of the placement position after the operation, a sense of reality will not be obtained. I can't.

[0005] The present invention has been made in consideration of the above circumstances, and provides a method for displaying a virtual object in a virtual environment, regardless of the display position of the virtual object. The aim is to provide a virtual object display technology that maintains a sense of realism while achieving high operability. The target. [Means for solving the problem]

[0006] The present invention is a display terminal, which has a display and a color image capturing device for capturing a color image of a predetermined photographing range. a color image camera for capturing a distance image of the photographing range; and a control unit for controlling the color image and the distance image to obtain an image within the shooting range. A three-dimensional map of the structure is generated, and the three-dimensional map and the placement position data of the virtual object are compared. Based on the data, the virtual object is displayed in the line of sight direction of the user of the display terminal. The area behind the structure is identified as a rear area, and the virtual object is displayed on the display. The control unit controls the display to display an object in front of the display. The display mode when displaying the virtual object includes: a first display mode in which only an area excluding the back area of the virtual object is displayed; and displaying an additional object around the virtual object. The display mode is characterized by having a second display mode. [Effects of the Invention]

[0007] According to the present invention, it is possible to maintain a sense of reality and achieve high operability regardless of the display position of a virtual object. It is possible to provide a virtual object display technology that realizes the above-mentioned problems, configurations, and The effects will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0008] [Figure 1] 1A to 1C are explanatory diagrams illustrating an overview of a virtual object display process according to an embodiment of the present invention. [Figure 2] FIG. 1A is a hardware configuration diagram of an HMD according to an embodiment of the present invention, and FIG. 1B is an external view of the HMD according to an embodiment of the present invention. [Figure 3] FIG. 2 is a functional block diagram of a controller of the HMD according to the embodiment of the present invention. [Figure 4] 1A and 1B are explanatory diagrams illustrating an overview of a virtual object display process according to an embodiment of the present invention. [Figure 5] 10 is a flowchart of a virtual object display process according to an embodiment of the present invention. [Figure 6] 1A and 1B are explanatory diagrams for explaining a display according to an embodiment of the present invention. [Figure 7] 1A to 1C are explanatory diagrams illustrating an example of a virtual object display mode according to an embodiment of the present invention. [Figure 8] 10(a) to 10(c) are explanatory diagrams illustrating other examples of virtual object display aspects according to an embodiment of the present invention. [Figure 9] 10(a) and 10(b) are explanatory diagrams illustrating a first modified example of a virtual object display mode according to an embodiment of the present invention. [Figure 10] FIG. 10 is an explanatory diagram for explaining a second modified example of a virtual object display mode according to the embodiment of the present invention. [Figure 11] FIG. 10 is an explanatory diagram for explaining a third modified example of a virtual object display mode according to the embodiment of the present invention. [Figure 12] FIG. 10 is an explanatory diagram for explaining a fourth modified example of a virtual object display mode according to an embodiment of the present invention. [Figure 13] FIG. 10 is an explanatory diagram illustrating a fifth modified example of a virtual object display mode according to an embodiment of the present invention. [Figure 14] FIG. 10 is an explanatory diagram for explaining a sixth modified example of a virtual object display mode according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. , unless otherwise specified, the same reference numerals are used to denote components having the same functions, and repeated explanations are omitted. It should be noted that the present invention is not limited to the embodiments described herein.

[0010] In this embodiment, the display terminal is an HMD ( A head mounted display, It is a see-through type that has a transparent display and allows both the outside world and the displayed image to be seen. ) HMD as an example.

[0011] First, an outline of this embodiment will be described with reference to FIGS. 1(a) to 1(c). As shown in FIG. 1(a), a user 201 wears an HMD 200 indoors and experiences a virtual The object 100 is displayed on the display of the HMD 200 and can be operated.

[0012] The HMD 200 has distance information (depth information) of the real space and the virtual object 100. Conventionally, the display of the HMD 200 has been designed to reflect the real world, such as walls, as shown in Figure 1(b). The portion of the virtual object 100 located behind the structure 300 is not displayed. In such a case, the user 201 cannot grasp the entire picture of the virtual object 100. cannot be done.

[0013] Generally, the virtual object 100 is located on or near the virtual object 100. By operating the operation point (transform controller) 110 set in Therefore, when the image is displayed as shown in Figure 1(b), The operation point 110 behind the structure 300 is not displayed and therefore cannot be operated.

[0014] In this embodiment, in order to solve this problem, as shown in FIG. 1(c), a structure 300 is provided. Even if the virtual object 100 is moved, all of the operation points 110 are displayed. The virtual object 100 itself may also be displayed in its entirety. The additional object 400 is displayed so that the hidden part can be seen. It may be shown.

[0015] The HMD 200 of this embodiment that realizes such display control will be described below.

[0016] [Hardware configuration diagram] FIG. 2(a) is a hardware configuration diagram of the HMD 200. FIG. 2(b) is a hardware configuration diagram of the HMD 200. FIG. 2 is an external view of an HMD 200 according to the embodiment.

[0017] The HMD 200 of this embodiment is basically the same as a general-purpose computer (information processing device). That is, as shown in the figure, the HMD 200 has a controller 210 and Camera 214, display 215, audio interface (I / F) 216, and communication I / F 217, sensor 218, bus 219 electrically connecting each part, and gaze detection device Furthermore, the HMD 200 of this embodiment includes a support for each part and a user 20 1 is provided with a frame 241 for mounting.

[0018] The controller 210 performs various processes according to a predetermined program. In this state, the controller 210 may, for example, display a virtual object at a predetermined position on the display 215. Display Project 100.

[0019] The controller 210 of this embodiment includes a CPU 211, a RAM 212, and a ROM 213. The CPU 211 executes a program stored in the ROM 213 in advance. By loading the program into RAM 212 and executing it, various functions are realized. When there is no need to distinguish between the ROM 213 and the memory device 230 (see FIG. 3). The controller 210 is, for example, arranged on a frame 241. .

[0020] The camera 214 includes a color image camera 214a and a distance image camera 214b. The color image camera 214a captures an image of a range including the visual field of the user 201, and The distance image camera 214b acquires a color image. The camera 214 (color image camera 214a and The distance image camera 214b is located at the front of the frame 241 (closest to the display 21). 5) in a position that allows the above-mentioned shooting range to be photographed.

[0021] The display 215 displays the image acquired by the camera 214 and the display generated within the HMD 200. The display 215 is a device on which display data is displayed. The display 215 is, for example, a transmissive liquid crystal display. devices, organic EL devices, or MEMS (micro electro mechanical It is composed of optical scanning devices using optical systems. However, the device is not limited to this, and while displaying an image on the display 215, A device that can realize a transparent display structure that allows you to see through Play215 That would be good.

[0022] In the HMD 200 of this embodiment, a transmissive display 215 is provided for one eye of the user 201. Alternatively, the display 215 may be held in front of both eyes. The display 215 may include right and left display panels. Ray 215 displays one or more UI objects of the graphical user interface. This may be done.

[0023] The audio I / F 216 is an audio output device such as a microphone, a speaker, a buzzer, etc. The voice I / F 216 inputs external sounds and communicates with the sound created within the HMD 200. In this embodiment, the audio I / F 2 outputs sounds such as voice and music transmitted via the audio I / F 2. 16 may not be provided.

[0024] The communication I / F 217 is equipped with an encoding circuit, a decoding circuit, an antenna, etc., and transmits data via a network. In this embodiment, the communication I / F 217 transmits and receives data to and from other devices (data communication). The communication is performed via an access point (not shown) or a mobile telephone (not shown). This is an interface that connects to the network via a base station of a communication network. The HMD 200 then transmits and receives data to and from each server connected to the network.

[0025] The connection between the HMD 200 and the access point is, for example, Wi-Fi (registered trademark) or the like. The communication between the HMD 200 and the mobile phone is performed by the wireless communication method of the above or other communication methods. The connection to the base station of the network is, for example, W-CDMA (registered trademark) e Division Multiple Access) system and GSM (registered trademark) Global System for Mobile communications) system, LTE (Long Term Evolution) system, or other communication systems The HMD 200 of this embodiment does not include the communication I / F 217. That's fine.

[0026] The sensor 218 detects the current position, tilt, speed, and operation by the user 201 of the HMD 200. The HMD 200 detects the position of a GPS receiver 218a, etc. as a sensor 218. Position information acquisition sensor, gyro sensor 218b, acceleration sensor 218c, geomagnetic sensor 21 8d, touch sensor 218e, etc. Note that it is not necessary to provide all of the sensors 218. stomach.

[0027] The gaze detection device 214c detects the gaze direction of the user 201. This is realized, for example, by a gaze detection camera that detects the gaze direction of the user 201. The camera is attached so that the iris, pupil, etc. of the user's 201 eye are included in the photographing range.

[0028] The frame 241 includes a HM such as a display 215, a camera 214, and a controller 210. Supports D200 components.

[0029] [Function Block] Next, the controller 210 of this embodiment realizes the following related to the display of virtual objects. FIG. 3 shows the functions of the HMD 200 according to the present embodiment, which are related to the virtual object display process. As shown in this figure, the controller 21 of this embodiment 0 performs the functions of an image acquisition unit 228, a display control unit 220, and an audio output control unit 229. The display control unit 220 also includes a space recognition unit 221, an instruction receiving unit 222, and a display device. The display correction unit 224 includes a data generation unit 223 and a display correction unit 224 .

[0030] Each function is executed by the CPU 211, which loads a program stored in the ROM 213 into the RAM 212. This is achieved by downloading and running the program.

[0031] The storage device 230 also stores color image data 231, distance image data 232, and space image data. Interaction recognition data 233, virtual object data (virtual OJT data) 234, and additional object Object data (additional OJT data) 235 and voice data 236 are stored.

[0032] The color image data 231 is an image acquired by the color image camera 214a. The image data 232 is an image captured by the distance image camera 214b.

[0033] The image acquisition unit 228 acquires the image using a color image camera 214a and a distance image camera 214b. The acquired color image and distance image are respectively referred to as color image data 231 and distance image data. In this embodiment, the color image and the distance image are stored in the storage device 230 as the data 232. The images are acquired approximately synchronously.

[0034] The space recognition unit 221 recognizes the surrounding real space and outputs the result as space recognition data 233. The image data is then stored in the storage device 230. The image data is then recognized by the color image data 231 and This is done using the distance image data 232.

[0035] The space recognition unit 221 recognizes each image at predetermined time intervals according to the scanning action of the user 201. From the image data, a spatial image, which is three-dimensional data (three-dimensional map) of the structure 300 in the imaging range, is obtained. The recognition data 233 is generated and stored in the storage device 230. The surroundings scan may be performed, for example, Immediately after startup, the user 201 performs the initial settings.

[0036] The spatial recognition data 233 is created, for example, in a world coordinate system that defines the entire three-dimensional space. For example, the origin and axis directions of this world coordinate system are set as the starting point of spatial recognition. The HMD200 is determined by the position and orientation (initial posture) of the HMD200 when it receives a command. The origin and each axis direction of the local coordinate system of MD200 are used. In this coordinate system, for example, In the initial position of the HMD 200 body, a predetermined position on the display 215 of the HMD 200 The origin is the display 215, the display 215 is the xy plane, and the z-axis direction is the xy plane (display The direction is perpendicular to the playing surface (215).

[0037] The world coordinate system of the HMD 200 based on the scanning action of the user 201 The amount of displacement and rotation relative to the coordinate system is calculated using data obtained by various sensors 218. is calculated.

[0038] Spatial recognition can be achieved using existing technologies such as spatial mapping. That is, the HMD 200 of this embodiment uses the color image camera 214a and the distance image camera 214b. The surroundings are scanned by the image camera 214b. The results are then used to 21, but using applications such as Spatial Mapping to convert 3D data The space recognition data 233 is stored as mesh data, for example.

[0039] In addition to the three-dimensional data, Spatial analysis is also performed to recognize the type of structure 300. The spatial recognition unit 221 may be configured to simultaneously perform spatial understanding. By using Spatial Understanding, the system can identify 300 structures within the shooting range. That is, the structure 300 can be recognized as a "wall," a "floor," a "ceiling," etc. The space recognition unit 221 of this embodiment can recognize whether the space is These recognition results are stored in the storage device 230 as attribute data of the recognition data 233 .

[0040] The instruction receiving unit 222 receives an instruction from the user 201 to The display instructions and operation instructions for the object 100 are received. For example, there are methods based on gaze and gestures. .

[0041] The information on the line of sight used for the gaze is detected by, for example, the line of sight detection device 214c. can be.

[0042] The gesture may be, for example, a click on an operation point 110 of a virtual object 100. There are various events such as tap and hold, bloom, and so on. 22 is provided within the photographing range of the color image camera 214a and the distance image camera 214b. The movement of the fingers within the gesture frame is detected, and display instructions, operation instructions, etc. are detected.

[0043] For example, when a display instruction is received, the instruction receiving unit 222 receives the virtual object data 2 34 extracts data of the virtual object 100 designated by the display data generating unit 35, and 223 to generate display data.

[0044] Furthermore, when an operation instruction is received, the instruction receiving unit 222 detects the operation and The display data generating unit 223 is notified.

[0045] The display data generating unit 223 performs the following in accordance with an instruction from the user 201 via the instruction receiving unit 222: The virtual object 100 designated from the virtual object data 234 is displayed. The instruction receiving unit 222 generates display data to be displayed in a predetermined shape at a predetermined position on the display unit 215. The display data generated in accordance with the instruction is displayed on the display 215, thereby The object 100 is displayed so that it can be moved, rotated, and transformed according to the instructions of the user 201. is shown.

[0046] At this time, the display data generating unit 223 determines the direction of the user's line of sight (when the user 201 is wearing (calculated from the spatial coordinate position of the HMD200 and information on the up / down / left / right orientation) A display in which the area of the virtual object 100 behind the object 300 is identified as the back area 101 The rear area 101 (FIG. 4) is a 3D map generated by the space recognition unit 221. The position of the virtual object 100 to be displayed in the real space (space recognition data 233) The position data in the real space is specified based on the space recognition unit 2. 21 is registered in the same coordinate system as the coordinate system used for spatial recognition.

[0047] The placement position data of the virtual object 100 in the real space is used for the virtual object to be displayed. The virtual object data 234 of the object 100 is obtained. 4 includes size, shape, and initial placement position information of each virtual object 100. The placement position information is, for example, a placement position point set in advance for each virtual object 100. The placement position point is, for example, a virtual object. The operation point 110 is the three-dimensional center of gravity position of the object 100. This is the point at which an instruction to deform the display shape of the virtual object 100 is accepted.

[0048] The display data generating unit 223 reflects the instruction from the instruction receiving unit 222 in the current arrangement position information. By this, virtual object data 23 corresponding to the virtual object 100 to be displayed is The display data generating unit 223 then obtains the latest arrangement position information of the arrangement position point 4. The virtual object 100 is calculated using the latest position information of the virtual object, as well as information on its size, shape, etc. Obtain placement position data in real space.

[0049] The display data generation unit 223 further uses the latest placement position data in real space to The rear area 101 of the virtual object 100 to be displayed is specified. For example, The virtual object generation unit 223 generates the virtual object based on the placement position data of the virtual object 100 in the real space. The coordinate position of the object 100 in the space of the HMD 200 worn by the user 201, Distance to virtual object calculated from downward, leftward, and rightward direction information, virtual object 10 The shape data of the structure 300 is identified as depth information from the coordinate position of the structure 300.

[0050] Then, the virtual object located relatively further back than the depth information of the structure 300 is The area (part) of the object 100 and the operation point 110 is defined as a back area 101.

[0051] In addition, the back surface area 101 is an area that is subjected to hidden surface processing in normal processing. For example, in FIG. As shown in (a), a part of the virtual object 100 is behind a structure 300 such as a wall or furniture. When the virtual object 100 is placed, the display data generating unit 223 conventionally Hidden surface removal is performed on the area 101, and display data is generated so that it is not displayed.

[0052] However, in this embodiment, in order to aim for a more user-friendly display, display correction Even in such a case, the unit 224 can generate the virtual object 1 as shown in FIG. The display is corrected to show the entire 00 and the associated operation point 110. Such processing is performed by the display correction unit 224.

[0053] The display correction unit 224 corrects the virtual object data 234 to be displayed when the virtual object data 234 includes the rear area 101. In this case, the display data of the virtual object data 234 is corrected.

[0054] Specifically, the display correction unit 224 cancels hidden surface processing for the rear area 101. The rear area 101 is also displayed as if the structure 300 does not exist. That is, the display correction unit 224 corrects the virtual object data 234. Even if the area of the part is behind the structure 300, the display is Correct the data.

[0055] Furthermore, the display correction unit 224 corrects the virtual object 100 in the original display mode. To indicate that there is no additional object, an additional object 400 is displayed. For example, the virtual object 100 may be seen breaking through the structure 300, The virtual object 100 is displayed more naturally and realistically, such as showing a hole in 00. It is an object that makes it appear as though there is something there.

[0056] The data of the additional object 400 to be displayed is stored in the additional object data of the storage device 230. The additional object data 235 is prepared in advance as the additional object data 235. 400, the display mode, initial size, and initial display position relative to the display position of the virtual object 100 The information such as the location is registered in association with the information.

[0057] The display correction unit 224 corrects the virtual object 100 by the operation of the user 201. When the display direction relative to the line of sight of the user 201 changes, the additional object The display shape of the object 400 may be changed. Similarly, when the display direction of the virtual object 100 is changed, the additional object 400 The additional object 400 may be deformed by, for example, changing the line of sight. The same program as the program used to process the transformation of the display of the virtual object 100 according to the transformation Follow Ram.

[0058] Next, the display process of the virtual object 100 by the controller 210 of this embodiment will be described. 5 is a processing flow of the virtual object display processing of this embodiment. It is assumed that the spatial recognition process has already been performed as an initial process. The object 100 is displayed in a position where the entire object can be displayed.

[0059] The display data is kept up until an end instruction is received from the user 201 via the instruction receiving unit 222. The generation unit 223 and the display correction unit 224 repeat the following process (step S1101). .

[0060] A user 201 inputs a movement operation for the virtual object 100 via an instruction receiving unit 222. When the operation is received (step S1102), the display data generating unit 223 generates display data. (step S1103).

[0061] In step S1103, the display data generating unit 223 first 00 on the display 215. Then, the display data generating unit 223 calculates the display position of , the gaze direction is identified, and the corresponding virtual object data 234 of the virtual object 100 is The depth information of the structure 300 is also calculated. The display data generation unit 223 identifies the rear area 101 on the display data.

[0062] Next, the display data generation unit 223 generates a back area 101 for the virtual object 100 to be displayed. It is determined whether or not there is (step S1104).

[0063] If there is no rear area 101 (S1104; No), the display data generating unit 223 notifies the user of this fact. The display correction unit 224 then notifies the display correction unit 224. 215, the display device of the virtual object 100 is set at the position calculated in step S1103. Then, the controller 210 displays the next operation instruction (step S1112). wait.

[0064] If the rear area 101 exists (S1104; Yes), the display data generating unit 223 It is determined whether the entire object 100 is the rear area 101 (step S110 5).

[0065] If the entire area is the rear area 101 (S1105; Yes), the display data generating unit 223 The display correction unit 224 is then notified of this. The display of the virtual object 100 is erased (step S1111), and the process ends.

[0066] On the other hand, if the entire area is not the back area 101 (S1105; No), that is, if the entire area is not the back area 101, the virtual object If the back area 101 is included in a part of the object 100, the display data generating unit 223 notifies the back area 101 of this fact. The display correction unit 224 then corrects the virtual object using the above-mentioned method. The display data of the object 100 is corrected (step S1106). 4 is a diagram showing all the operation points 110 of the virtual object 100 and the operation point 11 Correct the display data so that the portion of the virtual object 100 operated by 0 is displayed. Then, the display correction unit 224 displays the corrected display data on the display 215. It is displayed at the position calculated in S1103 (step S1107).

[0067] After that, the display correction unit 224 retrieves the additional object data 235 from the storage device 230. The obtained image is displayed superimposed on the rear area 101 of the virtual object 100 (step S110 8) Then, the controller 210 waits for the next operation instruction.

[0068] The above processing will be explained using a specific example. The display 215 displays the following as shown in FIG. In this way, the virtual object 100 and the virtual object 101 are displayed within a frame 215a defined by the display 215. The structure 300 is a real object in the real space, and an additional object 400 is displayed. In the following description of the display mode, the frame of the display 215 will be omitted, and only the display 215 shown in FIG. As shown in FIG. 1, the real object and the virtual object are viewed through the display 215. Only the display data such as 100 will be described and explained.

[0069] For example, suppose that a virtual object 100 is displayed at the position shown in FIG. 7(a). In response to this, a user 201 moves a finger 202 in the direction of an arrow 203 in real space. By continuing this operation, the virtual object The display position of the object 100 relative to the structure 300, which is a real object in the real space, moves. .

[0070] According to this embodiment, as shown in FIG. 7(b), a partial area of the virtual object 100 Even if the display position of the virtual object 100 is behind the structure 300, the virtual object 100 The whole image is displayed together with the object 110. At this time, it is as if the structure 300 The additional object 400 is displayed as if a hole had been made in the wall.

[0071] Furthermore, at this time, as shown in FIG. 7(c), the audio output control unit 229 outputs audio. When the display correction unit 224 displays the additional object 400, The audio output control unit 229 then notifies the display correction unit 224 When receiving a notification from the audio interface 216, the audio is extracted from the audio data 236 and output from the audio interface 216. do.

[0072] In addition, different data are registered in the voice data 236 in association with the material of the structure 300. The material of the structure 300 may be determined by, for example, the Spatial Recognition Unit 221 described above. Recognize through tial understanding.

[0073] Furthermore, at this time, even if onomatopoeia expressing the output voice data 236 as a character string is displayed, For example, the sound "blink" shown in FIG. 7(c) is used. In this case, the storage device 230 stores the sound. Onomatopoeia data 237 that corresponds to the voice data 236 and expresses the voice data 236 as a character string The display correction unit 224 then registers the audio data 236 to be output. The onomatopoeia data 237 registered in association with the additional object 400 is displayed near the additional object 400. Generate display data.

[0074] In addition, a speech bubble is further displayed, and onomatopoeia data 237 is displayed in the speech bubble. Alternatively, only the onomatopoeia data 237 may be displayed without outputting the voice data 236. Good too.

[0075] In addition, the display correction unit 224 corrects the virtual object 100 so that it is embedded in the structure 300. For example, the shape of the additional object 400 shown in FIG. As shown in the figure, the virtual object 100 is a combination of two rectangular parallelepipeds with different horizontal widths. Here, the virtual object 100 has a shape that is a combination of the two. is pushed into the structure 300 in the direction of the arrow 203, starting from the rectangular parallelepiped with a smaller horizontal width. It shall be possible.

[0076] In this case, the display correction unit 224 corrects the virtual object 100 as shown in FIG. The size of the additional object 400 is adjusted according to the horizontal width. As shown in FIG. 8(a), a rectangular parallelepiped area with a small horizontal width is virtually formed on the structure 300. If the object is embedded in the image, the additional object 400 is displayed in a small size. The area with a horizontal width larger than that in FIG. 8(a) is virtually sunk into the structure 300. In this case, as shown in Fig. 8(b), the additional object is larger than that in Fig. 8(a). Display number 400.

[0077] Furthermore, the line of sight changes depending on the position of the user 201. For example, in FIG. ), when the user 201 faces the wall of the structure 300, the virtual object Even if the display position of the object 100 in the world coordinate system is the same, ), the display mode of the virtual object 100 changes. The display shape of the object 100 also changes.

[0078] In this case, the display correction unit 224 also corrects the virtual object 100 as shown in FIG. The display shape of the additional object 400 is changed in accordance with the change in the display shape of the additional object 400. Good too.

[0079] As described above, the HMD 200 of this embodiment captures a color image of a predetermined shooting range. and a color image camera 214a that captures distance images of approximately the same shooting range. The camera 214b, the display 215, and the virtual object 100 are displayed on the display 21 The display control unit 220 controls the color image to be displayed on the display unit 5. A spatial recognition unit that generates a 3D map of the structure 300 within the shooting range using the image and the range image. 221, and the 3D map and the position data of the virtual object 100 to be displayed in the real space. Based on this data, the virtual object 100 behind the structure 300 in the line of sight is a display data generating unit (223) that generates display data specifying the area as the rear area (101); and a display correction unit 224 that corrects the display data and displays it on the display 215. In addition, the display correction unit 224 adjusts the display so as to display the operation point 110 in the rear area 101. Then, the operation point 110 corrects the virtual data. This is a point where an operation instruction for the virtual object 100 is accepted.

[0080] In this way, according to this embodiment, the image behind the structure 300, which would not normally be displayed, is The operation point 110 in the rear area 101 of the virtual object 100 is also displayed. Therefore, even if the virtual object 100 is moved by the operation of the user 201, An operation can be performed on the virtual object 100.

[0081] Furthermore, when the operation point 110 in the rear area 101 is displayed, the display correction unit 224 The rear area 101 is also displayed, and additional objects are displayed around the virtual object 100. Therefore, according to this embodiment, the virtual object 100 is displayed. Regardless of the display position on the display 215, it maintains a natural appearance without sacrificing realism. This allows for high operability.

[0082] [Variation 1] Note that the display of the virtual object 100 having a partial rear area 101 is not limited to the above. For example, the display correction unit 224 may correct the virtual object 100 by dividing the virtual object 100 into the rear area 101 and the The other area (hereinafter referred to as the front area) 102 may be displayed so as to be distinguishable from the other area. .

[0083] For example, as shown in FIG. 9(a), the display correction unit 224 corrects the rear area 101 and the front area 1 The display data is corrected so that line 121 is displayed between 02 and 121. As shown in the figure, the display correction unit 224 corrects the rear area 101 by correcting the original virtual object 100. The display data is corrected so that it is displayed in a surface aspect 122 different from the front surface. The correction modes of FIG. 9(b) and FIG. 9(c) may be combined.

[0084] [Variation 2] The additional object 400 may also be highlighted. An example of the additional object 400 is shown in FIG. 10. The highlighting mode of the additional object 400 is determined by the The image data may be registered in the object data 235. By using the above, the display correction unit 224 highlights the registered additional object 400. The image may be processed and displayed as follows.

[0085] By highlighting the additional object 400, the virtual object 100 is The user 201 can easily understand that the display mode is different from the previous one. The user 201 can more intuitively see that even the invisible area is displayed. It can be understood objectively.

[0086] [Variation 3] In addition, the virtual object 100 is inserted as additional object data 235. Different textures are available depending on the material and / or type of structure 300. In this case, the display correction unit 224 may correct the spatial recognition data 23 by the spatial recognition unit 221. 3 to identify the material and / or type of the structure 300. 24 is additional object data of texture according to the specified material and / or type 235 is extracted and displayed as an additional object 400.

[0087] For example, if the structure 300 is made of a hard material such as a wall, the wall may crack as shown in FIG. The additional object 400 representing such a state is used as the additional object data 235. On the other hand, when the structure 300 is made of a soft material such as a cushion, As shown in the figure, an additional object is added to represent a state in which the virtual object 100 sinks into the structure 300. A project 402 is prepared and displayed.

[0088] Furthermore, at this time, the manner of highlighting may also be determined based on the material and / or type of the structure 300, Alternatively, it may be configured to change depending on the texture of the additional object 400.

[0089] [Variation 4] Furthermore, the display correction unit 224 converts the color image data 231 into the additional object 400. In particular, as shown in FIG. 12, a part of the virtual object 100 The image is created using the same texture as the structure 301 behind the structure 300 into which the object is pushed. In this case, the display correction unit 224 may correct the additional object 400 according to the arrangement position of the additional object 400. Then, the additional object data 235 is processed using the space recognition data 233. The display correction unit 224 then arranges the processed additional object data 235.

[0090] In this way, the additional object 400 uses the same texture data as the background image. By doing so, the user 201 can change the display mode of the virtual object 100 of this embodiment as follows: It can be perceived more naturally.

[0091] [Variation 5] In the above embodiment, the virtual object 100 is positioned so that the entire area is in the rear area 101. When the virtual object 100 is moved to such a position, the display of the virtual object 100 is erased. Even if the entire area becomes the rear area 101, the virtual object The object 100 and / or the operation point 110 may be displayed.

[0092] For example, as shown in FIG. 13, the display correction unit 224 displays the virtual object at the calculated position. The display correction unit 224 corrects the display data so that the target 100 is displayed in the foreground. , a miniature virtual object 401 for operation may be displayed as an additional object. In this case, the display correction unit 224 converts the virtual object 100 into a miniature virtual object. The object 401 is generated and displayed.

[0093] This allows the virtual object 100 to be displayed even if the entire virtual object 100 is located further back in the depth direction than the structure 300. Even if the object is placed in a different location, the user 201 can perform the operation while maintaining a sense of reality. Cut.

[0094] [Variation 6] The display 215 may also be a non-transmissive type. In this case, as shown in FIG. The display correction unit 224 corrects the color image (through image) acquired by the color image camera 214a. 500), the virtual object 100 and the additional object 400 are superimposed and displayed.

[0095] In this case, for example, the display terminal is not limited to the HMD 200 having a transmissive display. For example, portable or mobile devices such as HMDs and mobile terminals with non-transparent displays It may be an information processing device.

[0096] [Variation 7] Furthermore, input of operation instructions is not limited to gaze and gestures. A controller or the like may also be used.

[0097] [Variation 8] In the above embodiment, all operation points 110 of the virtual object 100 are displayed. The display correction unit 224 corrects the display of the virtual object 100 so that The displayed operation points 110 are not limited to those shown in the back area 101. That is, at least one operation point in the back area 101 may be 110 may be displayed.

[0098] The same applies to the shape of the virtual object 100, and it is not always necessary to display the entire shape. It is not necessary to display only a part of the rear area 101.

[0099] The present invention is not limited to the above-described embodiment and modifications, and various modifications are possible. For example, the above-described embodiments and modifications are included to clearly explain the present invention. The present invention is not limited to the above embodiments, and is not limited to those having all the configurations described above. Furthermore, it is not intended to substitute a part of the configuration of one embodiment or modification for another embodiment or modification. In addition, the configuration of one embodiment or modification can be replaced with another embodiment. It is also possible to add the configuration of the embodiment or modification. It is possible to add, delete, or replace part of the configuration with other configurations.

[0100] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be implemented in part or in whole by, for example, For example, it may be realized by hardware, such as by designing an integrated circuit. The functions are realized by the processor interpreting and executing the programs that realize the respective functions. It may be realized by software. Programs, tables, files, etc. that realize each function. This information is stored in the memory, hard disk, and SSD (Solid State Drive) ) or on recording media such as IC cards, SD cards, DVDs, etc. Cut.

[0101] In addition, control lines and information lines are shown as they are considered necessary for explanation purposes, and do not necessarily represent all the lines in the product. This does not necessarily show all control and information lines. In reality, almost all components are interconnected. It can be said that this is the case. [Explanation of symbols]

[0102] 100: Virtual object, 101: Back area, 102: Front area, 110: Operation point nt, 121: line, 122: surface aspect, 200: HMD, 201: User, 202: Finger, 203: Arrow, 210: Controller 211: CPU, 212: RAM, 213: ROM, 214: Camera, 214a: 214b: distance image camera; 214c: line of sight detection device; 215: display Play, 215a: inside the frame, 216: audio I / F, 217: communication I / F, 218: sensor, 218a: GPS receiver, 218b: gyro sensor, 218c: acceleration sensor, 218 d: geomagnetic sensor, 218e: touch sensor, 219: bus, 220: Display control unit, 221: Space recognition unit, 222: Instruction reception unit, 223: Display data Generation unit, 224: display correction unit, 228: image acquisition unit, 229: audio output control unit, 230: Storage device, 231: Color image data, 232: Range image data, 233: Sky Inter-recognition data, 234: Virtual object data, 235: Additional object data, 2 36: Audio data, 237: Onomatopoeia data, 241: Frame, 300: Structure, 301: Structure, 400: Additional object, 401: Miniature Imaginary object, 402: Additional object, 500: Through image

Claims

1. A display terminal, The display and a color image camera that captures a color image of a predetermined shooting range; a distance image camera for capturing a distance image of the imaging range; A control unit; Equipped with The control unit A three-dimensional map of the structure within the photographing range is created using the color image and the distance image. Generate, The display terminal displays the three-dimensional map and the virtual object placement position data. In the user's line of sight, the area of the virtual object behind the structure is Identify it as the back area, Control is performed to display the virtual object on the display, 、 The display mode when the control unit displays the virtual object on the display includes: a first display mode in which only an area excluding the rear area of the virtual object is displayed; The entire area including the rear area of the virtual object is displayed, and further, the virtual object a second display mode in which an additional object is displayed around the A display terminal characterized by:

2. 2. The display terminal according to claim 1, The virtual object has an object body and a display mode of the object body. and an operation point that receives an operation instruction to change the and In the first display mode, the control unit controls the object body and the operation point. a part of the virtual object corresponding to an area excluding the rear area of the virtual object on the display Displayed in A, In the second display mode, the control unit controls the object body and the operation point. the part of the virtual object corresponding to the entire area including the rear area of the virtual object on the display. Show on Ray A display terminal characterized by:

3. 3. The display terminal according to claim 2, The control unit The display mode of the object body based on the operation instruction received by the operation point When the display mode of the object body is changed, the additional object is changed in accordance with the change in the display mode of the object body. The shape of the object is also transformed and displayed on the display. A display terminal characterized by:

4. 2. The display terminal according to claim 1, The additional object is an image of the structure being broken through by the virtual object. , a hole in the structure, or a crack in the structure. A display terminal characterized by:

5. 5. The display terminal according to claim 4, The control unit If a second structure is present behind the structure within the photographing range, the additional object a color image of the second structure as an object on the display; A display terminal characterized by:

6. 2. The display terminal according to claim 1, The additional object changes the structure when the virtual object sinks. It shows the shape of A display terminal characterized by:

7. 2. The display terminal according to claim 1, further comprising a storage unit for storing a plurality of additional objects; In the second display mode, the control unit: The additional object extracted from the storage unit is Display on the display A display terminal characterized by:

8. 2. The display terminal according to claim 1, further comprising an audio interface for outputting an audio signal; In the second display mode, the control unit: When the additional object is displayed on the display, a predetermined audio signal is input to the audio Output from the voice interface A display terminal characterized by:

9. 9. A display terminal according to claim 8, In the second display mode, the control unit: and further displaying a character string representing the predetermined audio signal on the display. A display terminal characterized by:

10. A display terminal according to any one of claims 1 to 9, The display terminal is a head-mounted display. A display terminal characterized by:

11. A display terminal according to any one of claims 1 to 9, The display terminal is a portable information processing device. A display terminal characterized by:

12. a display, a color image camera for capturing a color image of a predetermined photographing range, and a front A display terminal including a distance image camera for acquiring a distance image of the photographing range and a control unit. A virtual object display method comprising: A three-dimensional map of structures within the imaging range is generated using the color image and the distance image. a three-dimensional map generation step; Based on the three-dimensional map and the placement position data of the virtual object, In the user's line of sight, the area of the virtual object behind the structure is a back surface region identifying step of identifying the back surface region as a surface region; a virtual object display step of displaying the virtual object on the display; 、 Equipped with In the virtual object display step, the virtual object is displayed on the display. In the display mode when displaying the virtual object, only the area excluding the rear area of the virtual object is displayed. a first display mode in which the entire area including the rear area of the virtual object is displayed; a second display mode in which an additional object is displayed around the virtual object; be A virtual object display method comprising:

Images