Display terminal

Abstract

To provide a virtual object display terminal configured to realize high operability regardless of a display position of a virtual object, while maintaining a sense of reality.SOLUTION: An HMD (head-mounted display) 200, which is to be worn on the head of a user (wearer) 201, as a display terminal includes: a color image camera and a distance image camera for acquiring a color image and a distance image of a predetermined imaging range, respectively; and a display control unit. The display control unit includes: a space recognition unit which uses the color image and the distance image to generate a three-dimensional map of a structural object 300; a display data generation unit for generating display data in which a rear region of a virtual object 100 behind the structural object 300 in a line-of-sight direction is specified, based on the three-dimensional map and real-space placement position data of the virtual object 100; and a display correction unit for correcting the display data so as to display operation points 110 of the rear region, which accept an operation instruction on the virtual object 100.SELECTED DRAWING: Figure 1

Description

[[Technical Field]]

[0001] The present invention relates to a technology for displaying a virtual object on a display terminal. [[Background Art]]

[0002] HMD (Head Mounted Display) A technology for improving the visibility of augmented reality (AR, Augmented Reality) images, in which computer-generated information is added to objects in real space displayed on a display terminal such as the above, is known in the art. For example, Patent Document 1 discloses a head-mounted display device comprising a display unit that allows the outside world to be seen through, the device comprising: a superimposed image display control unit that causes a predetermined image to be displayed on the display unit so as to be superimposed on the outside world that is seen through transmission; an imaging unit that images at least a predetermined range of the outside world that is seen through transmission; a partial image specifying unit that specifies, from a captured image obtained by the imaging unit, a partial image in a predetermined range whose position corresponds to the predetermined image; and a visibility correction unit that corrects the appearance of the predetermined image displayed by the superimposed image display control unit according to color information of the specified partial image (abstract excerpt). [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2016-142887 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] A virtual object or the like displayed in an AR image is set in advance around the ​​​​​​​By issuing instructions to the designated control points, operations such as movement, rotation, and deformation can be performed. However, depending on the display position of the virtual object, a part of that virtual object may be The area is hidden behind surrounding structures, eliminating shadows, and consequently, the necessary operating points are displayed. Sometimes they are not available. Therefore, high usability is not always achieved. On the other hand, virtual objects If you display the virtual object as is, regardless of its position after the operation, you will not get a sense of realism. I can't.

[0005] This invention was made in view of the above circumstances, and regardless of the display position of the virtual object, Our aim is to provide virtual object display technology that maintains a sense of realism while achieving high operability. To target. [Means for solving the problem]

[0006] The present invention is a display terminal equipped with a display, comprising: a color image camera that acquires a color image of a predetermined shooting range; and a distance image camera that acquires a distance image of the said shooting range. , A gyro sensor, an accelerometer, a touch sensor, the color image and the depth image are used to generate a 3D map of the shooting range including structures within the shooting range, and the color image camera and the depth image camera are used. ,before The gyro sensor or the acceleration sensor at least one ofBased on the input from, a first line of sight direction is determined; based on spatial recognition data, an object for placing a virtual object in the 3D map is displayed; via the touch sensor, an operation instruction is received to place the virtual object at a position within the area on the 3D map; depth information of the virtual object is calculated based on the virtual object data corresponding to the virtual object; depth information of the structure is calculated; the virtual object is displayed at the position on the 3D map of the shooting range; based on the depth information of the virtual object and the depth information of the structure, the back region of the virtual object that is behind the structure is determined; based on the depth information of the virtual object and the depth information of the structure, the back region of the virtual object that is behind the structure is identified; based on the first line of sight direction, the virtual object, the structure, and the back region of the virtual object within the shooting range corresponding to the first line of sight are displayed on the display as a first display mode; the color image camera, the distance image camera ,before The gyro sensor or the acceleration sensor at least one of The system is characterized by comprising: a display control unit configured to determine a second viewing direction different from the first viewing direction based on input from the system, and to control the display of the virtual object, the structure, and the back region of the virtual object within the shooting range corresponding to the second viewing direction on the display as a second display mode different from the first display mode based on the second viewing direction. [Effects of the Invention]

[0007] According to the present invention, regardless of the display position of virtual objects, high operability is achieved while maintaining a sense of realism. We can provide virtual object display technology that realizes this. Other challenges, configurations and The effects will be revealed by the following description of the embodiments. [Brief explanation of the drawing]

[0008] [Figure 1](a) to (c) are explanatory diagrams for outlining the virtual object display processing according to the embodiment of the present invention. [Figure 2] (a) is a hardware configuration diagram of the HMD according to the embodiment of the present invention, and (b) is an external view of the HMD according to the embodiment of the present invention. [Figure 3] It is a functional block diagram of a controller of the HMD according to the embodiment of the present invention. [Figure 4] (a) and (b) are explanatory diagrams for outlining the virtual object display processing according to the embodiment of the present invention. [Figure 5] It is a flowchart of the virtual object display processing according to the embodiment of the present invention. [Figure 6] (a) and (b) are explanatory diagrams for explaining display on a display according to the embodiment of the present invention. [Figure 7] (a) to (c) are explanatory diagrams for explaining an example of an aspect of virtual object display according to the embodiment of the present invention. [Figure 8] (a) to (c) are explanatory diagrams for explaining another example of an aspect of virtual object display according to the embodiment of the present invention. [Figure 9] (a) and (b) are explanatory diagrams for explaining Modification 1 of an aspect of virtual object display according to the embodiment of the present invention. [Figure 10] It is an explanatory diagram for explaining Modification 2 of an aspect of virtual object display according to the embodiment of the present invention. [Figure 11] It is an explanatory diagram for explaining Modification 3 of an aspect of virtual object display according to the embodiment of the present invention. [Figure 12] It is an explanatory diagram for explaining Modification 4 of an aspect of virtual object display according to the embodiment of the present invention. [Figure 13] It is an explanatory diagram for explaining Modification 5 of an aspect of virtual object display according to the embodiment of the present invention. [Figure 14] It is an explanatory diagram for explaining Modification 6 of an aspect of virtual object display according to the embodiment of the present invention. DESCRIPTION OF EMBODIMENTS

[0009] Embodiments of the present invention will be described below with reference to the drawings. In the following description of this specification , components having the same function are denoted by the same reference symbols unless otherwise stated, and repeated description is omitted . Note that the present invention is not limited to the embodiments described herein.

[0010] In the present embodiment, as a display terminal, an HMD ( Head Mounted Display) that a user (wearer) wears on their head to use, which is provided with a transmissive display, and is a transmissive (see-through type ) HMD that allows visual recognition of both the external world and a displayed image will be described as an example.

[0011] First, an outline of the present embodiment will be described with reference to FIG. 1(a) to FIG. 1(c). In the present embodiment , as shown in FIG. 1(a), a user 201 wears an HMD 200 indoors, displays and operates a virtual object 100 on the display of the HMD 200.

[0012] The HMD 200 has distance information (depth information) for both the real space and the virtual object 100 . Conventionally, as shown in FIG. 1(b) on the display of the HMD 200, the portion of the virtual object 100 that is arranged deeper than a structure 300 such as a wall in real space is not displayed. Therefore , in such a case, the user 201 cannot grasp the entire appearance of the virtual object 100 .

[0013] In general, the virtual object 100 is operated by operating an operation point (transform controller) 110 set on or near the vicinity of the virtual object 100 ​Operations such as moving, rotating, and transforming are performed. Therefore, when displayed as in Figure 1(b), the structure The control point 110 on the back of the structure 300 is not displayed and therefore cannot be operated.

[0014] In this embodiment, in order to solve this problem, as shown in Figure 1(c), the structure 300 is Even so, the virtual object 100 displays all operation points 110. The virtual object 100 itself may also be displayed in its entirety. Furthermore, in this case, originally To show that even the parts that are not normally displayed are being shown, the additional object 400 is displayed. You may show it.

[0015] The following describes the HMD200 of this embodiment that realizes such display control.

[0016] [Hardware Configuration Diagram] Figure 2(a) is a hardware configuration diagram of the HMD200. Figure 2(b) is a diagram of the actual This is an external view of the HMD200 in its installed form.

[0017] The HMD200 in this embodiment is basically the same as a general-purpose computer (information processing device). It has the following configuration. That is, the HMD200 has a controller 210 and, as shown in this figure, Camera 214, display 215, audio interface (I / F) 216, communication I / F217, sensor218, bus219 for electrically connecting each part, and gaze detection device It includes 214c and, furthermore, the HMD200 of this embodiment supports each part, and user 20 It has a frame 241 for mounting.

[0018] The controller 210 performs various processes according to a predetermined program. In this state, the controller 210, for example, places a virtual object at a predetermined position on the display 215. Display JECT 100.

[0019] The controller 210 of this embodiment includes a CPU 211, RAM 212, and ROM 213 The CPU 211 processes the program pre-stored in the ROM 213 into the RAM 2 By loading and executing it in 12, various functions are realized. Note that RAM 212 and If there is no particular need to distinguish between them, the ROM 213 and the ROM 213 together constitute the storage device 230 (Figure (See 3) This is referred to as [this]. Note that the controller 210 is, for example, located on frame 241. .

[0020] The camera 214 comprises a color image camera 214a and a depth image camera 214b. The color image camera 214a captures the shooting range including the user 201's field of view, and - Acquire an image. Also, the distance image camera 214b is almost the same as the color image camera 214a. The camera 214 (color image camera 214a and The distance image camera 214b) captures, for example, the foremost part of frame 241 (the most prominent part of display 21 It is positioned on side 5) in a location where the above shooting range can be captured.

[0021] Display 215 shows images acquired by camera 214 and tables generated within HMD200. This is a device on which data is displayed. The display 215 is, for example, a transmissive liquid crystal display. Vise, organic EL devices, or MEMS (microelectronic machinery) It consists of optical scanning devices using (analytic systems). However, However, the device is not limited to this, and while displaying video on the display 215, A device capable of realizing a transparent display structure that allows you to see through to what's on the other side of the Play215. It would be nice to have.

[0022] In the HMD200 of this embodiment, the transparent display 215 is used for one eye of the user 201. Alternatively, it is supported in front of both eyes. The display 215 can take any shape. The display 215 may have right and left display panels, and the display Ray 215 displays one or more UI objects of the graphical user interface. It may also be used.

[0023] The audio interface 216 is, for example, an audio output device such as a microphone, speaker, or buzzer. The voice interface 216 accepts external sound input and sounds created within the HMD200, as well as the communication interface 217. It outputs sound such as voice and music that is sent via the audio I / F2. Item 16 does not need to be prepared.

[0024] The communication I / F217 is equipped with coding circuits, decoding circuits, antennas, etc., and communicates via the network. It performs data transmission and reception (data communication) with other devices. In this embodiment, the communication I / F 217 This is done via access points, etc. (not shown in the diagram), or via mobile telephone services (not shown in the diagram). This is an interface that connects to the network via base stations and other infrastructure. It connects via communication interface 217. The HMD200 then transmits and receives data with each server connected to the network.

[0025] The connection between the HMD200 and the access point is, for example, via Wi-Fi (registered trademark), etc. This is done by wireless communication or other communication methods. HMD200 and the mobile telephone communication Connection to the base station of the network is, for example, W-CDMA (registered trademark) (Wideband Cod e Division Multiple Access) method and GSM (registered trademark) Global System for Mobile communications) The method, LTE (Long Term Evolution) method, or other communication method It is performed by formula. Note that the HMD200 in this embodiment does not have a communication I / F217. That's fine.

[0026] Sensor 218 detects the current position, tilt, speed, and user 201 operations of the HMD200. Detects. The HMD200 uses a sensor 218, for example, a GPS receiver 218a, to detect position. Location information acquisition sensor, gyro sensor 218b, acceleration sensor 218c, geomagnetic sensor 21 It is equipped with 8d, touch sensor 218e, etc. Note that it is not necessary to have all of the sensors 218. stomach.

[0027] The gaze detection device 214c detects the direction of the user's gaze 201. This can be achieved, for example, by an eye-tracking camera that detects the direction of the user 201's gaze. The external camera is mounted so that the iris, pupil, etc. of user 201's eye are included in the shooting range.

[0028] Frame 241 is an HM (Handheld Motor) containing a display 215, camera 214, controller 210, etc. Supports the components of the D200.

[0029] [Function Block] Next, the controller 210 of this embodiment implements the following related to virtual object display. The functions will be explained. Figure 3 shows the virtual object display processing of the HMD200 in this embodiment. This is a functional block diagram of the related functions. As shown in this figure, the controller 21 of this embodiment 0 represents the functions of the image acquisition unit 228, the display control unit 220, and the audio output control unit 229. The display control unit 220 also includes a spatial recognition unit 221, an instruction receiving unit 222, and a display data It includes a data generation unit 223 and a display correction unit 224.

[0030] Each function involves the CPU 211 loading the program stored in ROM 213 into RAM 212. This is achieved by running the command.

[0031] Furthermore, the storage device 230 contains color image data 231, distance image data 232, and empty Intercognition data 233, virtual object data (virtual OJT data) 234, and additional data Project data (additional OJT data) 235 and audio data 236 are stored.

[0032] Color image data 231 is an image acquired by the color image camera 214a. Image data 232 is an image acquired by the depth image camera 214b.

[0033] The image acquisition unit 228 uses a color image camera 214a and a depth image camera 214b to acquire images. The acquired color image and distance image are then processed as color image data 231 and distance image data. It is stored in the storage device 230 as data 232. In this embodiment, a color image and a distance image are stored. The images are acquired in roughly synchronous manner.

[0034] The spatial recognition unit 221 recognizes the surrounding real space and processes the result as spatial recognition data 233. The data is then stored in the memory device 230. Recognition is performed using the color image data 231 acquired almost simultaneously. This is done using distance image data 232.

[0035] The spatial recognition unit 221, in accordance with the user 201's scanning operation, processes each image at predetermined time intervals. From the image data, the spatial data (3D map) of 300 structures within the shooting range is obtained. Recognition data 233 is generated and stored in the storage device 230. The surrounding scan is performed, for example, Immediately after activation, user 201 performs the initial setup.

[0036] Spatial recognition data 233 is created, for example, in a world coordinate system that defines the entire 3D space. As an example, the origin and axis of this world coordinate system are the starting points for spatial recognition. The H is identified by the position and orientation (initial posture) of the HMD200 unit when the signal is received. The origin and axis directions of the MD200 local coordinate system are used. In this coordinate system, for example, In the initial orientation of the HMD200 main unit, a predetermined position on the display 215 of the HMD200 With the origin at , the xy-plane is defined as the area within display 215, and the z-axis direction is defined as the xy-plane (display The direction should be perpendicular to the play surface (215).

[0037] Furthermore, the world coordinate system of the HMD200's local coordinate system, as determined by user 201's scan operation. Displacement and rotation amounts relative to the coordinate system are obtained using data from various sensors 218. It is calculated.

[0038] Furthermore, spatial recognition can be performed using existing technologies such as Spatial Mapping. To do so, the HMD200 of this embodiment includes a color image camera 214a and a distance image camera. The image camera 214b scans the surroundings. Then, using the results, the spatial recognition unit 2 21 uses applications such as Spatial Mapping to obtain 3D data The spatial recognition data 233 is generated and stored, for example, as mesh data.

[0039] Furthermore, at this time, Spati recognizes not only 3D data but also 300 types of structures. The system may also be configured to perform al-Understanding simultaneously. (Spatial recognition unit 221) Spatial Understanding allows for the imaging of 300 structures within the imaging range. The material and type can be recognized. That is, the structure 300 can be identified as, for example, a "wall", "floor", "ceiling". It can recognize which of the following is true. The spatial recognition unit 221 of this embodiment recognizes the spatial These recognition results are stored in the storage device 230 as attribute data for the recognition data 233.

[0040] The instruction receiving unit 222 receives instructions from the user 201 regarding the virtual object displayed on the display 215. It accepts display and operation instructions for ECT 100. Display and operation instructions are Examples include those based on gaze and those based on hand gestures. .

[0041] Information on the direction of gaze used in the gaze is detected, for example, using the gaze detection device 214c. It can be done.

[0042] A gesture can, for example, be a click on the operation point 110 of a virtual object 100. Features include catch events (air tap), tap and hold, bloom, etc. Instruction reception unit 2 22 is located within the shooting range of the color image camera 214a and the depth image camera 214b. It detects hand and finger movements within a defined gesture frame and uses this to detect display instructions, operation instructions, etc.

[0043] For example, when a display instruction is received, the instruction receiving unit 222 receives virtual object data 2 The data of the virtual object 100, as instructed by 34, is extracted and displayed in the display data generation unit described below. Generate the display data using 223.

[0044] Furthermore, when an operation instruction is received, the instruction receiving unit 222 detects the operation and, as described in the table below, The data generation unit 223 is notified.

[0045] The display data generation unit 223, in accordance with the instructions of the user 201 via the instruction reception unit 222, Display the specified virtual object 100 from the virtual object data 234. The instruction receiving unit 222 generates display data to be displayed in a predetermined shape at a predetermined location on 215. By displaying the generated display data on the display 215 according to the instructions, virtual Object 100 is displayed to move, rotate, and deform according to the instructions of User 201. It will be shown.

[0046] At this time, the display data generation unit 223 determines the direction of the user 201's line of sight (where the user 201 is wearing) (Calculated from the spatial coordinate position and orientation information of the HMD200) The area of ​​virtual object 100 behind 300 is identified as the back area 101. Data is generated. The back region 101 (Figure 4) is a 3D map generated by the spatial recognition unit 221. The real-space position of the virtual object 100, which is displayed as "P" (spatial recognition data 233). It is identified based on the data. The spatial position data is obtained from the spatial recognition unit 2. 21 is registered in the same coordinate system as the coordinate system used for spatial recognition.

[0047] Note that the real-world position data of virtual object 100 is used to display the virtual object. Obtained from virtual object data 234 of Effect 100. Virtual object data 23 4 includes the size, shape, and initial placement information for each virtual object 100. Placement location information includes, for example, the placement location points pre-set for each virtual object 100. The operation point 110, etc., are registered. The placement location point is, for example, a virtual object. This is the three-dimensional center of gravity position of object 100, etc. Also, the operation point 110 is as described above. This is the point where instructions for deforming the display shape of virtual object 100 are received.

[0048] The display data generation unit 223 reflects the instruction from the instruction reception unit 222 into the current placement position information. This results in virtual object data 23 corresponding to the virtual object 100 to be displayed. 4. Obtain the latest placement location information. Then, the display data generation unit 223 generates the placement location point. Using the latest placement information, size, shape, etc. of the virtual object 100 Obtain location data in real space.

[0049] The display data generation unit 223 further uses the latest real-space placement position data to generate the data. Identify the back area 101 of the virtual object 100 to be displayed. For example, raw display data The adult part 223, based on the real-space placement position data of the virtual object 100, generates a virtual O The spatial coordinate position of the HMD200 worn by user 201 of object 100, above Distance to the virtual object calculated from the information of the downward, left, and right orientation, virtual object 10 The depth information is identified from the shape data of 0 and the coordinate position of structure 300.

[0050] And, among the depth information, a virtual object located relatively further back than the depth information of structure 300. The area (part) of the object 100 and the operating point 110 are defined as the back area 101.

[0051] Note that the back surface region 101 is the area that is normally subjected to hidden surface processing. For example, see Figure 4. As shown in (a), a part of the virtual object 100 is behind a structure 300 such as a wall or furniture. When placed, conventionally, the display data generation unit 223 is on the back of the virtual object 100. Area 101 is subjected to a hidden surface treatment, and display data is generated so that it is not displayed.

[0052] However, in this embodiment, in order to aim for a more user-friendly display, display correction is performed. Even in such a case, part 224, as shown in Figure 4(b), virtual object 1 The display is corrected to show the entire 00 and its associated operating points 110. The display correction unit 224 performs the processing.

[0053] The display correction unit 224 has a back area 101 in the virtual object data 234 to be displayed. In that case, the display data of the virtual object data 234 is corrected.

[0054] Specifically, the display correction unit 224 cancels the negative surface processing for the back surface area 101. And the rear area 101 is also displayed as if the structure 300 does not exist. The data is corrected. That is, the display correction unit 224 corrects one of the virtual object data 234. Even if the area of ​​the section is located further back than structure 300, the display will also show that area. Correct the data.

[0055] Furthermore, the display correction unit 224 then adjusts the display so that the virtual object 100 is in its original display state. To show that it does not exist, display additional object 400. Additional object 400 For example, this shows a virtual object 100 penetrating structure 300, structure 3 To make the display of virtual object 100 more natural and realistic, such as showing a hole being made in 00. It is an object used to make it appear as if something exists.

[0056] The data of the additional object 400 to be displayed is stored in the additional object data of the storage device 230. It is prepared in advance as data 235. Additional object data 235 is an additional object 400 display modes, initial size, initial display position relative to the display position of the virtual object 100 Information such as location is registered in association with other relevant data.

[0057] Furthermore, the display correction unit 224 adjusts the virtual object 100 based on the user 201's operation. If the display direction of the -za 201 changes relative to the line of sight, the additional object will follow suit. The display shape of the ct 400 may be modified. Furthermore, by changing the direction of the user 201's line of sight Similarly, if the display direction of virtual object 100 changes, the additional object 400 will also change. The display shape may be deformed. Deformation of the additional object 400 may, for example, change the direction of view. The same program used for the process of transforming the display of virtual object 100 during the transformation process. Follow the instructions.

[0058] Next, the display processing of the virtual object 100 by the controller 210 of this embodiment Let's explain the process. Figure 5 shows the processing flow of the virtual object display process in this embodiment. Furthermore, spatial recognition processing is assumed to have already been performed as an initial process. Effect 100 is assumed to be displayed in a position where the entire image is visible.

[0059] Until a termination instruction is received from user 201 via the instruction receiving unit 222, the display data The generation unit 223 and the display correction unit 224 repeat the following process (step S1101): .

[0060] User 201, via the instruction receiving unit 222, controls the movement of the virtual object 100. Upon receiving the input (step S1102), the display data generation unit 223 generates the display data. To accomplish (Step S1103).

[0061] In step S1103, the display data generation unit 223 first generates the virtual object 1 The display position of 00 on the display 215 is calculated. Then the display data generation unit 223 , identify the direction of gaze, and the corresponding virtual object data 234 of virtual object 100 The depth information is calculated accordingly. The depth information of structure 300 is also calculated. Then, the table The display data generation unit 223 identifies the back area 101 on the display data.

[0062] Next, the display data generation unit 223 generates a back area 101 for the virtual object 100 to be displayed. Determine whether or not it exists (step S1104).

[0063] If the rear area 101 is not available (S1104; No), the display data generation unit 223 will indicate that fact. The display correction unit 224 is then notified. The display correction unit 224 then proceeds to... At the position calculated in step S1103, the display data of the virtual object 100 is displayed. The data is displayed (step S1112). Then the controller 210 gives the following operation instructions. wait.

[0064] If there is a rear area 101 (S1104; Yes), the display data generation unit 223 is virtual Determine whether the entire object 100 is the back region 101 (step S110) 5).

[0065] If the entire area is the back region 101 (S1105; Yes), the display data generation unit 223 will The display correction unit 224 is notified of this. The display correction unit 224 then proceeds with the provisional The display of the thought object 100 is erased (step S1111), and the process is terminated.

[0066] On the other hand, if the entire area is not the back region 101 (S1105; No), that is, the virtual object If a portion of the projection 100 has a back area 101, the display data generation unit 223 will indicate that fact. The display correction unit 224 is notified. The display correction unit 224 then processes the virtual object using the method described above. Correct the display data of the unit 100 (step S1106). Here, the display correction unit 22 4 refers to all operation points 110 of the virtual object 100 and said operation point 11 The display data is corrected to show the portion of virtual object 100 that is manipulated by 0. Then, the display correction unit 224 displays the corrected display data in steps on the display 215. Display at the position calculated in S1103 (step S1107).

[0067] Subsequently, the display correction unit 224 retrieves the additional object data 235 from the storage device 230. The virtual object 100 is then displayed superimposed on the back region 101 (step S110). 8) The controller 210 then waits for the next operation instruction.

[0068] The above process will be explained with a specific example. Note that the display 215 is as shown in Figure 6(a). As shown above, within the frame 215a defined by the display 215, the virtual object 100 and The additional object 400 is displayed. Here, structure 300 is a real object in real space. This is the case. In the following description of the display configuration, the frame of the display 215 is omitted, as shown in Figure 6(b). As shown, real objects and virtual objects are visible through the display 215. Only the displayed data, such as 100, will be listed and explained.

[0069] For example, let's assume that a virtual object 100 is displayed at the position shown in Figure 7(a). In response, user 201, in real space, uses their fingers 202 to point in the direction of arrow 203. Then, perform an operation instruction (gesture) to press. By continuing this, the virtual object The display position of ct100 relative to structure 300, which is a real object in real space, moves. .

[0070] According to this embodiment, as shown in Figure 7(b), a portion of the virtual object 100 Even if the display position is behind structure 300, virtual object 100 will still be able to move to its control point. The entire structure is displayed along with the int 110. Also, at this time, it appears as if it were structure 300. The additional object 400 appears as if there's a hole in the wall.

[0071] Furthermore, as shown in Figure 7(c), the audio output control unit 229 performs audio output. It may be configured in such a way. The display correction unit 224 displays the additional object 400. The audio output control unit 229 is notified accordingly. The audio output control unit 229 then controls the display correction unit 224 Upon receiving further notification, the audio is extracted from audio data 236 and output from audio I / F 216. ru.

[0072] Furthermore, the audio data 236 is registered with different data corresponding to the material of the structure 300. It may be left as is. The material of the structure 300 is, for example, Spa recognized by the spatial recognition unit 221 described above. Recognition is achieved through tial understanding.

[0073] Furthermore, even if you display onomatopoeic words that represent the output audio data 236 as strings at this time Good. For example, the "crackling" sound shown in Figure 7(c). In this case, the sound is stored in the memory device 230. Onomatopoeic data 237 that represents the voice data 236 as a string, in association with the voice data 236. This is registered in advance. Then, the display correction unit 224 corresponds to the output audio data 236. The onomatopoeic data 237 that is registered is to be displayed near the attached object 400. Generate display data.

[0074] Furthermore, a speech bubble is displayed, and the onomatopoeic data 237 is displayed within that speech bubble. It is also possible to display only the onomatopoeia data 237 without outputting the audio data 236. That's good too.

[0075] Furthermore, the display correction unit 224 corrects the virtual object 100 to the extent that it appears to be embedded in the structure 300. The shape of the added object 400 is then changed to follow it. For example, as shown in Figure 8(a) In this way, the virtual object 100 combines two rectangular prisms with different horizontal widths. Let's take the case where the shapes are combined as an example. Here, virtual object 100 In the direction of arrow 203, the rectangular prism with the smaller horizontal width is pushed into the structure 300. It shall be assumed that it is.

[0076] In this case, the display correction unit 224, as shown in Figure 8(b), The size of the additional object 400 is adjusted and displayed according to the horizontal width. As shown in Figure 8(a), a rectangular prism region with a small horizontal width is virtually represented in the structure 300. If it is embedded, display the additional object 400 in a smaller size. On the other hand, In the case of Figure 8(a), an area with a horizontal width larger than that of the structure 300 is virtually embedded within the structure. If present, an additional object larger in size than that shown in Figure 8(a) will be added, as shown in Figure 8(b). Display "To400".

[0077] Furthermore, the direction of the line of sight changes as the user 201's standing position changes. For example, see Figure 8(c As shown in ), when user 201 faces a wall which is structure 300, the virtual object Even if the display position of ct100 in the world coordinate system is the same, Figure 8(a) and Figure 8(b) In the case of ), the display mode of the virtual object 100 changes. Accordingly, the virtual object The display shape of Ect100 will also change.

[0078] In this case as well, the display correction unit 224, as shown in Figure 8(c), the virtual object 100 In response to changes in the display shape, the display shape of the additional object 400 is changed and displayed accordingly. That's good too.

[0079] As described above, the HMD200 of this embodiment captures a color image within a predetermined shooting range. A color image camera 214a acquires a color image, and a distance image camera acquires a distance image of approximately the same shooting range. Camera 214b, display 215, and virtual object 100 are displayed on display 21 The system includes a display control unit 220 that displays a color image. Spatial recognition unit generates a 3D map of the structures 300 within the shooting range using images and depth images. 221 and the real-space position data of the virtual object 100 to be displayed on the 3D map. Based on the data, the virtual object 100 that is behind the structure 300 in terms of the line of sight A display data generation unit 223 generates display data that identifies the region as the back region 101, It includes a display correction unit 224 that corrects the display data and displays it on the display 215. Furthermore, the display correction unit 224 displays the operation points 110 of the rear area 101. The data is corrected. Then, the operation point 110 is temporarily This is the point that receives operation instructions for the imagined object 100.

[0080] Thus, according to this embodiment, the area behind the structure 300, which would normally not be visible, Therefore, the operation points 110 of the back area 101 of the virtual object 100 are also displayed. Even if the virtual object 100 is moved by user 201's operation, Operations can be performed on virtual object 100.

[0081] Furthermore, when the display correction unit 224 displays the operation point 110 of the rear area 101, The back area 101 is also displayed, and additional objects are added around the virtual object 100. Display the 400. Therefore, according to this embodiment, the data of the virtual object 100 Regardless of the display position on the 215 display, it maintains a sense of realism without sacrificing a natural appearance. This allows for high operability.

[0082] [Example 1] Furthermore, the display of a virtual object 100 that has a back area 101 in part is not limited to the above. No. For example, the display correction unit 224 measures the virtual object 100 in relation to the back area 101 and The other area (hereinafter referred to as the front area) 102 may be displayed in a way that makes it distinguishable. .

[0083] For example, as shown in Figure 9(a), the display correction unit 224 has a rear area 101 and a front area 1 The display data is corrected so that line 121 is displayed between 02 and 02. Also, in Figure 9(b) As shown, the display correction unit 224 corrects the back area 101 to the original virtual object 100 The display data is corrected so that it is displayed in a surface configuration 122 different from the surface. Figure 9(a) The correction methods shown in Figure 9(b) may be combined.

[0084] [Differentiation 2] Additionally, the additional object 400 may be highlighted. An example of object 400 is shown in Figure 10. The highlighting mode of the additional object 400 is predetermined. You may also register it in the object data 235. Furthermore, various image processing software... Using A, the display correction unit 224 highlights the additional object 400 that is registered. It may be processed and displayed in this way.

[0085] By highlighting the additional object 400, the virtual object 100 becomes this User 201 can easily understand that this is different from the previous display method. User 201 can more intuitively see that even areas that would normally be invisible are being displayed. It can be grasped precisely.

[0086] [Difference 3] Additionally, the virtual object 100 is inserted as additional object data 235. Different textures are available depending on the material and / or type of structure 300. This is also possible. In this case, the display correction unit 224 receives the spatial recognition data 23 from the spatial recognition unit 221. Refer to 3 to identify the material and / or type of structure 300. Then, display correction unit 2 24 is additional object data for textures corresponding to the identified material and / or type. Extract 235 and display it as an additional object 400.

[0087] For example, if the structure 300 is made of a hard material such as a wall, the wall may crack as shown in Figure 10. An additional object 400 representing such a state is used as additional object data 235. It is intended. On the other hand, if the structure 300 is made of a soft material such as a cushion, as shown in Figure 11. As shown, an additional object represents a state in which the virtual object 100 sinks into the structure 300. Ject 402 is prepared and displayed.

[0088] Furthermore, at this time, the manner of highlighting may also include the material and / or type of the structure 300. Alternatively, it may be configured to change according to the texture of the added object 400.

[0089] [Differentiation Example 4] Furthermore, the display correction unit 224 adds color image data 231 as an additional object 400. A portion of it may be cut out and used. In particular, as shown in Figure 12, virtual object 100 The structure 300 into which it is pushed uses the same texture image as the structure 301 behind it. It is acceptable. In this case, the display correction unit 224 adjusts according to the placement position of the additional object 400. Then, the spatial recognition data 233 is used to process the additional object data 235. The display correction unit 224 places the added object data 235 after processing.

[0090] In this way, the same texture data as the background image is used as the additional object 400. By doing so, user 201 can change the display mode of the virtual object 100 in this embodiment. It can be perceived more naturally.

[0091] [Difference 5] In the above embodiment, all virtual objects 100 become the back area 101. Configure the system to erase the display of virtual object 100 when it is moved to a specific position. However, it is not limited to this. Even if the entire area becomes the back region 101, the virtual object The ect 100 and / or operating point 110 may be displayed.

[0092] For example, as shown in Figure 13, the display correction unit 224 displays a virtual object at the calculated position. The display data is corrected so that "To 100" is displayed. Furthermore, the display correction unit 224 is positioned in front Even if you display a miniature virtual object 401 for manipulation as an additional object, Good. In this case, the display correction unit 224 converts the virtual object 100 to a miniature virtual object. Generate and display Ject 401.

[0093] This means that even if the entire virtual object 100 is located deeper in the depth direction than the structure 300, Even when placed in such a position, user 201 can perform operations while maintaining a sense of reality. Cut.

[0094] [Modification 6] Furthermore, the display 215 may be non-transparent. In this case, as shown in Figure 14. The display correction unit 224 then corrects the color image (through image) acquired by the color image camera 214a. The virtual object 100 and the additional object 400 are superimposed on 500).

[0095] In this case, for example, the display terminal is limited to the HMD200 equipped with a transparent display. No. Examples include HMDs with non-transparent displays, portable or handheld devices such as mobile terminals. It may also be an information processing device.

[0096] [Difference 7] Furthermore, input of operation instructions is not limited to gaze and gestures. Voice and motion control are also supported. You may also use a controller or similar device.

[0097] [Differentiation 8] In the above embodiment, all operation points 110 of the virtual object 100 are displayed. To that end, the display correction unit 224 corrects the display of the virtual object 100, It is not limited to this. The operation points 110 to be displayed are some of the operation points in the rear area 101. It may also be T110. That is, at least one operating point in the rear area 101 It's okay if 110 is displayed.

[0098] Similarly, the shape of virtual object 100 does not necessarily need to be displayed in its entirety. It is not necessary. You may display a portion of the back area 101.

[0099] Furthermore, the present invention is not limited to the embodiments and modifications described above, and various modifications are possible. Examples are included. For example, the embodiments and modifications described above clearly illustrate the present invention. This is a detailed explanation, and is not necessarily limited to those that possess all the configurations described. This does not mean that a part of the configuration of one embodiment or modified example may be used in other embodiments or modified examples. It is possible to replace the configuration with that configuration. Also, in one embodiment or modified configuration, other embodiments It is also possible to add configurations of different embodiments or variations. Furthermore, each embodiment or variation It is possible to add, delete, or replace parts of the configuration with other components.

[0100] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely modified, for example. Alternatively, it may be implemented in hardware by designing it with an integrated circuit. Functions are achieved by the processor interpreting and executing programs that implement each function. This can also be implemented in software. This includes programs, tables, files, etc., that implement each function. This information includes the memory section, hard disk, and SSD (Solid State Drive). ) can be placed on recording devices such as IC cards, SD cards, DVDs, etc. Cut.

[0101] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes and do not necessarily represent all of them in the actual product. It does not necessarily indicate control lines or information lines. In reality, almost all components are interconnected. It is reasonable to assume that this is the case. [Explanation of Symbols]

[0102] 100: Virtual object, 101: Back area, 102: Front area, 110: Operation point 121: line, 122: surface configuration, 200: HMD, 201: User, 202: Fingers, 203: Arrow, 210: Controller RA, 211: CPU, 212: RAM, 213: ROM, 214: Camera, 214a: Camera 214b: distance image camera, 214c: gaze detection device, 215: display Play, 215a: Frame, 216: Audio I / F, 217: Communication I / F, 218: Sensor, 218a: GPS receiver, 218b: Gyroscope sensor, 218c: Accelerometer, 218 d: Geomagnetic sensor, 218e: Touch sensor, 219: Bus, 220: Display control unit, 221: Spatial recognition unit, 222: Instruction reception unit, 223: Display data Generation unit, 224: Display correction unit, 228: Image acquisition unit, 229: Audio output control unit, 230: Storage device, 231: Color image data, 232: Distance image data, 233: Empty Intercognition data, 234: virtual object data, 235: additional object data, 2 36: Audio data, 237: Onomatopoeia data, 241: Frame, 300: Structure, 301: Structure, 400: Additional object, 401: Miniature (temporary) Idea object, 402: Additional object, 500: Through image

Claims

1. A display terminal equipped with a display, A color image camera that acquires a color image within a predetermined shooting range, A distance image camera that acquires a distance image of the aforementioned shooting range, Gyro sensor and Accelerometer and Touch sensor and Using the color image and the depth image, a three-dimensional map of the shooting range including structures within the shooting range is generated. Based on input from at least one of the color image camera, the distance image camera, the gyro sensor, or the accelerometer, a first line of sight direction is determined. Based on spatial recognition data, an object is displayed to place a virtual object within the 3D map. The touch sensor receives an operation instruction to place the virtual object at a location within the area on the 3D map. Based on the virtual object data corresponding to the virtual object, the depth information of the virtual object is calculated, and the depth information of the structure is calculated. The virtual object is displayed at the aforementioned position on the three-dimensional map of the shooting range. Based on the depth information of the virtual object and the depth information of the structure, the back region of the virtual object that is behind the structure is determined. Based on the depth information of the virtual object and the depth information of the structure, the back region of the virtual object that is behind the structure is identified. Based on the first line of sight direction, the virtual object, the structure, and the back region of the virtual object within the shooting range corresponding to the first line of sight direction are displayed on the display as a first display mode. Based on input from at least one of the color image camera, the distance image camera, the gyro sensor, or the accelerometer, a second line of sight direction different from the first line of sight direction is determined. Based on the second line of sight direction, the virtual object, the structure, and the back region of the virtual object within the shooting range corresponding to the second line of sight direction are displayed on the display as a second display mode different from the first display mode. A display control unit configured to perform control, A display terminal characterized by having the following features.

2. A display terminal according to claim 1, A display terminal characterized in that the aforementioned structure is a real object.

3. A display terminal according to claim 2, The aforementioned structure is a wall or a floor, and is a display terminal.

4. A display terminal according to claim 1, The display control unit, A display terminal characterized by controlling the display to show an object for positioning the virtual object along a structure within the shooting range.

5. A display terminal according to claim 4, The display control unit, A display terminal characterized by controlling the display to deform and display an object for positioning the virtual object according to the structure within the shooting range.

6. A display terminal according to claim 1, The display control unit, A display terminal characterized by being controlled to display associated strings along with objects for arranging the virtual object.

7. A display terminal according to claim 1, The display control unit, A display terminal characterized by controlling the display to highlight objects for arranging the virtual objects.

8. A display terminal according to claim 1, Furthermore, the display terminal is characterized by comprising a storage device configured to record spatial recognition data, which includes a three-dimensional map of the shooting range and position information on the three-dimensional map of the shooting range for displaying the virtual object.

9. A display terminal according to claim 1, A display terminal characterized in that the virtual object displayed on the display in the first display mode or the second display mode is displayed on the structure.

10. A display terminal according to claim 1, The display control unit, A display terminal characterized by being controlled to display all shapes of the back region of the virtual object.

11. A display terminal equipped with a display, A color image camera that acquires a color image within a predetermined shooting range, A distance image camera that acquires a distance image of the aforementioned shooting range, Gyro sensor and Accelerometer and Touch sensor and Using the color image and the distance image, a three-dimensional map of the shooting range including the first structure within the shooting range is generated. Based on input from at least one of the color image camera, the distance image camera, the gyro sensor, or the accelerometer, a first line of sight direction is determined. Based on spatial recognition data, an object is displayed to place a virtual object within the 3D map. The touch sensor receives an operation instruction to place the virtual object at a location within the area on the 3D map. Based on the virtual object data corresponding to the virtual object, the depth information of the virtual object is calculated, and the depth information of the first structure is calculated. The virtual object is displayed at the aforementioned position on the three-dimensional map of the shooting range. Based on the depth information of the virtual object and the depth information of the first structure, the back region of the virtual object that is behind the first structure is determined. Based on the depth information of the virtual object and the depth information of the first structure, the back region of the virtual object that is behind the first structure is identified. Based on the first line of sight direction, the virtual object, the first structure, and the back region of the virtual object within the shooting range corresponding to the first line of sight direction are displayed on the display. A display control unit configured to perform control, A display terminal characterized by having the following features.

12. A display terminal according to claim 11, A display terminal characterized in that the first structure is a physical object.

13. A display terminal according to claim 12, A display terminal characterized in that the first structure is a wall or a floor.

14. A display terminal according to claim 11, The display control unit, A display terminal characterized by being controlled to display an object for positioning the virtual object along the first structure within the shooting range.

15. A display terminal according to claim 14, The display control unit, A display terminal characterized by controlling the display to deform and display an object for positioning the virtual object according to the first structure within the shooting range.

16. A display terminal according to claim 11, The display control unit, A display terminal characterized by being controlled to display associated strings along with objects for arranging the virtual object.

17. A display terminal according to claim 11, The display control unit, A display terminal characterized by controlling the display to highlight objects for arranging the virtual objects.

18. A display terminal according to claim 11, Furthermore, the display terminal is characterized by comprising a storage device configured to record spatial recognition data, which includes a three-dimensional map of the shooting range and position information on the three-dimensional map of the shooting range for displaying the virtual object.

19. A display terminal according to claim 11, A display terminal characterized in that the virtual object displayed on the display is displayed on the first structure.

20. A display terminal according to claim 11, The display control unit, A display terminal characterized by being controlled to display all shapes of the back region of the virtual object.

Images