Wearable terminal device, program, and display method

Abstract

To provide a wearable terminal device capable of recognizing a virtual image outside a user's view region in VR and MR.SOLUTION: A wearable terminal device that a user wears and uses comprises at least one processor. The at least one processor detects a view region of the user in space. The at least one processor lets a display part display a first virtual image, located inside the view region, of the virtual image located in space. The at least one processor lets the display part make a display showing the presence of a second virtual image in a predetermined manner when the there is the second virtual image located outside the view region.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a wearable terminal device, a program, and a display method. [Background technology]

[0002] Conventionally, VR (Virtual Reality), MR (Mixed Reality), and AR (Augmented Reality) are known as technologies that allow a user to experience virtual images and / or virtual spaces using a wearable terminal device that the user wears on the head. A wearable terminal device has a display unit that covers the user's field of vision when worn by the user. By displaying virtual images and / or virtual spaces on this display unit according to the user's position and orientation, a visual effect is realized that makes it seem as if these images and / or virtual spaces are actually present (for example, Patent Document 1 and Patent Document 2).

[0003] MR is a technology that allows users to experience mixed reality, which is a fusion of real space and virtual images, by displaying a virtual image in a specific location in the real space while allowing the user to view the real space. VR is a technology that allows users to experience as if they are in a virtual space by allowing them to view a virtual space instead of the real space in MR.

[0004] The virtual image displayed in VR and MR has a fixed display position in the space in which the user is located, and is displayed on the display unit and viewed by the user when the display position is within the user's visual field. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 2019 / 0087021 [Patent Document 2] US Patent Application Publication No. 2019 / 0340822 Summary of the Invention

[0006] A wearable terminal device of the present disclosure is a wearable terminal device worn by a user and includes at least one processor. The at least one processor detects a visual recognition area of ​​the user in a space. The at least one processor causes a display unit to display a first virtual image located inside the visual recognition area among virtual images located in the space. When there is a second virtual image located outside the visual recognition area, the at least one processor causes the display unit to display the second virtual image based on a first operation. and when a second operation is performed while the second virtual image is being displayed on the display unit based on the first operation, at least a part of the second virtual image is moved to an original position along a path passing in front of the user. .

[0007] The program disclosed herein causes a computer capable of controlling a display unit of a wearable terminal device worn by a user to execute a process of detecting a visual recognition area of ​​the user in a space. The program causes the computer to execute a process of displaying, on the display unit, a first virtual image located inside the visual recognition area among virtual images located in the space. When there is a second virtual image located outside the visual recognition area, the program causes the computer to execute a process of displaying, on the display unit, the second virtual image based on a first operation. a process of moving at least a part of the second virtual image to an original position along a path passing in front of the user when a second operation is performed while the second virtual image is being displayed on the display unit based on the first operation; Execute the command.

[0008] The display method of the present disclosure is a display method for a wearable terminal device worn by a user. In the display method, a visual recognition area of ​​the user in a space is detected. In the display method, a first virtual image located inside the visual recognition area among virtual images located in the space is displayed on a display unit. In the display method, when there is a second virtual image located outside the visual recognition area, the second virtual image is displayed on the display unit based on a first operation. and when a second operation is performed while the second virtual image is being displayed on the display unit based on the first operation, at least a part of the second virtual image is moved to an original position along a path passing in front of the user. . [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic perspective view showing a configuration of a wearable terminal device according to a first embodiment. [Diagram 2]1A to 1C are diagrams illustrating an example of a visual recognition area and a virtual image visually recognized by a user wearing a wearable terminal device. [Diagram 3] FIG. 1 is a diagram illustrating a visual recognition area in space. [Figure 4] FIG. 2 is a block diagram showing a main functional configuration of the wearable terminal device. [Diagram 5] 13 is a flowchart showing a control procedure for virtual image display processing. [Figure 6] 13A and 13B are diagrams showing a list screen for notifying a user of the presence of a second virtual image, and an indicator that is displayed in response to an operation on the list screen. [Figure 7] 13A to 13C are diagrams illustrating changes in display modes in response to operations on a list screen. [Figure 8] 13A and 13B are diagrams illustrating a copying operation of a virtual image in response to an operation on the list screen. [Figure 9] 13A and 13B are diagrams illustrating a moving operation of a virtual image in response to an operation on the list screen. [Figure 10] 13A and 13B are diagrams illustrating a deleting operation of a virtual image in response to an operation on the list screen. [Figure 11] FIG. 13 is a diagram showing another example of the list screen. [Figure 12] 13A to 13C are diagrams illustrating a display operation for making the user aware of the presence of a second virtual image by means of an indicator. [Figure 13] 13 is a flowchart showing a control procedure for virtual image display processing. [Figure 14] 13A and 13B are diagrams illustrating an operation of moving a second virtual image into a visible area to make the presence of the second virtual image recognized. [Figure 15] FIG. 13 illustrates an example of aligning virtual images in a face-up state. [Figure 16] FIG. 13 illustrates an example of aligning virtual images in a backside-facing state. [Figure 17] 11A-11C are diagrams illustrating examples of aligning a first virtual image and a second virtual image according to different rules. [Figure 18] FIG. 13 is a diagram showing an example of displaying a scroll screen. [Figure 19] FIG. 13 is a diagram showing an example in which a second virtual image is displayed superimposed on a first virtual image. [Figure 20] FIG. 13 is a diagram showing an example in which a second virtual image is displayed in a predetermined emphasis mode. [Figure 21] 11 is a diagram showing an example in which a first virtual image is displayed in a predetermined suppressed manner. FIG. [Figure 22] 13A and 13B are diagrams illustrating a display operation for moving a first virtual image and a second virtual image so that the first virtual image and the second virtual image do not overlap each other. [Figure 23] 13A and 13B are diagrams illustrating a display operation for returning a second virtual image to its original position. [Figure 24] FIG. 13 shows the line associated with the second virtual image returned to its original position. [Diagram 25] FIG. 13 is a diagram showing an example of a list image when there are multiple spaces. [Figure 26] 13A and 13B are diagrams illustrating the operation of moving a second virtual image into the viewing area when there are multiple spaces. [Figure 27] FIG. 11 is a schematic diagram showing the configuration of a display system according to a second embodiment. [Figure 28] 1 is a block diagram showing a main functional configuration of an information processing device; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described with reference to the drawings. However, for the sake of convenience, each of the drawings referred to below shows only the main components necessary for explaining the embodiments in a simplified manner. Therefore, the wearable terminal device 10 and the information processing device 20 of the present disclosure may include any components not shown in each of the drawings referred to.

[0011] [First embodiment] As shown in FIG. 1, the wearable terminal device 10 includes a main body 10a, a visor 141 (display member) attached to the main body 10a, and the like.

[0012] The main body 10a is an annular member whose circumference is adjustable. Various devices such as a depth sensor 153 and a camera 154 are built in the main body 10a. When the main body 10a is worn on the head, the user's field of vision is covered by a visor 141.

[0013] The visor 141 is optically transparent. The user can view the real space through the visor 141. An image such as a virtual image is projected and displayed on a display surface of the visor 141 facing the user's eyes from a laser scanner 142 (see FIG. 4) built into the main body 10a. The user views the virtual image by light reflected from the display surface. At this time, the user also views the real space through the visor 141, so a visual effect is obtained as if the virtual image exists in the real space.

[0014] As shown in FIG. 2, in a state where the virtual image 30 is displayed, the user visually recognizes the virtual image 30 facing a predetermined direction at a predetermined position in the space 40. In this embodiment, the space 40 is a real space visually recognized by the user through a visor 141. Since the virtual image 30 is projected onto the visor 141 having optical transparency, it is visually recognized as a semi-transparent image overlapping the real space. In FIG. 2, a planar window screen is illustrated as the virtual image 30, but the present invention is not limited thereto, and the virtual image 30 may be, for example, an object such as an arrow, or may be various three-dimensional images. When the virtual image 30 is a window screen, the virtual image 30 has a front surface (first surface) and a back surface (second surface), and necessary information is displayed on the front surface, and usually, no information is displayed on the back surface.

[0015] The wearable terminal device 10 detects the visual recognition area 41 of the user based on the position and orientation of the user in the space 40 (in other words, the position and orientation of the wearable terminal device 10). As shown in FIG. 3, the visual recognition area 41 is an area in the space 40 located in front of the user U wearing the wearable terminal device 10. For example, the visual recognition area 41 is an area within a predetermined angular range in the left-right direction and the up-down direction from the front of the user U. In this case, the shape of the cut edge when a solid corresponding to the shape of the visual recognition area 41 is cut by a plane perpendicular to the front direction of the user U is rectangular. Note that the shape of the visual recognition area 41 may be determined so that the shape of the cut edge is other than rectangular (for example, circular or elliptical). The shape of the visual recognition area 41 (for example, the angular range in the left-right direction and the up-down direction from the front) can be specified by, for example, the following method.

[0016] In the wearable terminal device 10, the field of view is adjusted (hereinafter, referred to as calibration) in a predetermined procedure at a predetermined timing such as the initial startup. This calibration specifies the range that the user can see, and thereafter, the virtual image 30 is displayed within this range. The shape of the visible range specified by this calibration can be the shape of the visible area 41.

[0017] Furthermore, the calibration is not limited to being performed according to the above-mentioned predetermined procedure, and may be performed automatically during normal operation of the wearable terminal device 10. For example, when the user does not react to a display that should be reacted to, the range in which the display is being performed may be considered to be outside the range of the user's field of view, and the field of view (and the shape of the visual recognition area 41) may be adjusted. Also, a trial display may be performed at a position that is determined to be outside the range of the field of view, and when the user reacts to the display, the range in which the display is being performed may be considered to be within the range of the user's field of view, and the field of view (and the shape of the visual recognition area 41) may be adjusted.

[0018] The shape of the visible region 41 may be determined and fixed in advance at the time of shipment, etc., without being based on the result of adjusting the field of view. For example, the shape of the visible region 41 may be determined to be the maximum displayable range in terms of the optical design of the display unit 14.

[0019] The virtual image 30 is generated in a state where the display position and orientation in the space 40 are determined in response to a predetermined operation by the user. The wearable terminal device 10 projects and displays, on the visor 141, the virtual image 30 whose display position is determined to be within the visible region 41, among the generated virtual images 30. In FIG. 2, the visible region 41 is indicated by a dashed line.

[0020] The display position and orientation of the virtual image 30 on the visor 141 are updated in real time according to the change in the user's viewing area 41. That is, the display position and orientation of the virtual image 30 change according to the change in the viewing area 41 so that the user recognizes that "the virtual image 30 is located in the space 40 at the set position and orientation." For example, when the user moves from the front side of the virtual image 30 to the back side, the shape (angle) of the virtual image 30 displayed according to this movement gradually changes. Also, when the user turns toward the direction of the virtual image 30 after going around to the back side of the virtual image 30, the back side of the virtual image 30 is displayed so that the back side of the virtual image 30 is visible. Also, according to the change in the viewing area 41, the virtual image 30 whose display position is out of the viewing area 41 is no longer displayed, and if there is a virtual image 30 whose display position is in the viewing area 41, the virtual image 30 is newly displayed.

[0021] 2, when a user holds out his / her hand (or finger) in front, the direction in which the hand is extended is detected by the wearable terminal device 10, and a virtual line 51 extending in that direction and a pointer 52 are displayed on the display surface of the visor 141 and visually recognized by the user. The pointer 52 is displayed at the intersection of the virtual line 51 and the virtual image 30. If the virtual line 51 does not intersect with the virtual image 30, the pointer 52 may be displayed at the intersection of the virtual line 51 and a wall surface or the like of the space 40. If the distance between the user's hand and the virtual image 30 is within a predetermined reference distance, the display of the virtual line 51 may be omitted, and the pointer 52 may be directly displayed at a position corresponding to the position of the user's fingertip.

[0022] By changing the direction in which the user extends his / her hand, the direction of the virtual line 51 and the position of the pointer 52 can be adjusted. By performing a predetermined gesture in a state in which the pointer 52 is adjusted so as to be positioned on a predetermined operation target (for example, the function bar 31, the window shape change button 32, the close button 33, etc.) included in the virtual image 30, the gesture is detected by the wearable terminal device 10, and a predetermined operation can be performed on the operation target. For example, by performing a gesture of selecting the operation target (for example, a gesture of pinching the fingertips) in a state in which the pointer 52 is aligned with the close button 33, the virtual image 30 can be closed (deleted). Also, by performing a gesture of selecting the operation target in a state in which the pointer 52 is aligned with the function bar 31, and performing a gesture of moving the hand back and forth and left and right while in the selected state, the virtual image 30 can be moved in the depth direction and left and right directions. The operations on the virtual image 30 are not limited to these.

[0023] In this way, the wearable terminal device 10 of the present embodiment realizes a visual effect as if the virtual image 30 exists in real space, and can accept a user's operation on the virtual image 30 and reflect it in the display of the virtual image 30. That is, the wearable terminal device 10 of the present embodiment provides MR.

[0024] Next, the functional configuration of the wearable terminal device 10 will be described with reference to FIG. The wearable terminal device 10 includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, a display unit 14, a sensor unit 15, a communication unit 16, and the like, and these units are connected via a bus 17. All of the components shown in Fig. 4 except for the visor 141 of the display unit 14 are built into the main body unit 10a and operate using power supplied from a battery also built into the main body unit 10a.

[0025] The CPU 11 is a processor that performs various calculation processes and controls the overall operation of each part of the wearable terminal device 10. The CPU 11 performs various control operations by reading and executing a program 131 stored in the storage unit 13. The CPU 11 executes, for example, a visual recognition area detection process and a display control process by executing the program 131. Of these, the visual recognition area detection process is a process for detecting a visual recognition area 41 of a user in a space 40. Moreover, the display control process is a process for displaying, on the display unit 14, a virtual image 30 whose position in the space 40 is determined and whose position is determined within the visual recognition area 41.

[0026] 4 illustrates a single CPU 11, but is not limited to this. Two or more processors such as CPUs may be provided, and the processing executed by the CPU 11 of the present embodiment may be shared and executed by these two or more processors.

[0027] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0028] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer. The storage unit 13 stores a program 131 executed by the CPU 11, various setting data, and the like. The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. As the storage unit 13, for example, a non-volatile storage device such as an SSD (Solid State Drive) equipped with a flash memory is used.

[0029] The data stored in the storage unit 13 includes virtual image data 132 related to the virtual image 30. The virtual image data 132 includes data related to the display content of the virtual image 30 (e.g., image data), data on the display position, and data on the orientation.

[0030] The display unit 14 has a visor 141, a laser scanner 142, and an optical system that guides light output from the laser scanner 142 to the display surface of the visor 141. The laser scanner 142 irradiates the optical system with pulsed laser light, which is controlled to be turned on / off for each pixel, while scanning the optical system in a predetermined direction according to a control signal from the CPU 11. The laser light incident on the optical system forms a display screen consisting of a two-dimensional pixel matrix on the display surface of the visor 141. The method of the laser scanner 142 is not particularly limited, but for example, a method of scanning the laser light by operating a mirror using MEMS (Micro Electro Mechanical Systems) can be used. The laser scanner 142 has three light-emitting units that emit laser light of, for example, RGB colors. The display unit 14 can perform color display by projecting light from these light-emitting units onto the visor 141.

[0031] The sensor unit 15 includes an acceleration sensor 151, an angular velocity sensor 152, a depth sensor 153, a camera 154, and an eye tracker 155. The sensor unit 15 may further include a sensor not shown in FIG.

[0032] The acceleration sensor 151 detects acceleration and outputs the detection result to the CPU 11. From the detection result by the acceleration sensor 151, the translational motion of the wearable terminal device 10 in three orthogonal axial directions can be detected.

[0033] The angular velocity sensor 152 (gyro sensor) detects the angular velocity and outputs the detection result to the CPU 11. From the detection result by the angular velocity sensor 152, the rotational motion of the wearable terminal device 10 can be detected.

[0034] The depth sensor 153 is an infrared camera that detects the distance to a subject by a ToF (Time of Flight) method, and outputs the distance detection result to the CPU 11. The depth sensor 153 is provided on the front surface of the main body 10a so as to be able to capture an image of the visible area 41. By repeatedly performing measurements by the depth sensor 153 each time the position and orientation of the user change in the space 40 and synthesizing the results, it is possible to perform three-dimensional mapping of the entire space 40 (i.e., to obtain a three-dimensional structure).

[0035] The camera 154 captures an image of the space 40 using a group of RGB imaging elements, acquires color image data as the captured image, and outputs the color image data to the CPU 11. The camera 154 is provided on the front side of the main body 10a so as to capture an image of the visible area 41. The image output from the camera 154 is used to detect the position and orientation of the wearable terminal device 10, and is also used to transmit the image to an external device from the communication unit 16 and display the visible area 41 of the user of the wearable terminal device 10 on the external device.

[0036] The eye tracker 155 detects the line of sight of the user and outputs the detection result to the CPU 11. The method of detecting the line of sight is not particularly limited, but for example, a method of photographing the reflection point of near-infrared light on the user's eye with an eye tracking camera and analyzing the photographed result and the image photographed by the camera 154 to identify the object that the user is viewing can be used. A part of the configuration of the eye tracker 155 may be provided on the periphery of the visor 141, etc.

[0037] The communication unit 16 is a communication module having an antenna, a modulation / demodulation circuit, a signal processing circuit, etc. The communication unit 16 transmits and receives data to and from an external device via wireless communication in accordance with a predetermined communication protocol.

[0038] In the wearable terminal device 10 configured as above, the CPU 11 performs the following control operations.

[0039] The CPU 11 performs three-dimensional mapping of the space 40 based on distance data to the subject input from the depth sensor 153. The CPU 11 repeats this three-dimensional mapping every time the position and orientation of the user changes, and updates the result each time. The CPU 11 also performs three-dimensional mapping for each continuous space 40. Therefore, when the user moves between multiple rooms separated by walls or the like, the CPU 11 recognizes each room as one space 40 and performs three-dimensional mapping separately for each room.

[0040] The CPU 11 detects the visual recognition area 41 of the user in the space 40. In detail, the CPU 11 specifies the position and orientation of the user (wearable terminal device 10) in the space 40 based on the detection results by the acceleration sensor 151, the angular velocity sensor 152, the depth sensor 153, the camera 154, and the eye tracker 155, and the accumulated results of the three-dimensional mapping. Then, the CPU 11 detects (specifies) the visual recognition area 41 based on the specified position and orientation and the predetermined shape of the visual recognition area 41. The CPU 11 also continuously detects the position and orientation of the user in real time, and updates the visual recognition area 41 in conjunction with changes in the position and orientation of the user. The detection of the visual recognition area 41 may be performed using some of the detection results by the acceleration sensor 151, the angular velocity sensor 152, the depth sensor 153, the camera 154, and the eye tracker 155.

[0041] CPU 11 generates virtual image data 132 related to virtual image 30 in response to a user operation. That is, when CPU 11 detects a predetermined operation (gesture) instructing generation of virtual image 30, CPU 11 specifies the display content (e.g., image data), display position, and orientation of the virtual image, and generates virtual image data 132 including data representing the specified results.

[0042] The CPU 11 displays the virtual image 30, the display position of which is determined within the visible region 41, on the display unit 14. Hereinafter, the virtual image 30, the display position of which is determined within the visible region 41, i.e., the virtual image 30 located within the visible region 41, will also be referred to as the "first virtual image 30A." The virtual image 30, the display position of which is determined outside the visible region 41, i.e., the virtual image 30 located outside the visible region 41, will also be referred to as the "second virtual image 30B." Here, the "outside the visible region 41" includes a space 40 separate from the space 40 in which the user is located. The CPU 11 identifies the first virtual image 30A based on the information on the display position included in the virtual image data 132, and generates image data of the display screen to be displayed on the display unit 14 based on the positional relationship between the visible region 41 at that time and the display position of the first virtual image 30A. The CPU 11 causes the laser scanner 142 to perform a scanning operation based on this image data, and forms a display screen including the first virtual image 30A on the display surface of the visor 141. That is, the CPU 11 causes the first virtual image 30A to be displayed on the display surface of the visor 141 so that the first virtual image 30A is visible in the space 40 visible through the visor 141. The CPU 11 continuously performs this display control process, thereby updating the display content of the display unit 14 in real time in accordance with the user's movement (changes in the visible area 41). If the setting is such that the virtual image data 132 is retained even when the wearable terminal device 10 is in a power-off state, the next time the wearable terminal device 10 is started up, the existing virtual image data 132 is read, and if there is a first virtual image 30A located inside the visible area 41, it is displayed on the display unit 14.

[0043] Note that the virtual image data 132 may be generated based on instruction data acquired from an external device via the communication unit 16, and the virtual image 30 may be displayed based on the virtual image data 132. Alternatively, the virtual image data 132 itself may be acquired from an external device via the communication unit 16, and the virtual image 30 may be displayed based on the virtual image data 132. For example, an image of the camera 154 of the wearable terminal device 10 may be displayed on an external device operated by a remote instructor, and an instruction to display the virtual image 30 may be received from the external device, and the instructed virtual image 30 may be displayed on the display unit 14 of the wearable terminal device 10. This allows, for example, an operation in which the virtual image 30 showing the work content is displayed near a work target object and the remote instructor instructs the user of the wearable terminal device 10 to perform the work.

[0044] CPU 11 detects the position and orientation of the user's hand (and / or finger) based on the images captured by depth sensor 153 and camera 154, and displays virtual line 51 extending in the detected direction and pointer 52 on display unit 14. CPU 11 also detects a gesture of the user's hand (and / or finger) based on the images captured by depth sensor 153 and camera 154, and executes processing according to the content of the detected gesture and the position of pointer 52 at that time.

[0045] Next, the operation of the wearable terminal device 10 when the second virtual image 30B is outside the visible area 41 will be described.

[0046] As described above, in the wearable terminal device 10, the first virtual image 30A, whose display position is determined within the visible area 41, is displayed on the display unit and viewed by the user. For this reason, in the past, there was a problem that the user could not check whether the second virtual image 30B was outside the visible area 41 while remaining in that position. Once the virtual image 30 is generated, it remains in the space 40 until it is deleted. For this reason, if the user moves while the virtual image 30 is generated, it becomes difficult to grasp where and what kind of virtual image 30 is located, and the above problem becomes a problem. In particular, in a case where the virtual image 30 (virtual image data 132) is set not to be deleted even when the wearable terminal device 10 is turned off, it is inconvenient that the existing second virtual image 30B outside the visible area 41 cannot be recognized when the wearable terminal device 10 is started again.

[0047] Therefore, when there is a second virtual image 30B located outside the visible region 41, the CPU 11 of the wearable terminal device 10 of this embodiment causes the display unit 14 to perform a display indicating the presence of the second virtual image 30B in a predetermined manner. This allows the user to easily recognize that the second virtual image 30B is located outside the visible region 41 without changing his / her posture. For example, when an output image from the camera 154 is displayed on an external device, the area displayed on the external device may be considered to be inside the visual recognition area 41, and the area not displayed on the external device may be considered to be outside the visual recognition area 41. In other words, the visual recognition area 41 recognized by the wearable terminal device 10 may match the output image from the camera 154 displayed on the external device. Furthermore, when the viewing angle (field angle) of the camera 154 and the viewing angle of a human do not match, the visual recognition area 41 recognized by the wearable terminal device 10 may not be the same as the output image from the camera 154. That is, when the viewing angle (field angle) of the camera 154 is wider than the viewing angle of a human, the visual recognition area 41 recognized by the wearable terminal device 10 may be an area corresponding to a part of the output image from the camera 154 displayed on the external device. Furthermore, the visual field of a human can be roughly divided into an effective visual field (generally, the effective visual field using both the left and right eyes is about 60 degrees horizontally and about 40 degrees vertically) in which a human can maintain high visual acuity and recognize fine objects, and a peripheral visual field (a range in which fine objects cannot be recognized) outside the effective visual field. The visual recognition area 41 may be defined to correspond to the effective visual field, or may be defined to correspond to a visual field including the peripheral visual field (generally, the visual field using both the left and right eyes is about 200 degrees horizontally and about 130 degrees vertically). The visual area 41 may include an area defined to correspond to the effective visual field and an area defined to correspond to the visual field including the peripheral visual field, and the CPU 11 of the wearable terminal device 10 may appropriately change which definition the visual area 41 is based on depending on a predetermined condition (for example, a mode change by a predetermined user operation, etc.). Hereinafter, examples of displays indicating the presence of second virtual image 30B will be described with reference to FIGS.

[0048] First, a control procedure by the CPU 11 of a virtual image display process according to one aspect of the present disclosure will be described with reference to the flowchart of Fig. 5. The virtual image display process of Fig. 5 includes at least a feature of displaying a predetermined list screen 61 (see Fig. 6) when a second virtual image 30B is present.

[0049] When the virtual image display process shown in FIG. 5 is started, the CPU 11 detects the visual recognition area 41 based on the position and orientation of the user (step S101).

[0050] The CPU 11 determines whether or not there is a first virtual image 30A whose display position is determined within the detected visible area 41 (step S102), and if it is determined that there is a first virtual image 30A ("YES" in step S102), it causes the first virtual image 30A to be displayed on the display unit 14 (step S103).

[0051] When step S103 is completed, or when it is determined in step S102 that there is no first virtual image 30A ("NO" in step S102), the CPU 11 determines whether or not there is a second virtual image 30B whose display position is determined outside the visible area 41 (step S104). When it is determined that there is a second virtual image 30B ("YES" in step S104), the CPU 11 causes the display unit 14 to display a predetermined list screen 61.

[0052] When step S105 is completed, or when it is determined in step S104 that the second virtual image 30B is not present ("NO" in step S104), the CPU 11 determines whether or not an instruction to end the display operation by the wearable terminal device 10 has been given (step S106). When it is determined that the instruction has not been given ("NO" in step S106), the CPU 11 returns the process to step S101, and when it is determined that the instruction has been given ("YES" in step S106), the CPU 11 ends the virtual image display process.

[0053] A specific display operation of the list screen 61 in step S105 will be described below.

[0054] As shown in FIG. 6, when there is a second virtual image 30B located outside the viewing area 41, the CPU 11 causes the display unit 14 to display a list screen 61 including a list of the first virtual image 30A and the second virtual image 30B. In FIG. 6, three first virtual images 30A (images a to c) are displayed inside the viewing area 41, and one second virtual image 30B (image d) exists outside the viewing area 41. In the state of FIG. 6, image d is not displayed on the display unit 14. In this case, the list screen 61 in which images a to d are listed is displayed in the viewing area 41. This allows the user to easily recognize that image d exists outside the viewing area 41.

[0055] The list screen 61 may be in any display form as long as the user can recognize that the images a to d are listed. For example, the list screen 61 may display the file names of the images a to d, icons representing the images a to d, reduced images of the images a to d, or a combination of these.

[0056] The display position of the list screen 61 on the display unit 14 may be fixed regardless of the position and orientation of the user in the space 40. In other words, the display position of the list screen 61 in the space 40 may not be set (fixed), and even if the visual recognition area 41 moves, the list screen 61 may continue to be displayed at a predetermined position on the display surface of the visor 141. This allows the user to always view the list screen 61 regardless of his or her own position and orientation.

[0057] As shown in FIG. 6, the CPU 11 may display an indicator 62 indicating the direction in which one of the second virtual images 30B (here, image d) included in the list screen 61 is located on the display unit 14 in response to a predetermined operation on the second virtual image 30B (here, image d). By displaying the indicator 62, the user can intuitively grasp the direction in which the image d is located. In FIG. 6, the predetermined operation is an operation of tapping the item of the image d on the list screen 61 with a finger, but is not limited to this, and may be, for example, an operation of selecting the item of the image d using the pointer 52. The shape and display mode of the indicator 62 may be any shape and display mode that can indicate the direction in which the second virtual image 30B is located, and is not limited to those shown in FIG. 6.

[0058] As shown in FIG. 7, the CPU 11 may change the display mode of one virtual image 30 (here, image a) included in the list screen 61 in response to a predetermined operation on the virtual image 30. Specifically, the CPU 11 changes the color of the image a (for example, makes the color darker) in response to an operation of tapping an item of the image a on the list screen 61 to highlight the image a. This allows the user to intuitively grasp the position of the image a. The change in the display mode is not limited to highlighting, and may be, for example, changing the size of the virtual image 30, blinking the virtual image 30, changing the orientation of the virtual image 30 to face the user, moving the virtual image 30 to a position near the user where it is easy to see, or displaying a predetermined mark near the virtual image 30. The above-mentioned predetermined operation may be an operation of selecting an item on the list screen 61 using the pointer 52. When an operation is performed on an item of the second virtual image 30B on the list screen 61, the display mode of the second virtual image 30B may be changed. This allows the user to easily recognize the second virtual image 30B when the second virtual image 30B enters the visible region 41.

[0059] As shown in FIG. 8, the CPU 11 may copy one virtual image 30 (here, image d) included in the list screen 61 in response to a predetermined copy operation on the one virtual image 30 and display the copy on the display unit 14. Here, the copy operation includes, for example, a drag operation and a drop operation on an item of the one virtual image 30 included in the list screen 61. In this case, the CPU 11 copies the one virtual image 30 at the position where the drop operation is performed and displays the copy on the display unit 14. Here, what is copied is not the item (file name, icon, etc.) of the image d included in the list screen 61, but the image d itself located outside the visual recognition area 41. This allows the user to check the contents of the second virtual image 30B outside the visual recognition area 41 without changing his / her position or orientation. Note that in cases where the item of the image d included in the list screen 61 includes at least a part of the contents of the image d itself (for example, in the case of a reduced image), the item of the image d included in the list screen 61 may be copied (enlarged as necessary) in response to the drag operation and the drop operation. Furthermore, the target of the duplication operation is not limited to the second virtual image 30B, but may be the first virtual image 30A. For example, the first virtual image 30A, which is located within the visible region 41 but is far from the user and the content of which is difficult to confirm, can be duplicated in a position close to the user, so that the content of the first virtual image 30A can be easily confirmed.

[0060] Furthermore, when CPU 11 receives an operation to edit one of the duplicated virtual images 30 (here, image d), the edited content by the operation may be reflected in the original virtual image 30, i.e., image d located outside the visible area 41. This allows the user to edit the content of the second virtual image 30B located outside the visible area 41 without changing his / her position or orientation.

[0061] As shown in FIG. 9, the CPU 11 may move one virtual image 30 (here, image d) included in the list screen 61 to a position corresponding to the move operation in response to a predetermined move operation on the one virtual image 30. Here, the move operation includes, for example, a drag operation and a drop operation on an item of the one virtual image 30 included in the list screen 61. In this case, the CPU 11 moves the one virtual image 30 to the position where the drop operation has been performed. Here, what is moved is not the item (file name, icon, etc.) of the image d included in the list screen 61, but the image d itself located outside the visual recognition area 41. This allows the user to check the contents of the second virtual image 30B located outside the visual recognition area 41 without changing his / her position or orientation. The target of the move operation is not limited to the second virtual image 30B, but may be the first virtual image 30A. For example, by moving a first virtual image 30A, which is located within the visible region 41 but is far from the user and the content of which is difficult to confirm, to a position close to the user, the content of the first virtual image 30A can be easily confirmed. The moved virtual image 30 may be able to be returned to its original position in response to a predetermined operation.

[0062] As shown in FIG. 10, the CPU 11 may delete the selected virtual image 30 from the space 40 in response to a deletion operation including an operation of selecting one or more virtual images 30 (here, images c and d) included in the list screen 61. Specifically, in the list screen 61 in the upper diagram of FIG. 10, a check box 63 for selecting the virtual image 30 of the item is displayed on the right side of the item. By selecting a delete button 64 in a state in which the check box 63 of the virtual image 30 to be deleted is checked, the checked virtual images 30 can be deleted (erased) from the space 40 all at once, as shown in the lower diagram of FIG. 10. This allows the user to easily delete unnecessary virtual images 30 without moving the virtual images 30 to a position where they can be operated. Note that the check box 63 and the delete button 64 may be displayed when called by the user. In addition, when an instruction is given to turn off the power of the wearable terminal device 10, the check box 63 and the delete button 64 may be displayed to inquire the user whether or not to delete each virtual image 30. Moreover, among the virtual images 30 included in the list screen 61, the virtual images 30 that have not been checked (selected) may be deleted from the space 40.

[0063] 11, when there is a second virtual image 30B (here, images d and e) located outside the visible region 41, the CPU 11 may cause the display unit to display a list screen 61 including a list of the second virtual images 30B. That is, the first virtual images 30A (images a to c) viewed in the visible region 41 may not be listed on the list screen 61, and only the second virtual images 30B (images d and e) that are not viewed may be listed on the list screen 61. This allows the second virtual images 30B located outside the visible region 41 to be easily grasped.

[0064] 6 to 11, as shown in FIG. 12, an indicator 62 indicating the direction in which the second virtual image 30B (here, image d) is located may be displayed on the display unit 14 without displaying the list screen 61. By displaying the indicator 62 in this manner, the user can be made aware of the presence of the second virtual image 30B. That is, the display of the indicator 62 is one aspect of "a display indicating the presence of the second virtual image 30B in a predetermined manner." The shape and display manner of the indicator 62 need only be capable of indicating the direction in which the second virtual image 30B is located, and are not limited to those shown in FIG. 12.

[0065] Next, a control procedure by CPU 11 of a virtual image display process according to another aspect of the present disclosure will be described with reference to the flowchart of Fig. 13. The virtual image display process of Fig. 13 includes at least a feature of displaying second virtual image 30B on display unit 14 (i.e., inside visible area 41) when second virtual image 30B is present and a first operation is performed by the user.

[0066] When the virtual image display process shown in FIG. 13 is started, the CPU 11 detects the visual recognition area 41 based on the position and orientation of the user (step S201).

[0067] The CPU 11 determines whether or not there is a first virtual image 30A whose display position is determined within the detected visible area 41 (step S202), and if it is determined that there is a first virtual image 30A ("YES" in step S202), it causes the first virtual image 30A to be displayed on the display unit 14 (step S203).

[0068] When step S203 is completed, or when it is determined in step S202 that there is no first virtual image 30A ("NO" in step S202), the CPU 11 determines whether there is a second virtual image 30B whose display position is determined to be outside the visible area 41 (step S204).

[0069] If it is determined that second virtual image 30B exists ("YES" in step S204), CPU 11 determines whether or not a predetermined first operation has been performed (step S205). If it is determined that the first operation has been performed ("YES" in step S205), CPU 11 moves second virtual image 30B to visible area 41 and displays it on display unit 14 (step S206).

[0070] When step S206 is completed, when it is determined in step S204 that the second virtual image 30B is not present ("NO" in step S204), or when it is determined in step S205 that the first operation has not been performed ("NO" in step S205), the CPU 11 determines whether or not an instruction to end the display operation by the wearable terminal device 10 has been given (step S207). When it is determined that the instruction has not been given ("NO" in step S207), the CPU 11 returns the process to step S201, and when it is determined that the instruction has been given ("YES" in step S207), the CPU 11 ends the virtual image display process.

[0071] A specific display operation of second virtual image 30B in step S206 will be described below.

[0072] As shown in FIG. 14, the CPU 11 displays the second virtual image 30B (here, images d and e) on the display unit 14 based on the first operation. That is, the CPU 11 moves the second virtual image 30B to the inside of the visual recognition area 41. This allows the user to recognize that the second virtual image 30B is outside the visual recognition area 41 and to check the contents of the second virtual image 30B without changing the user's position or orientation. The above-mentioned first operation can be any operation determined in advance. For example, the first operation may be an operation of making a gesture of clasping hands in a state where the pointer 52 does not overlap any of the operation targets. The position of the second virtual image 30B after the movement can be determined arbitrarily. For example, the second virtual image 30B (image d in FIG. 14) that was on the left side of the viewing area 41 may be displayed within the left half of the viewing area 41, and the second virtual image 30B (image e in FIG. 14) that was on the right side of the viewing area 41 may be displayed within the right half of the viewing area 41.

[0073] Here, CPU 11 may display second virtual image 30B at a position within a predetermined operation target range from the user's position in visible area 41. The operation target range may be determined appropriately. For example, it may be a range in which operation is possible with pointer 52 without using virtual line 51, or it may be a distance range set in advance by the user.

[0074] Furthermore, CPU 11 may change the size of second virtual image 30B (here, image e) and display it on display unit 14. As shown in Fig. 14, when image e before movement is small and difficult to view, image e may be enlarged and then moved to viewable area 41. Furthermore, the sizes of the moved second virtual images 30B may be uniform.

[0075] Furthermore, when the second operation is performed while the second virtual image 30B is being displayed on the display unit 14 based on the first operation, the CPU 11 may return at least a part of the second virtual image 30B to the original position. This allows the second virtual image 30B to be easily returned to the original arrangement after the content of the second virtual image 30B is confirmed. The second operation may be the same as the first operation, or may be predetermined as an operation different from the first operation. For example, the second operation may be a gesture of snapping fingers. Furthermore, when the first operation is performed and there is a first virtual image 30A that has moved within the visible region 41, the first virtual image 30A may be returned to the original position in response to the second operation. An arbitrary virtual image 30 may be made selectable, and the selected virtual image 30 may be returned to the original position in response to the second operation. Alternatively, the unselected virtual image 30 may be returned to the original position.

[0076] As shown in Fig. 15, the CPU 11 may align the first virtual image 30A and the second virtual image 30B in a predetermined manner. Here, the first virtual image 30A and the second virtual image 30B are arranged in a matrix. The arrangement is not limited to this, and the first virtual image 30A and the second virtual image 30B may be arranged in a line, for example. This makes it easier to view each virtual image 30.

[0077] Further, the CPU 11 may display the first virtual image 30A and the second virtual image 30B on the display unit 14 so that one of the front side (first surface) and the back side (second surface) faces the user. In FIG. 15, the second virtual image 30B is displayed so that the front side faces the user, and the orientation of the first virtual image 30A is changed. This allows the contents of the front side of each virtual image 30 to be confirmed. Alternatively, as shown in FIG. 16, the first virtual image 30A and the second virtual image 30B may be displayed so that the back side faces the user. This allows an operation such as showing the virtual image 30 to another user on the opposite side of the virtual image 30 from the user.

[0078] As shown in FIG. 17, the CPU 11 may display the first virtual image 30A on the display unit 14 in a manner that follows a first rule, and may display the second virtual image 30B on the display unit 14 in a manner that follows a second rule that is different from the first rule. In the example of FIG. 17, the first rule is "adjust the virtual image 30 so that it faces the user without inverting it". The second rule is "display it so that the front faces the user". The first rule and the second rule are not limited to these. This allows the first virtual image 30A and the second virtual image 30B to be displayed in a manner desired by the user.

[0079] Moreover, after displaying the second virtual image 30B on the display unit 14, the CPU 11 may change the display surface of at least a part of the virtual images 30 in response to a predetermined operation. For example, after displaying the first virtual image 30A and the second virtual image 30B in a state where the front and back are mixed as shown in the lower diagram of FIG. 17, the display surface may be inverted in response to a predetermined operation so that the front of all the virtual images 30 faces the user as shown in the lower diagram of FIG. 15. Moreover, the transition of the display in response to the predetermined operation is not limited to this, and may be transitioned between any two states shown in the lower diagrams of FIG. 14 to FIG. 17, for example. This allows the display to be easily transitioned to a display mode desired by the user.

[0080] The CPU 11 may also arrange the first virtual image 30A and the second virtual image 30B in an order based on a predetermined condition. For example, the virtual images 30 may be arranged in an order based on the name of the virtual image 30, an order based on the display size of the virtual image 30, an order based on the attribute of the virtual image 30, an order based on the distance between the display position of the virtual image 30 and the position of the user, an order based on the surface (front or back) of the virtual image 30 facing the user, or the like. The arrangement may be in a matrix form as shown in FIG. 15, or in a line. Also, the virtual images 30 may be arranged in the depth direction so that at least a part of each of the virtual images 30 overlaps with each other as seen by the user. This makes it easier to find the desired virtual image 30.

[0081] As shown in FIG. 18, when the CPU 11 displays a plurality of second virtual images 30B on the display unit 14, the CPU 11 may display a scroll screen 65 on the display unit 14, which can display any part of the plurality of second virtual images 30B by a scroll operation. In the scroll screen 65 shown in FIG. 18, the first virtual image 30A and the second virtual image 30B are vertically arranged in a line, and a part of this arrangement is displayed. The part displayed on the scroll screen 65 can be changed by moving a scroll bar 66 up and down. Here, the first virtual image 30A is displayed on the scroll screen 65 together with the second virtual image 30B, but only the second virtual image 30B may be displayed on the scroll screen 65. In addition, the virtual images 30 may be arranged on the scroll screen 65 in an order based on a predetermined condition as described above. By displaying such a scroll screen 65, even when a large number of second virtual images 30B exist, the second virtual image 30B can be easily confirmed.

[0082] 19, CPU 11 may display second virtual image 30B on display unit 14 so as to overlap at least a portion of first virtual image 30A. This makes it possible to display second virtual image 30B in an easily visible state while maintaining the display state of first virtual image 30A.

[0083] As shown in FIG. 20, the CPU 11 may display one of the first virtual image 30A and the second virtual image 30B on the display unit 14 in a predetermined emphasized manner that makes the first virtual image 30A and the second virtual image 30B more prominent than the other. This allows the first virtual image 30A and the second virtual image 30B to be easily distinguished from each other. FIG. 20 shows an example in which the second virtual image 30B is highlighted by changing the color of the second virtual image 30B (for example, by making the color darker). Conversely, the first virtual image 30A may be highlighted more prominently than the second virtual image 30B. The highlighting is not limited to the highlighting shown in FIG. 20, and may be, for example, a change in size of the virtual image 30, a blinking display, a change in the orientation of the virtual image 30 to face the user, a change in the orientation of the virtual image 30 to face the user, a change in the position of the virtual image 30 near the user where it is easy to see, or a predetermined mark may be displayed near the virtual image 30. The above-mentioned highlighting may be performed in Figs. 14 to 18, and Figs. 21 and 22 which will be described later.

[0084] As shown in FIG. 21, the CPU 11 may display the first virtual image 30A and the second virtual image 30B on the display unit 14 in a predetermined suppression mode that makes one of the first virtual image 30A and the second virtual image 30B less noticeable than the other. This also makes it possible to easily distinguish the first virtual image 30A from the second virtual image 30B. FIG. 21 shows an example in which the transparency of the first virtual image 30A is increased to make the first virtual image 30A less noticeable than the second virtual image 30B. Conversely, the second virtual image 30B may be made less noticeable than the first virtual image 30A. The suppression mode is not limited to the display mode in which the transparency is increased as shown in FIG. 21, and may be, for example, a mode in which the virtual image 30 is made smaller or the virtual image 30 is temporarily erased. The suppression display may be performed in FIG. 14 to FIG. 18 and FIG. 22 described later.

[0085] 22, the CPU 11 may change the positions of the virtual images 30 other than the specific virtual image 30 so as to avoid the specific virtual image 30. For example, the position of the first virtual image 30A may be changed so that the first virtual image 30A does not overlap the second virtual image 30B as viewed by the user. Also, the positions of the virtual images 30 may be changed so that all the virtual images 30 do not overlap one another as viewed by the user. This can improve the visibility of the virtual images 30.

[0086] As shown in FIG. 23, when the second operation described above is performed, the CPU 11 may return only the specific virtual image 30 to its original position. Here, the specific virtual image 30 may be, for example, a virtual image 30 designated by the user, or may be one that satisfies a predetermined condition (for example, the second virtual image 30B that was outside the visible region 41 before being moved). The specific virtual image 30 may be the first virtual image 30A that was originally inside the visible region 41 and whose position has been moved. The specific virtual image 30 may be displayed in an emphasized manner as shown in FIG. 23. Furthermore, when the second operation is performed, the CPU 11 may move the second virtual image 30B to its original position along a path 67 that passes in front of the user (in front of the user's eyes). This makes it easier to grasp that the second virtual image 30B is returning to its original position. Furthermore, it is possible to confirm which second virtual image 30B is returning to its original position.

[0087] 24, CPU 11 may display on display unit 14 a line 68 linking second virtual image 30B that has been returned to its original position with a predetermined position within visible region 41. Line 68 may be straight, or may be curved so as to increase the distance that it passes through inside visible region 41. Displaying such line 68 makes it easier to recognize that second virtual image 30B has been returned to its original position, and the direction of second virtual image 30B.

[0088] 24, CPU 11 may display second virtual image 30B associated with line 68 on display unit 14 in response to a predetermined operation (third operation) on line 68. This allows a desired second virtual image 30B among second virtual images 30B that have been returned to the outside of visible region 41 to be easily displayed again and the contents thereof to be confirmed. The third operation may be, for example, an operation of touching 68 with a finger or an operation of selecting using pointer 52, but is not limited thereto.

[0089] Each operation described with reference to FIG. 6 to FIG. 24 can also be applied to the case where the first virtual image 30A and the second virtual image 30B are located in separate spaces. In the following, an example will be described in which there are three first virtual images 30A (images a to c) positioned in the first space 40A, and two second virtual images 30B (images d and e) positioned in a second space 40B separate from the first space 40A. In such an example, when the user's own device (user) is located in the first space 40A, the CPU 11 may cause the display unit 14 to perform a display indicating the presence of the second virtual image 30B positioned in the second space 40B. In addition, when the user's own device (user) moves from the second space 40B to the first space 40A, the CPU 11 may cause the display unit 14 to perform a display indicating the presence of the second virtual image 30B positioned in the second space 40B. This allows the user to easily recognize that the second virtual image 30B exists in the space before the user moves, when the user moves from one space to another, such as when the user moves to a different room.

[0090] 25, the CPU 11 may display a list screen 61 including a list of second virtual images 30B (here, images d and e) located in the second space 40B on the display unit 14. If there is a second virtual image 30B outside the visible area 41 in the first space 40A, the second virtual image 30B may also be displayed on the list screen 61.

[0091] 26, CPU 11 may display second virtual image 30B (here, images d and e) in second space 40B on display unit 14 based on the first operation. This allows the user to check the content of second virtual image 30B in second space 40B while being in first space 40A.

[0092] Second Embodiment Next, a configuration of a display system 1 according to a second embodiment will be described. The second embodiment differs from the first embodiment in that an external information processing device 20 executes part of the processing executed by the CPU 11 of the wearable terminal device 10 in the first embodiment. Below, the differences from the first embodiment will be described, and a description of the commonalities will be omitted.

[0093] 27, the display system 1 includes a wearable terminal device 10 and an information processing device 20 (server) communicatively connected to the wearable terminal device 10. At least a part of the communication path between the wearable terminal device 10 and the information processing device 20 may be wireless communication. The hardware configuration of the wearable terminal device 10 may be the same as that of the first embodiment, but a processor for performing the same processing as that performed by the information processing device 20 may be omitted.

[0094] As shown in FIG. 28, the information processing device 20 includes a CPU 21, a RAM 22, a storage unit 23, an operation display unit 24, a communication unit 25, and the like, and these units are connected to each other via a bus .

[0095] The CPU 21 is a processor that performs various types of arithmetic processing and generally controls the operations of each unit of the information processing device 20. The CPU 21 reads and executes a program 231 stored in the storage unit 23 to perform various control operations.

[0096] The RAM 22 provides a working memory space for the CPU 21 and stores temporary data.

[0097] The storage unit 23 is a non-transitory recording medium readable by the CPU 21 as a computer. The storage unit 23 stores a program 231 executed by the CPU 21, various setting data, and the like. The program 231 is stored in the storage unit 23 in the form of a computer-readable program code. As the storage unit 23, for example, a non-volatile storage device such as an SSD equipped with a flash memory or an HDD (Hard Disk Drive) is used.

[0098] The operation display unit 24 includes a display device such as a liquid crystal display, and an input device such as a mouse and a keyboard. The operation display unit 24 displays various information such as the operation status and processing results of the display system 1 on the display device. Here, the operation status of the display system 1 may include real-time images captured by the camera 154 of the wearable terminal device 10. The operation display unit 24 also converts the user's input operation on the input device into an operation signal and outputs the operation signal to the CPU 21.

[0099] The communication unit 25 transmits and receives data by communicating with the wearable terminal device 10. For example, the communication unit 25 receives data including a part or all of the detection results by the sensor unit 15 of the wearable terminal device 10, and information related to a user's operation (gesture) detected by the wearable terminal device 10. The communication unit 25 may also be capable of communicating with devices other than the wearable terminal device 10.

[0100] In the display system 1 having such a configuration, the CPU 21 of the information processing device 20 executes at least a part of the processing executed by the CPU 11 of the wearable terminal device 10 in the first embodiment. For example, the CPU 21 may perform three-dimensional mapping of the space 40 based on the detection result by the depth sensor 153. The CPU 21 may also detect the user's visual recognition area 41 in the space 40 based on the detection result by each part of the sensor unit 15. The CPU 21 may also generate virtual image data 132 related to the virtual image 30 in response to the operation of the user of the wearable terminal device 10. The CPU 21 may also detect the position and orientation of the user's hand (and / or fingers) based on the captured image by the depth sensor 153 and the camera 154. The CPU 21 may also execute processing related to the display of the list screen 61 and / or processing for moving the second virtual image 30B to the visual recognition area 41.

[0101] The above-mentioned processing results by the CPU 21 are transmitted to the wearable terminal device 10 via the communication unit 25. The CPU 11 of the wearable terminal device 10 operates each unit (e.g., the display unit 14) of the wearable terminal device 10 based on the received processing results. The CPU 21 may also transmit a control signal to the wearable terminal device 10 to perform display control of the display unit 14 of the wearable terminal device 10.

[0102] In this way, by executing at least a part of the processing in the information processing device 20, the device configuration of the wearable terminal device 10 can be simplified and the manufacturing cost can be reduced. Furthermore, by using a higher performance information processing device 20, various processes related to MR can be performed at high speed and with high accuracy. Therefore, it is possible to improve the accuracy of 3D mapping of the space 40, improve the display quality of the display unit 14, and increase the response speed of the display unit 14 to the user's actions.

[0103] 〔others〕 The above embodiment is merely an example, and various modifications are possible. For example, in each of the above-described embodiments, the visor 141 having optical transparency is used to allow the user to view the real space, but the present invention is not limited to this. For example, the visor 141 having light blocking properties may be used to allow the user to view the image of the space 40 captured by the camera 154. That is, the CPU 11 may display, on the display unit 14, the image of the space 40 captured by the camera 154 and the first virtual image 30A superimposed on the image of the space 40. With such a configuration, MR in which the virtual image 30 is blended with the real space can also be realized.

[0104] Moreover, by using a pre-generated image of a virtual space instead of an image of a real space captured by camera 154, a VR can be realized that allows the user to experience being in the virtual space. In this VR as well, the user's viewing area 41 is specified, and a portion of the virtual space that is inside the viewing area 41 and a virtual image 30 whose display position is determined inside the viewing area 41 are displayed. Thus, as in each of the above embodiments, a display operation can be applied to show a second virtual image 30B that is outside the viewing area 41.

[0105] The wearable terminal device 10 is not limited to the one having the ring-shaped main body 10a illustrated in Fig. 1, and may have any structure as long as it has a display unit that is visible to the user when worn. For example, it may have a structure that covers the entire head like a helmet. Also, it may have a frame that is hung on the ears like glasses, and various devices may be built into the frame.

[0106] The virtual image 30 does not necessarily have to be stationary in the space 40, but may move within the space 40 along a predetermined trajectory.

[0107] Although an example in which a user's gesture is detected and accepted as an input operation has been described, the present invention is not limited to this. For example, an input operation may be accepted by a controller that the user holds in his / her hand or wears on his / her body.

[0108] In addition, the specific details of the configurations and controls shown in the above embodiments can be appropriately changed without departing from the spirit of the present disclosure. Also, the configurations and controls shown in the above embodiments can be appropriately combined without departing from the spirit of the present disclosure. [Industrial Applicability]

[0109] The present disclosure can be used in a wearable terminal device, a program, and a display method. [Explanation of symbols]

[0110] 1 Display System 10 Wearable terminal device 10a Main body 11 CPU (Processor) 12 RAM 13 Storage section 131 Programs 132 Virtual Image Data 14 Display section 141 Visor (display part) 142 Laser Scanner 15 Sensor section 151 Accelerometer 152 Angular velocity sensor 153 Depth Sensor 154 Camera 155 Eye Tracker 16 Communications Department 17 Bus 20 Information processing device 21 CPU 22 RAM 23 Memory section 231 Programs 24 Operation display section 25 Communications Department 26 Bus 30 Virtual Images 30A First display image 30B Second display image 31 Feature Bar 32 Window shape change button 33 Close button 40 space 40A First Space 40B 2nd space 41 Visibility Zone 51 Virtual Line 52 Pointer 61 List Screen 62 Indicator 63 Checkbox 64 Delete button 65 Scrolling Screen 66 Scrollbar 67 Routes 68 Line U User

Claims

1. A wearable terminal device that is worn by a user, At least one processor; The at least one processor: Detecting a visual recognition area of ​​the user in a space; displaying, on a display unit, a first virtual image located within the visible region among the virtual images located in the space; When a second virtual image is located outside the visible region, the second virtual image is displayed on the display unit based on a first operation; A wearable terminal device that, when a second operation is performed while the second virtual image is being displayed on the display unit based on the first operation, moves at least a portion of the second virtual image to its original position along a path that passes in front of the user.

2. The display unit includes a light-transmitting display member, The wearable terminal device according to claim 1 , wherein the at least one processor causes the first virtual image to be displayed on a display surface of the display member so that the first virtual image is viewed in the space viewed through the display member.

3. A camera is provided for photographing the space, The wearable terminal device according to claim 1 , wherein the at least one processor causes the display unit to display an image of the space captured by the camera and the first virtual image superimposed on the image of the space.

4. The wearable terminal device according to any one of claims 1 to 3, wherein the at least one processor displays the second virtual image at a position within a predetermined operation target range from the user's position in the visible area.

5. The wearable terminal device according to claim 1 , wherein the at least one processor changes a size of the second virtual image and displays the second virtual image on the display unit.

6. The wearable terminal device according to claim 1 , wherein the at least one processor aligns the first virtual image and the second virtual image in a predetermined manner.

7. the virtual image has a first surface and a second surface; The wearable terminal device according to any one of claims 1 to 6, wherein the at least one processor causes the first virtual image and the second virtual image to be displayed on the display unit so that one of the first surface and the second surface faces the user.

8. A wearable terminal device according to any one of claims 1 to 7, wherein the at least one processor causes the first virtual image to be displayed on the display unit in a manner that complies with a first rule, and causes the second virtual image to be displayed on the display unit in a manner that complies with a second rule that is different from the first rule.

9. the virtual image has a first surface and a second surface; The at least one processor changes the surface on which at least a portion of the virtual image is displayed in accordance with a predetermined operation after the second virtual image is displayed on the display unit. The wearable terminal device according to any one of claims 1 to 8.

10. The wearable terminal device according to any one of claims 1 to 9, wherein the at least one processor arranges the first virtual image and the second virtual image in an order based on a predetermined condition.

11. The wearable terminal device according to any one of claims 1 to 10, wherein when the at least one processor causes a plurality of the second virtual images to be displayed on the display unit, the at least one processor causes a scroll screen on the display unit that can display any portion of the plurality of second virtual images by a scroll operation.

12. The wearable terminal device according to any one of claims 1 to 11, wherein the at least one processor causes the second virtual image to be displayed on the display unit so as to overlap at least a portion of the first virtual image.

13. The wearable terminal device according to any one of claims 1 to 12, wherein the at least one processor displays one of the first virtual image and the second virtual image on the display unit in a predetermined emphasis manner that makes the first virtual image and the second virtual image more prominent than the other.

14. The wearable terminal device according to any one of claims 1 to 13, wherein the at least one processor causes the display unit to display a line linking the second virtual image returned to its original position to a predetermined position within the visual field.

15. The wearable terminal device according to claim 14 , wherein the at least one processor causes the display unit to display the second virtual image associated with the line in response to a predetermined operation on the line.

16. The at least one processor: generating the virtual image located in one of a first space and a second space separate from the first space; The wearable terminal device according to any one of claims 1 to 15, wherein when the device is located in the first space, the second virtual image located in the second space is displayed on the display unit.

17. The wearable terminal device according to claim 16 , wherein the at least one processor causes the display unit to display the second virtual image located in the second space when the wearable terminal device moves from the second space to the first space.

18. A computer capable of controlling a display unit of a wearable terminal device that is worn by a user, A process of detecting a visual recognition area of ​​the user in a space; A process of displaying, on the display unit, a first virtual image located within the visible region among the virtual images located in the space; a process of displaying a second virtual image on the display unit based on a first operation when a second virtual image is located outside the visible area; a process of moving at least a part of the second virtual image to an original position along a path passing in front of the user when a second operation is performed while the second virtual image is being displayed on the display unit based on the first operation; A program to execute.

19. A display method for a wearable terminal device worn by a user, comprising: Detecting a visual recognition area of ​​the user in a space; displaying, on a display unit, a first virtual image located within the visible region among the virtual images located in the space; When a second virtual image is located outside the visible region, the second virtual image is displayed on the display unit based on a first operation; A display method in which, when a second operation is performed while the second virtual image is being displayed on the display unit based on the first operation, at least a portion of the second virtual image is moved to its original position via a path that passes in front of the user.

Images