Wearable terminal device, program, and display method

Abstract

SOLUTION: A wearable terminal device worn by a user comprises at least one processor. The at least one processor causes a display unit to display an instruction image located in a space based on an instruction from an instructor. The at least one processor causes the display unit to display a plurality of instruction images based on instructions from the plurality of instructors in a different display mode for each instructor who gave the instruction.EFFECT: This allows a user to easily and intuitively grasp whose instructions caused each of a plurality of instruction images to be displayed.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a wearable terminal device, a program, and a display method. [Background technology]

[0002] Conventionally, VR (Virtual Reality), MR (Mixed Reality), and AR (Augmented Reality) are known as technologies that allow a user to experience virtual images and / or virtual spaces using a wearable terminal device that the user wears on the head. A wearable terminal device has a display unit that covers the user's field of vision when worn by the user. By displaying virtual images and / or virtual spaces on this display unit according to the user's position and orientation, a visual effect is realized that makes it seem as if these images and / or virtual spaces are actually present (for example, Patent Document 1 and Patent Document 2).

[0003] MR is a technology that allows users to experience mixed reality, which is a fusion of real space and virtual images, by displaying a virtual image in a specific location in the real space while allowing the user to view the real space. VR is a technology that allows users to experience as if they are in a virtual space by allowing them to view a virtual space instead of the real space in MR.

[0004] The virtual image displayed in VR and MR has a fixed display position in the space in which the user is located, and is displayed on the display unit and viewed by the user when the display position is within the user's visual field. [Prior art documents] [Patent documents]

[0005] A wearable terminal device according to the present disclosure is a wearable terminal device worn by a user and includes at least one processor. The at least one processor causes a display unit to display an instruction image located within a space based on an instruction from an instructor. The at least one processor causes the display unit to display a plurality of identification signs corresponding to a plurality of instructors and a plurality of the instruction images based on the instructions from the plurality of instructors in a different color for each instructor who has given an instruction, and in a manner that the identification signs and the instruction images corresponding to the same instructor are the same color. When the distance between the position of the user in the space and the display position of the instruction image is equal to or less than a predetermined reference distance, a different character or symbol is added to the display of the instruction image for each instructor who has made an instruction. . In addition, the wearable terminal device of the present disclosure is a wearable terminal device worn and used by a user, and is equipped with at least one processor, which causes an instruction image located within a space to be displayed on a display unit based on instructions from an instructor, and causes a plurality of identification signs corresponding to a plurality of the instructors and a plurality of the instruction images based on the instructions from the plurality of the instructors to be displayed on the display unit so that the identification signs and the instruction images corresponding to the same instructor are a different color for each instructor who gave the instructions, and so that when a predetermined operation is performed on the instruction image, a different letter or symbol is added to the display of the instruction image for each instructor who gave the instructions. Summary of the Invention

[0006] The program of the present disclosure causes a computer capable of controlling a display unit of a wearable terminal device worn by a user to execute a process of displaying, on the display unit, an instruction image located within a space based on an instruction from an instructor. The program causes the computer to execute a process of displaying, on the display unit, a plurality of identification signs corresponding to a plurality of instructors and a plurality of the instruction images based on the instructions from the plurality of instructors, in different colors for each instructor who has given an instruction, and in the same color for the identification signs and the instruction images corresponding to the same instructor. When the distance between the user's position in the space and the display position of the instruction image is equal to or less than a predetermined reference distance, a process of adding a different character or symbol to the display of the instruction image for each instructor who has made an instruction. Execute the command. In addition, the program disclosed herein causes a computer capable of controlling a display unit of a wearable terminal device worn by a user to perform the following processes: displaying, on the display unit, an instruction image located within a space based on instructions from an instructor; displaying, on the display unit, a plurality of identification signs corresponding to a plurality of instructors and a plurality of the instruction images based on instructions from the plurality of instructors, so that the identification signs and the instruction images corresponding to the same instructor are a different color; and, when a predetermined operation is performed on the instruction image, adding a different letter or symbol to the display of the instruction image for each instructor who gave the instruction.

[0007] A display method according to the present disclosure is a display method for a wearable terminal device worn by a user. In the display method, an instruction image located in a space is displayed on a display unit based on an instruction from an instructor. In the display method, a plurality of identification signs corresponding to a plurality of instructors and a plurality of the instruction images based on the instructions from the plurality of instructors are displayed on the display unit in a different color for each instructor who has given an instruction, and the identification signs and the instruction images corresponding to the same instructor are displayed in the same color. When the distance between the position of the user in the space and the display position of the instruction image is equal to or less than a predetermined reference distance, a different character or symbol is added to the display of the instruction image for each instructor who has made an instruction. . A display method according to the present disclosure is a display method for a wearable terminal device worn by a user, in which an instruction image located in a space is displayed on a display unit based on an instruction from an instructor, a plurality of identification signs corresponding to the plurality of instructors and a plurality of the instruction images based on the instructions from the plurality of instructors are displayed on the display unit in a different color for each instructor who has given an instruction, and the identification signs and the instruction images corresponding to the same instructor are displayed in the same color, and when a predetermined operation is performed on the instruction images, a different character or symbol is added to the display of the instruction images for each instructor who has given an instruction.

[0008] A display method according to the present disclosure is a display method for a wearable terminal device worn by a user. In the display method, an instruction image located in a space is displayed on a display unit based on an instruction from an instructor. In the display method, a plurality of instruction images based on instructions from the plurality of instructors are displayed on the display unit in a different display mode for each instructor who has given an instruction. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of a display system according to a first embodiment. [Diagram 2] FIG. 2 is a schematic perspective view showing a configuration of a wearable terminal device. [Diagram 3] 1A to 1C are diagrams illustrating an example of a visual recognition area and a virtual image visually recognized by a user wearing a wearable terminal device. [Figure 4] FIG. 1 is a diagram illustrating a visual recognition area in space. [Diagram 5] FIG. 2 is a block diagram showing a main functional configuration of the wearable terminal device. [Figure 6] FIG. 2 is a block diagram showing a main functional configuration of an external device. [Figure 7] 13A and 13B are diagrams showing a visual recognition area and a screen for an instructor when an action related to a remote instruction is being performed. [Figure 8] 11A and 11B are diagrams showing a visual recognition area in which an instruction image showing a trajectory of a pen input is displayed, and an instruction person screen; [Figure 9] 13 is a diagram showing a visual recognition area in which an instruction image of a document image is displayed, and a screen for an instructor; FIG. [Figure 10] 13 is a flowchart showing a control procedure of a remote instruction process. [Figure 11] 13A and 13B are diagrams showing a visual recognition area and a screen for an instructor in a remote instruction process. [Figure 12] FIG. 13 is a diagram showing a viewing area in which a color palette is displayed and a screen for an instructor. [Figure 13] FIG. 13 is a diagram showing a color palette when there is an existing instruction image. [Figure 14]13A and 13B are diagrams illustrating notification displays in response to operations on a color palette. [Figure 15] 13A and 13B are diagrams showing other examples of the display form of the instruction image in the visual recognition area and the instructor screen. [Figure 16] 13A and 13B are diagrams illustrating another example of the display operation of the instruction image in the visual recognition area and the instructor screen. [Figure 17] 13A and 13B are diagrams showing an example in which different characters are added to an instruction image for each instructor. [Figure 18] 13A and 13B are diagrams showing examples of how characters are added to a pointing image. [Figure 19] FIG. 11 is a schematic diagram showing the configuration of a display system according to a second embodiment. [Figure 20] 1 is a block diagram showing a main functional configuration of an information processing device; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described with reference to the drawings. However, for the sake of convenience, each of the drawings referred to below shows only the main components necessary for explaining the embodiments in a simplified manner. Therefore, the wearable terminal device 10, the external device 20, and the information processing device 80 of the present disclosure may include any components not shown in each of the drawings referred to.

[0011] [First embodiment] As shown in Fig. 1, the display system 1 includes a wearable terminal device 10 and a plurality of external devices 20. The display system 1 and the external devices 20 are communicatively connected via a network N. The network N may be, for example, the Internet, but is not limited to this. The display system 1 may include a plurality of wearable terminal devices 10. The display system 1 may include only one external device 20.

[0012] The wearable terminal device 10 is a device that allows a user (wearer) to experience MR. In this embodiment, a user performing a predetermined task wears the wearable terminal device 10. In addition, a remote instructor who gives instructions to the user wearing the wearable terminal device 10 from a remote location via the wearable terminal device 10 operates the external device 20.

[0013] As shown in FIG. 2, the wearable terminal device 10 includes a main body 10a, a visor 141 (display member) attached to the main body 10a, and the like.

[0014] The main body 10a is an annular member whose circumference is adjustable. Various devices such as a depth sensor 153 and a camera 154 are built in the main body 10a. When the main body 10a is worn on the head, the user's field of vision is covered by a visor 141.

[0015] The visor 141 is optically transparent. The user can view the real space through the visor 141. An image such as a virtual image is projected and displayed on a display surface of the visor 141 facing the user's eyes from a laser scanner 142 (see FIG. 5) built into the main body 10a. The user views the virtual image by light reflected from the display surface. At this time, the user also views the real space through the visor 141, so a visual effect is obtained as if the virtual image exists in the real space.

[0016] As shown in FIG. 3, in a state where the virtual image 30 is displayed, the user visually recognizes the virtual image 30 facing a predetermined direction at a predetermined position in the space 40. In this embodiment, the space 40 is a real space visually recognized by the user through a visor 141. Since the virtual image 30 is projected onto the visor 141 having optical transparency, it is visually recognized as a semi-transparent image overlapping the real space. In FIG. 3, a planar window screen is illustrated as an example of the virtual image 30. The virtual image 30 also includes an instruction image 31 for giving instructions or explanations to the user of the wearable terminal device 10. That is, the instruction image 31 is one aspect of the virtual image 30. In FIG. 3, an arrow instruction image 31 and an instruction image 31 of a trajectory of a pen input are illustrated as examples. The virtual image 30 is not limited to these, and may be, for example, various stereoscopic images. When virtual image 30 is a window screen, virtual image 30 has a front surface (first surface) and a back surface (second surface), of which necessary information is displayed on the front surface, and usually no information is displayed on the back surface.

[0017] The wearable terminal device 10 detects the visual recognition area 41 of the user based on the position and orientation of the user in the space 40 (in other words, the position and orientation of the wearable terminal device 10). As shown in FIG. 4, the visual recognition area 41 is an area in the space 40 located in front of the user A who wears the wearable terminal device 10. For example, the visual recognition area 41 is an area within a predetermined angular range in the left-right direction and the up-down direction from the front of the user A. In this case, the shape of the cut edge when a solid corresponding to the shape of the visual recognition area 41 is cut by a plane perpendicular to the front direction of the user A is rectangular. Note that the shape of the visual recognition area 41 may be determined so that the shape of the cut edge is other than rectangular (for example, circular or elliptical). The shape of the visual recognition area 41 (for example, the angular range in the left-right direction and the up-down direction from the front) can be specified by the following method, for example.

[0018] In the wearable terminal device 10, the field of view is adjusted (hereinafter, referred to as calibration) in a predetermined procedure at a predetermined timing such as the initial startup. This calibration specifies the range that the user can see, and thereafter, the virtual image 30 is displayed within this range. The shape of the visible range specified by this calibration can be the shape of the visible area 41.

[0019] Furthermore, the calibration is not limited to being performed according to the above-mentioned predetermined procedure, and may be performed automatically during normal operation of the wearable terminal device 10. For example, when the user does not react to a display that should be reacted to, the range in which the display is being performed may be considered to be outside the range of the user's field of view, and the field of view (and the shape of the visual recognition area 41) may be adjusted. Also, a trial display may be performed at a position that is determined to be outside the range of the field of view, and when the user reacts to the display, the range in which the display is being performed may be considered to be within the range of the user's field of view, and the field of view (and the shape of the visual recognition area 41) may be adjusted.

[0020] The shape of the visible region 41 may be determined and fixed in advance at the time of shipment, etc., without being based on the result of adjusting the field of view. For example, the shape of the visible region 41 may be determined to be the maximum displayable range in terms of the optical design of the display unit 14.

[0021] The virtual image 30 is generated in a state where the display position and orientation in the space 40 are determined according to a predetermined operation by the user. Furthermore, the instruction image 31 of the virtual image 30 is generated based on instruction data transmitted from the external device 20 to the wearable terminal device 10, for example. Furthermore, the instruction image 31 can be generated by the user of the wearable terminal device 10 for memorandum purposes or the like. The wearable terminal device 10 projects and displays, on the visor 141, the virtual image 30 whose display position is determined within the visible region 41, among the generated virtual images 30. In FIG. 3, the visible region 41 is indicated by a dashed line.

[0022] The display position and orientation of the virtual image 30 on the visor 141 are updated in real time according to the change in the user's viewing area 41. That is, the display position and orientation of the virtual image 30 change according to the change in the viewing area 41 so that the user recognizes that "the virtual image 30 is located in the space 40 at the set position and orientation." For example, when the user moves from the front side of the virtual image 30 to the back side, the shape (angle) of the virtual image 30 displayed according to this movement gradually changes. Also, when the user turns toward the direction of the virtual image 30 after going around to the back side of the virtual image 30, the back side of the virtual image 30 is displayed so that the back side of the virtual image 30 is visible. Also, according to the change in the viewing area 41, the virtual image 30 whose display position is out of the viewing area 41 is no longer displayed, and if there is a virtual image 30 whose display position is in the viewing area 41, the virtual image 30 is newly displayed.

[0023] 3, when a user holds out his / her hand (or finger) in front, the direction in which the hand is extended is detected by the wearable terminal device 10, and a virtual line 51 extending in that direction and a pointer 52 are displayed on the display surface of the visor 141 and visually recognized by the user. The pointer 52 is displayed at the intersection of the virtual line 51 and the virtual image 30. If the virtual line 51 does not intersect with the virtual image 30, the pointer 52 may be displayed at the intersection of the virtual line 51 and a wall surface or the like of the space 40. If the distance between the user's hand and the virtual image 30 is within a predetermined reference distance, the display of the virtual line 51 may be omitted, and the pointer 52 may be directly displayed at a position corresponding to the position of the user's fingertip.

[0024] By changing the direction in which the user extends his / her hand, the direction of the virtual line 51 and the position of the pointer 52 can be adjusted. By performing a predetermined gesture in a state in which the pointer 52 is adjusted so as to be positioned on a predetermined operation target (for example, the function bar 301, the window shape change button 302, the close button 303, etc.) included in the virtual image 30, the gesture is detected by the wearable terminal device 10, and a predetermined operation can be performed on the operation target. For example, by performing a gesture of selecting the operation target (for example, a gesture of pinching the fingertips) in a state in which the pointer 52 is aligned with the close button 303, the virtual image 30 can be closed (deleted). Also, by performing a gesture of selecting with the pointer 52 aligned with the function bar 301 and performing a gesture of moving the hand back and forth and left and right while in the selected state, the virtual image 30 can be moved in the depth direction and left and right directions. The operations on the virtual image 30 are not limited to these.

[0025] In this way, the wearable terminal device 10 of the present embodiment realizes a visual effect as if the virtual image 30 exists in real space, and can accept a user's operation on the virtual image 30 and reflect it in the display of the virtual image 30. That is, the wearable terminal device 10 of the present embodiment provides MR.

[0026] Next, the functional configuration of the wearable terminal device 10 will be described with reference to FIG. The wearable terminal device 10 includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, a display unit 14, a sensor unit 15, a communication unit 16, a microphone 17, a speaker 18, and the like, and these units are connected via a bus 19. All of the components shown in Fig. 5 except for the visor 141 of the display unit 14 are built into the main body unit 10a and operate using power supplied from a battery also built into the main body unit 10a.

[0027] The CPU 11 is a processor that performs various calculation processes and controls the overall operation of each part of the wearable terminal device 10. The CPU 11 performs various control operations by reading and executing a program 131 stored in the storage unit 13. The CPU 11 executes, for example, a visual recognition area detection process and a display control process by executing the program 131. Of these, the visual recognition area detection process is a process for detecting a visual recognition area 41 of a user in a space 40. Moreover, the display control process is a process for displaying, on the display unit 14, a virtual image 30 whose position in the space 40 is determined and whose position is determined within the visual recognition area 41.

[0028] 5 shows a single CPU 11, but is not limited thereto. Two or more processors such as CPUs may be provided, and the processing executed by the CPU 11 of the present embodiment may be shared and executed by these two or more processors.

[0029] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0030] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer. The storage unit 13 stores a program 131 executed by the CPU 11, various setting data, and the like. The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. As the storage unit 13, for example, a non-volatile storage device such as an SSD (Solid State Drive) equipped with a flash memory is used.

[0031] The data stored in the storage unit 13 includes virtual image data 132 related to the virtual image 30. The virtual image data 132 includes data related to the display contents of the virtual image 30 (e.g., image data), data on the display position, and data on the orientation. The virtual image data 132 related to the instruction image 31 of the arrow includes data on the size, color, display position, and orientation of the arrow. The virtual image data 132 related to the instruction image 31 of the trajectory of the pen input includes data on the thickness, color, display position (including the coordinates of each point of the trajectory), and data on the orientation of the trajectory. The virtual image data 132 related to the instruction image 31 of the document image includes data on the contents of the document image, the size, color, display position, and orientation of the window.

[0032] The display unit 14 has a visor 141, a laser scanner 142, and an optical system that guides light output from the laser scanner 142 to the display surface of the visor 141. The laser scanner 142 irradiates the optical system with pulsed laser light, which is controlled to be turned on / off for each pixel, while scanning the optical system in a predetermined direction according to a control signal from the CPU 11. The laser light incident on the optical system forms a display screen consisting of a two-dimensional pixel matrix on the display surface of the visor 141. The method of the laser scanner 142 is not particularly limited, but for example, a method of scanning the laser light by operating a mirror using MEMS (Micro Electro Mechanical Systems) can be used. The laser scanner 142 has three light-emitting units that emit laser light of, for example, RGB colors. The display unit 14 can perform color display by projecting light from these light-emitting units onto the visor 141.

[0033] The sensor unit 15 includes an acceleration sensor 151, an angular velocity sensor 152, a depth sensor 153, a camera 154, and an eye tracker 155. The sensor unit 15 may further include a sensor not shown in FIG.

[0034] The acceleration sensor 151 detects acceleration and outputs the detection result to the CPU 11. From the detection result by the acceleration sensor 151, the translational motion of the wearable terminal device 10 in three orthogonal axial directions can be detected.

[0035] The angular velocity sensor 152 (gyro sensor) detects the angular velocity and outputs the detection result to the CPU 11. From the detection result by the angular velocity sensor 152, the rotational motion of the wearable terminal device 10 can be detected.

[0036] The depth sensor 153 is an infrared camera that detects the distance to a subject by a ToF (Time of Flight) method, and outputs the distance detection result to the CPU 11. The depth sensor 153 is provided on the front surface of the main body 10a so as to be able to capture an image of the visible area 41. By repeatedly performing measurements by the depth sensor 153 each time the position and orientation of the user change in the space 40 and synthesizing the results, it is possible to perform three-dimensional mapping of the entire space 40 (i.e., to obtain a three-dimensional structure).

[0037] The camera 154 captures an image of the space 40 using a group of RGB imaging elements, acquires color image data as the captured image, and outputs the color image data to the CPU 11. The camera 154 is provided on the front side of the main body 10a so as to capture an image of the visible area 41. The image output from the camera 154 is used to detect the position and orientation of the wearable terminal device 10, and is also used to display the visible area 41 of the user of the wearable terminal device 10 on the external device 20 by transmitting the image from the communication unit 16 to the external device 20.

[0038] The eye tracker 155 detects the line of sight of the user and outputs the detection result to the CPU 11. The method of detecting the line of sight is not particularly limited, but for example, a method of photographing the reflection point of near-infrared light on the user's eye with an eye tracking camera and analyzing the photographed result and the image photographed by the camera 154 to identify the object that the user is viewing can be used. A part of the configuration of the eye tracker 155 may be provided on the periphery of the visor 141, etc.

[0039] The communication unit 16 is a communication module having an antenna, a modulation / demodulation circuit, a signal processing circuit, etc. The communication unit 16 transmits and receives data to and from the external device 20 via wireless communication in accordance with a predetermined communication protocol. The communication unit 16 can also communicate audio data with the external device 20. That is, the communication unit 16 transmits audio data collected by the microphone 17 to the external device 20, and receives audio data transmitted from the external device 20 to output audio from the speaker 18.

[0040] The microphone 17 converts sounds such as the user's voice into electrical signals and outputs them to the CPU 11 .

[0041] The speaker 18 converts the input audio data into mechanical vibrations and outputs the vibrations as sound.

[0042] In the wearable terminal device 10 configured as above, the CPU 11 performs the following control operations.

[0043] The CPU 11 performs three-dimensional mapping of the space 40 based on distance data to the subject input from the depth sensor 153. The CPU 11 repeats this three-dimensional mapping every time the position and orientation of the user changes, and updates the result each time. The CPU 11 also performs three-dimensional mapping for each continuous space 40. Therefore, when the user moves between multiple rooms separated by walls or the like, the CPU 11 recognizes each room as one space 40 and performs three-dimensional mapping separately for each room.

[0044] The CPU 11 detects the visual recognition area 41 of the user in the space 40. In detail, the CPU 11 specifies the position and orientation of the user (wearable terminal device 10) in the space 40 based on the detection results by the acceleration sensor 151, the angular velocity sensor 152, the depth sensor 153, the camera 154, and the eye tracker 155, and the accumulated results of the three-dimensional mapping. Then, the CPU 11 detects (specifies) the visual recognition area 41 based on the specified position and orientation and the predetermined shape of the visual recognition area 41. The CPU 11 also continuously detects the position and orientation of the user in real time, and updates the visual recognition area 41 in conjunction with changes in the position and orientation of the user. The detection of the visual recognition area 41 may be performed using some of the detection results by the acceleration sensor 151, the angular velocity sensor 152, the depth sensor 153, the camera 154, and the eye tracker 155.

[0045] CPU 11 generates virtual image data 132 related to virtual image 30 in response to a user operation. That is, when CPU 11 detects a predetermined operation (gesture) instructing generation of virtual image 30, CPU 11 specifies the display content (e.g., image data), display position, and orientation of the virtual image, and generates virtual image data 132 including data representing the specified results.

[0046] The CPU 11 displays the virtual image 30, the display position of which is determined within the visual recognition area 41, i.e., the virtual image 30 located within the visual recognition area 41, on the display unit 14. The CPU 11 specifies the virtual image 30 to be displayed based on the information on the display position included in the virtual image data 132, and generates image data of the display screen to be displayed on the display unit 14 based on the positional relationship between the visual recognition area 41 and the display position of the virtual image 30 at that time. The CPU 11 causes the laser scanner 142 to perform a scanning operation based on this image data, and forms a display screen including the virtual image 30 on the display surface of the visor 141. That is, the CPU 11 displays the virtual image 30 on the display surface of the visor 141 so that the virtual image 30 is visually recognized in the space 40 visually recognized through the visor 141. The CPU 11 continuously performs this display control process, thereby updating the display content by the display unit 14 in real time in accordance with the user's movement (change in the visual recognition area 41). If the wearable terminal device 10 is set to retain the virtual image data 132 even when the power is turned off, the next time the wearable terminal device 10 is started up, the existing virtual image data 132 is read, and if there is a virtual image 30 located within the visible area 41, it is displayed on the display unit 14.

[0047] CPU 11 detects the position and orientation of the user's hand (and / or finger) based on the images captured by depth sensor 153 and camera 154, and displays virtual line 51 extending in the detected direction and pointer 52 on display unit 14. CPU 11 also detects a gesture of the user's hand (and / or finger) based on the images captured by depth sensor 153 and camera 154, and executes processing according to the content of the detected gesture and the position of pointer 52 at that time.

[0048] As shown in FIG. 6, the external device 20 includes a CPU 21, a RAM 22, a memory unit 23, an operation display unit 24, a communication unit 25, a microphone 26, a speaker 27, etc., and each of these units is connected by a bus 28.

[0049] The CPU 21 is a processor that performs various types of arithmetic processing and generally controls the operations of each part of the external device 20. The CPU 21 reads and executes a program 231 stored in the storage unit 23 to perform various control operations.

[0050] The RAM 22 provides a working memory space for the CPU 21 and stores temporary data.

[0051] The storage unit 23 is a non-transitory recording medium readable by the CPU 21 as a computer. The storage unit 23 stores a program 231 executed by the CPU 21, various setting data, and the like. The program 231 is stored in the storage unit 23 in the form of a computer-readable program code. As the storage unit 23, for example, a non-volatile storage device such as an SSD equipped with a flash memory or an HDD (Hard Disk Drive) is used.

[0052] The operation display unit 24 includes a display device such as a liquid crystal display, and an input device such as a mouse and a keyboard. The operation display unit 24 displays various information such as the operation status and processing results of the display system 1 on the display device. The display includes, for example, an instructor screen 42 including an image of the visual recognition area 41 captured by the camera 154 of the wearable terminal device 10. The contents of the instructor screen 42 will be described in detail later. In addition, the operation display unit 24 converts the user's input operation on the input device into an operation signal and outputs the operation signal to the CPU 21.

[0053] The communication unit 25 transmits and receives data to and from the wearable terminal device 10 in accordance with a predetermined communication protocol. The communication unit 25 can also perform voice data communication with the wearable terminal device 10. That is, the communication unit 25 transmits voice data collected by the microphone 26 to the wearable terminal device 10, and receives voice data transmitted from the wearable terminal device 10 to output voice from the speaker 27. The communication unit 25 may be capable of communicating with devices other than the wearable terminal device 10.

[0054] The microphone 26 converts sounds such as the voice of a remote instructor into electrical signals and outputs them to the CPU 21 .

[0055] The speaker 27 converts input audio data into mechanical vibrations and outputs the vibrations as sound.

[0056] Next, the operation of the display system 1 will be described, focusing on the operation related to remote instruction (remote support) from a remote instructor operating the external device 20 to a user wearing the wearable terminal device 10.

[0057] In the display system 1 of the present embodiment, various data can be shared and a collaborative work can be performed by performing bidirectional data communication between the wearable terminal device 10 and one or more external devices 20. For example, by transmitting image data captured by the camera 154 of the wearable terminal device 10 to the external device 20 and displaying it on the operation display unit 24, the remote instructor can recognize in real time what the user of the wearable terminal device 10 is looking at. In addition, by transmitting voices collected by the microphone 17 of the wearable terminal device 10 and the microphone 26 of the external device 20 through bidirectional voice data communication, a voice call can be performed. Thus, the period during which the voice data communication is being performed by the wearable terminal device 10 and the external device 20 includes the period during which the user of the wearable terminal device 10 and the remote instructor are in a voice call. It is also possible for three or more parties, including the user of the wearable terminal device 10 and two or more remote instructors operating two or more external devices 20, to share camera images and perform a voice call. A remote instructor can provide instructions and assistance to the user of the wearable terminal device 10 through voice communication while viewing a real-time camera image.

[0058] Furthermore, the external device 20 can also instruct the wearable terminal device 10 to display the instruction image 31. That is, a remote instructor can cause the instruction image 31 to be displayed on the display unit 14 of the wearable terminal device 10 by performing a predetermined operation on the external device 20.

[0059] FIG. 7 illustrates a case where a user A of the wearable terminal device 10 and remote instructors B and C who operate different external devices 20 share camera images and have voice calls. FIG. 7 illustrates a visual recognition area 41 of the user A at a certain time and instructor screens 42 displayed on the two external devices 20. The instructor screen 42 is displayed based on photographed data captured by the camera 154 transmitted from the wearable terminal device 10, and usually reflects the visual recognition area 41 of the user A as is. The visual recognition area 41 and the instructor screen 42 in FIG. 7 include a valve handle as a work target. Here, an example will be described in which the user A of the wearable terminal device 10 receives instructions and support from remote instructors B and C for work on the handle.

[0060] In the visual recognition area 41 and the instructor screen 42 in FIG. 7, instruction images 31a to 31c of arrows are displayed. The instruction image 31a is displayed based on an instruction from the user A received in the wearable terminal device 10. The instruction image 31b is displayed based on an instruction from the remote instructor B received in the external device 20. The instruction image 31c is displayed based on an instruction from the remote instructor C received in another external device 20. In this way, the user A of the wearable terminal device 10 can also instruct the display of the instruction image 31. That is, the user A and the remote instructors B and C are each one of the instructors who instruct the display of the instruction image. Hereinafter, any one of the user A and the remote instructors B and C will also be simply referred to as the "instructor". The display of the instruction image 31a by the user A can be performed for the purpose of, for example, pointing out a position to confirm a work position in response to an instruction by the remote instructors B and C using the instruction images 31b and 31c.

[0061] In other words, the CPU 11 of the wearable terminal device 10 causes the display unit 14 to display the instruction image 31 located within the space 40 based on an instruction from the instructor. Similarly, the CPU 21 of the external device 20 causes the operation display unit 24 to display the instruction image 31 located within the space 40 based on an instruction from the instructor.

[0062] However, when multiple instructors each display an instruction image 31, if the display formats (color, etc.) are the same, there is a problem in that it is not possible to know who instructed each instruction image 31 to be displayed.

[0063] Therefore, the CPU 11 of the wearable terminal device 10 of this embodiment causes the display unit 14 to display a plurality of instruction images 31a to 31c based on instructions from a plurality of instructors in a different display mode for each instructor who provided the instruction. Specifically, the CPU 11 causes the display unit 14 to display the plurality of instruction images 31a to 31c in a different color for each instructor who provided the instruction. Similarly, the CPU 21 of the external device 20 causes the operation display unit 24 to display the plurality of instruction images 31a to 31c based on instructions from a plurality of instructors in a different color for each instructor who provided the instruction. This makes it possible to easily and intuitively know whose instruction each of the instruction images 31a to 31c was displayed.

[0064] The CPU 11 displays the instruction images 31a to 31c on the display unit 14 during the execution of the voice data communication via the communication unit 16. The CPU 11 also associates different colors (display modes) with the user A and the remote instructors B and C participating in the voice call by the voice data communication, and displays the instruction images 31a to 31c on the display unit 14 in the color associated with the instructor who gave the instruction. This allows the instruction images 31a to 31c to be displayed in different colors for each instructor through simple processing. Furthermore, by displaying the instruction images 31a to 31c during the execution of the voice call, the user A can perform a task in accordance with the instructions of the remote instructors B and C through the voice call while visually recognizing the instruction images 31a to 31c.

[0065] More specifically, in the voice communication, the user A and the remote instructors B and C are each associated with one of a plurality of user icons 61 (identification signs) with different colors. The user icons 61 associated with each instructor are displayed in the upper left corner of the visual recognition area 41 and the instructor screen 42 along with the name of each instructor. The CPU 11 displays the instruction image 31 on the display unit 14 in the color assigned to the user icon 61 associated with the instructor who performed the instruction. Therefore, for one instructor, the color of the user icon 61 associated with the instructor and the color of the instruction image 31 displayed based on the instruction of the instructor are the same. This allows the user to easily and intuitively understand which instructor's instruction each of the instruction images 31a to 31c was displayed based on.

[0066] An arrow icon 62, a pen input icon 63, and an image icon 64 are displayed in the visual recognition area 41 and at the top of the instructor screen 42. As shown in Fig. 7, by performing a predetermined operation while the arrow icon 62 is selected, an arrow instruction image 31 can be generated and displayed. In this manner, the instruction image 31 may include an image of a virtual object, and the virtual object may include an arrow-shaped object (arrow) whose position and direction are specified.

[0067] As shown in Fig. 8, by performing a predetermined operation while the pen input icon 63 is selected, an instruction image 31 of the trajectory of the pen input can be generated and displayed. In this way, the virtual object as the instruction image 31 may include an object representing the trajectory of the pen input. The object representing the trajectory of the pen input may be, for example, a handwritten character or a figure. The trajectory of the pen input may be identified from the detection result of the trajectory of the user's fingertip, or may be identified based on the movement trajectory of the tip of a predetermined pen-type input device held by the user or a remote instructor.

[0068] 8 is displayed based on an instruction from a user A received by the wearable terminal device 10. Moreover, an instruction image 31b is displayed based on an instruction from a remote instructor B received by the external device 20. As shown in FIG. 8, the instruction images 31a and 31b of the trajectory of the pen input are also displayed in different colors for each instructor who has performed the instruction. The color of each instruction image 31 of the trajectory of the pen input and the color of the user icon 61 may be the same for each instructor.

[0069] As shown in FIG. 9, by performing a predetermined operation while an image icon 64 is selected, an instruction image 31 of a document image can be generated and displayed. In this manner, the instruction image 31 may include a document image in a predetermined file format. The document image is displayed, for example, as a window screen as shown in FIG. 9. The document image may be an instruction sheet or manual showing the procedure of work content, etc. The file format of the document image may be, for example, a file format related to image data such as JPEG, PDF, or the file format of a file generated by any other software.

[0070] The instruction image 31a in FIG. 9 is displayed based on an instruction from a user A received by the wearable terminal device 10. The instruction image 31c is displayed based on an instruction from a remote instructor C received by the external device 20. As shown in FIG. 9, the instruction images 31a and 31c of the document image are also displayed in different colors for each instructor. Here, the colors of the function bars in the window screens of the instruction images 31a and 31c are different for each instructor, but this is not limited thereto, and the background color of the window screen may be different for each instructor. The colors of the instruction images 31 of the document image and the user icons 61 may be the same for each instructor.

[0071] Next, the remote instruction process for performing the operation of sharing the instruction image 31a as described above will be described with reference to Fig. 10 and Fig. 11. In the flowchart of Fig. 10, the control procedure by the CPU 21 of the remote instruction process executed in the external device 20 and the control procedure by the CPU 11 of the remote instruction process executed in the wearable terminal device 10 are described in parallel. Here, an example will be described in which an instruction to display the instruction image 31b is first given in the external device 20 operated by a remote instructor B, and then an instruction to display the instruction image 31a is given by a user A of the wearable terminal device 10.

[0072] 10, when the remote instruction process is started, the CPU 11 of the wearable terminal device 10 starts voice data communication with the external device 20, thereby starting a voice call. The CPU 11 also starts transmitting a camera image captured by the camera 154 to the external device 20 (step S101).

[0073] When the remote instruction process is started, the CPU 21 of the external device 20 starts voice data communication with the wearable terminal device 10, thereby starting a voice call. The CPU 21 also starts receiving a camera image captured by the camera 154 of the wearable terminal device 10 (step S102). After that, the CPU 21 displays the instructor screen 42 based on the camera image, and displays the camera image as a moving image in real time on the instructor screen 42.

[0074] The CPU 21 of the external device 20 accepts an operation by the remote instructor B for generating an instruction image 31b (hereinafter, referred to as an "instruction image generating operation") (step S103). The instruction image generating operation includes an operation for designating the size, color, display position, orientation, and the like of the instruction image 31b on the instructor screen 42 and the visual recognition area 41. While the instruction image generating operation is being accepted, the camera image on the instructor screen 42 may be temporarily halted to become a still image.

[0075] As shown in S104 of FIG. 11, the CPU 21 displays the instruction image 31b on the instructor screen 42 in accordance with the content of the instruction image generating operation (step S104).

[0076] The CPU 21 generates first instruction data for displaying the displayed instruction image 31b on the wearable terminal device 10, and transmits the first instruction data to the wearable terminal device 10 (step S105).

[0077] When the CPU 11 of the wearable terminal device 10 receives the first instruction data, as shown in S106 of FIG. 11, the CPU 11 displays the instruction image 31b in the visual recognition area 41 of the display unit 14 based on the first instruction data (step S106). In detail, the CPU 11 first generates virtual image data 132 related to the instruction image 31b based on the first instruction data. Based on the generated virtual image data 132, the CPU 11 displays the instruction image 31b at a display position in the space 40 specified by the virtual image data 132. In detail, when the display position of the instruction image 31b is inside the visual recognition area 41, the CPU 11 displays the instruction image 31b on the display unit 14. This makes it possible to display the instruction image 31b of the content desired by the remote instructor B at the position intended by the remote instructor B, and to allow the user A of the wearable terminal device 10 to view and share the instruction image 31b.

[0078] The CPU 11 accepts an instruction image generating operation from the user A (step S107).

[0079] As shown in S108 of FIG. 11, the CPU 11 displays the instruction image 31a in a different display mode (here, in a different color) from the instruction image 31b in the visual recognition area 41 of the display unit 14 according to the content of the instruction image generating operation (step S108). Here, the CPU 11 determines and displays the color of the instruction image 31a so that the color of the instruction image 31a is different from that of the instruction image 31b. That is, when an instruction to display the instruction image 31a is given by the user A, if the instruction image 31b is already displayed based on an instruction from another instructor, the CPU 11 displays a new instruction image 31a on the display unit 14 in a different display mode from the display mode of the instruction image 31b already displayed, regardless of whether or not the user A has performed an operation to specify the display mode. This allows each instruction image 31 to be displayed in a different display mode for each instructor, even if the instructor does not particularly intend to do so.

[0080] The CPU 11 generates second instruction data for displaying the displayed instruction image 31a on the external device 20, and transmits the second instruction data to the external device 20 (step S109).

[0081] 11, the CPU 21 of the external device 20 displays the instruction image 31a on the instructor screen 42 in a display mode (here, in a different color) different from that of the instruction image 31b based on the received second instruction data (step S110). The color of the instruction image 31a may be specified in the second instruction data, or the CPU 21 may determine the color of the instruction image 31a to be different from that of the instruction image 31b on the instructor screen 42.

[0082] When steps S109 and S110 are completed, the CPUs 11 and 21 end the remote instruction process.

[0083] Next, modified examples of the above embodiment will be described. In each of the following modified examples, differences from the above embodiment will be described, and descriptions of contents common to the above embodiment will be omitted. In addition, each of the following modified examples may be combined with any other modified examples.

[0084] [Variation 1] In the above embodiment, the CPU 11 or CPU 21 automatically set the display mode of each instruction image 31 so that it would be different for each instructor, but instead, the instruction image 31 may be displayed in a display mode selected by the instructor.

[0085] For example, as shown in FIG. 12, when the arrow icon 62, the pen input icon 63, or the image icon 64 is selected, a color palette 65 for selecting the color of the instruction image 31 to be generated may be displayed. FIG. 12 illustrates a state in which the color palette 65 is displayed in response to the selection of the arrow icon 62. A plurality of color icons having different colors are arranged in the color palette 65. When an instruction to display the instruction image 31 is given with one of the color icons selected, an instruction image 31 of an arrow having the color of the selected color icon is generated and displayed. For example, when the above operation is accepted in the wearable terminal device 10, the CPU 11 displays the instruction image 31 of an arrow of the selected color in the visual recognition area 41 of the display unit 14. In this way, the CPU 11 may accept an operation from the instructor to select one color from a plurality of colors (a plurality of display modes) of selection candidates, and display the instruction image 31 in the selected color on the display unit 14. This allows the instruction image 31 of the desired color to be displayed. When the above operation is accepted by the external device 20, the CPU 21 displays the instruction image 31 of the arrow of the selected color on the instructor screen .

[0086] Here, as shown in FIG. 13, when the user A of the wearable terminal device 10 selects a color using the color palette 65, if the instruction image 31b has already been displayed based on an instruction from another instructor, the color icon 65a of the color of the already displayed instruction image 31b may be excluded from the color icons of the color palette 65. In other words, the multiple colors (multiple display modes) of the selection candidates using the color palette 65 may be colors excluding the color of the instruction image 31b already displayed when the instructor makes a selection. Also, a specific color may already be selected before the user A selects a color using the color palette 65, and the specific color may be a color different from the color of the instruction image 31b already displayed. According to these, it is possible to more reliably display each instruction image 31 in a different display mode for each instructor. Note that a similar operation may also be performed in the external device 20.

[0087] Also, as shown in FIG. 14, when the instruction image 31b is already displayed based on the instruction of another instructor, when the user A selects a color icon 65a of the same color as the instruction image 31b, the notification display 66 may be performed. In other words, when the instructor attempts to select the color (display mode) of the instruction image 31b that is already displayed, the CPU 11 may cause the display unit 14 to perform a predetermined notification display 66. Here, the notification display 66 may be a character display indicating that an instruction image 31 of the same color has already been placed, as shown in FIG. 14. This can reduce the occurrence of a problem in which the instruction image 31a is unintentionally displayed in the same display mode as the instruction image 31b of another instructor. Also, as shown in FIG. 14, the CPU 11 may cause the display unit 14 to display the notification display 66 so that the notification display 66 is located closer to the user than the color palette 65 in the space 40. Also, the CPU 11 may determine the display mode of the notification display 66 based on the color of the user icon 61 associated with the user A. For example, the CPU 11 may control the notification display 66 shown in FIG. 14 so that the outer frame (the frame portion of the notification display 66 indicated by the thick line) is the color of the user icon 61 associated with the user A. After the CPU 11 causes the display unit 14 to display the notification display 66, the CPU 11 may stop the notification display 66 when a predetermined time has elapsed. When the notification display 66 is displayed, the CPU 11 may output a notification from the speaker 27 to alert the user. The notification output from the speaker 27 to alert the user may be, for example, a message such as "Arrows of the same color are arranged," a buzzer sound such as a beep, or the like. The external device 20 may also perform a similar operation.

[0088] [Modification 2] In the above embodiment, the display mode of the same instruction image 31 is the same in the visual recognition area 41 of the wearable terminal device 10 and the instructor screen 42 of the external device 20, but this is not limited thereto. For example, as shown in FIG. 15, the display mode (here, color) of the same instruction image 31 may be different between the visual recognition area 41 and the instructor screen 42. In other words, the display mode of the multiple instruction images 31 based on the instructions of multiple instructors in the wearable terminal device 10 may be determined independently of the display mode of the multiple instruction images 31 in the external device 20. In addition, the color of the user icon 61 associated with each instructor may also be determined independently in each of the wearable terminal device 10 and the external device 20. This can increase the degree of freedom in selecting the display mode in each of the wearable terminal device 10 and the external device 20.

[0089] [Modification 3] In the above embodiment, when the instruction image 31 is displayed, the display mode is determined so that the display mode is different for each instructor. That is, the display mode of each instruction image 31 is determined individually for each instruction image 31. However, this is not intended to be limited to this. For example, as shown in FIG. 16, the CPU 11 may change the color of the instruction image 31 being displayed so that the instruction images 31 based on the instructions of the multiple instructors have different colors (display modes) for each instructor who has made the instruction, after displaying the instruction image 31 in an arbitrary color (arbitrary display mode) on the display unit 14. In the example of FIG. 16, in the upper diagram, the instruction image 31a based on the instruction of the user A is once displayed in the same color as the existing instruction image 31b. After that, as shown in the lower diagram, the color of the instruction image 31a is changed at a predetermined timing so that the colors of the instruction images 31a and 31b are different for each instructor. The timing for changing the color of the instruction image 31 as shown in the lower diagram can be, for example, when a series of operations related to the instruction image 31 is completed, when a collaborative editing process related to the instruction image 31 is completed, or when data related to the instruction image 31 is saved. The external device 20 may also perform a similar operation.

[0090] [Modification 4] In the above embodiment, as a display mode of the instruction image 31 that differs for each instructor, an example in which the color of the instruction image 31 is different for each instructor has been described, but this is not limited to this. For example, as shown in FIG. 17, the CPU 11 may add different characters or symbols for each instructor to the display of multiple instruction images 31 based on the instructions of multiple instructors. The added characters may include the name, title, or abbreviation of the instructor who gave the instruction. The method of adding the characters or symbols may be a method of arranging them near the instruction image 31. Alternatively, a method may be a method of extending a leader line from each instruction image 31 and displaying the characters or symbols at the end of the leader line. This makes it possible to clearly understand whose instruction each instruction image 31 was displayed. Note that in FIG. 17, the characters or symbols are added and the colors of the instruction images 31 are also different for each instructor, but this is not limited to this, and the instruction images 31 may be distinguished by the characters or symbols and the color of the instruction images 31 may be the same.

[0091] Furthermore, when the distance between the position of user A in space 40 and the display position of instruction image 31 is equal to or less than a predetermined reference distance, CPU 11 may add a character or symbol to the display of instruction image 31. This makes it possible to clearly indicate who instructed to display instruction image 31 close to the user A's work target while avoiding cluttering the display.

[0092] 18, when a predetermined operation is performed on the instruction image 31, the CPU 11 may add a character or a symbol to the display of the instruction image 31. In FIG. 18, the predetermined operation is exemplified by tapping the instruction image 31 with a finger, but is not limited thereto and may be, for example, an operation of selecting using the pointer 52. This makes it possible to clearly indicate by whom the instruction was given to display the instruction image 31 desired by the user A while avoiding a cluttered display.

[0093] The display mode of the instruction image 31 that differs for each instructor is not limited to the above-mentioned change in color and addition of characters or symbols, but may be any display mode that allows the instruction images 31 to be distinguished from one another.

[0094] Second Embodiment Next, a configuration of a display system 1 according to a second embodiment will be described. The second embodiment differs from the first embodiment in that an external information processing device 80 executes part of the processing executed by the CPU 11 of the wearable terminal device 10 in the first embodiment. Below, the differences from the first embodiment will be described, and a description of the commonalities will be omitted.

[0095] 19, the display system 1 includes a wearable terminal device 10, a plurality of external devices 20, and an information processing device 80 (server) communicatively connected to the wearable terminal device 10. At least a part of the communication path between the wearable terminal device 10 and the information processing device 80 may be wireless communication. The hardware configuration of the wearable terminal device 10 may be the same as that of the first embodiment, but a processor for performing the same processing as that performed by the information processing device 80 may be omitted.

[0096] As shown in FIG. 20, an information processing device 80 includes a CPU 81, a RAM 82, a storage unit 83, an operation display unit 84, a communication unit 85, and the like, and these units are connected to each other via a bus 86.

[0097] The CPU 81 is a processor that performs various types of arithmetic processing and generally controls the operation of each unit of the information processing device 80. The CPU 81 reads out and executes a program 831 stored in the storage unit 83 to perform various control operations.

[0098] The RAM 82 provides a working memory space for the CPU 81 and stores temporary data.

[0099] The storage unit 83 is a non-transitory recording medium readable by the CPU 81 as a computer. The storage unit 83 stores a program 831 executed by the CPU 81, various setting data, and the like. The program 831 is stored in the storage unit 83 in the form of a computer-readable program code. As the storage unit 83, for example, an SSD equipped with a flash memory, or a non-volatile storage device such as an HDD is used.

[0100] The operation display unit 84 includes a display device such as a liquid crystal display, and an input device such as a mouse and a keyboard. The operation display unit 84 displays various information such as the operation status and processing results of the display system 1 on the display device. Here, the operation status of the display system 1 may include real-time images captured by the camera 154 of the wearable terminal device 10. The operation display unit 84 also converts the user's input operation on the input device into an operation signal and outputs the operation signal to the CPU 81.

[0101] The communication unit 85 transmits and receives data by communicating with the wearable terminal device 10. For example, the communication unit 85 receives data including a part or all of the detection results by the sensor unit 15 of the wearable terminal device 10, and information related to a user's operation (gesture) detected by the wearable terminal device 10. The communication unit 85 may also be capable of communicating with devices other than the wearable terminal device 10, such as an external device 20.

[0102] In the display system 1 configured as above, the CPU 81 of the information processing device 80 executes at least a part of the processing executed by the CPU 11 of the wearable terminal device 10 in the first embodiment. For example, the CPU 81 may perform three-dimensional mapping of the space 40 based on the detection result by the depth sensor 153. The CPU 81 may also detect the user's visual recognition area 41 in the space 40 based on the detection result by each part of the sensor unit 15. The CPU 81 may also generate virtual image data 132 related to the virtual image 30 (instruction image 31) in response to the operation of the user of the wearable terminal device 10 or based on instruction data received from the external device 20. The CPU 81 may also detect the position and orientation of the user's hand (and / or fingers) based on the captured image by the depth sensor 153 and the camera 154.

[0103] The above-mentioned processing results by the CPU 21 are transmitted to the wearable terminal device 10 via the communication unit 25. The CPU 11 of the wearable terminal device 10 operates each unit (for example, the display unit 14) of the wearable terminal device 10 based on the received processing results. The CPU 81 may also transmit a control signal to the wearable terminal device 10 to control the display of the display unit 14 of the wearable terminal device 10 and / or control the output of the speaker 18.

[0104] In this way, by executing at least a part of the processing in the information processing device 80, it is possible to simplify the device configuration of the wearable terminal device 10 and reduce the manufacturing cost. Furthermore, by using a higher performance information processing device 80, it is possible to increase the speed and accuracy of various processes related to MR. Therefore, it is possible to increase the accuracy of 3D mapping of the space 40, improve the display quality of the display unit 14, and increase the response speed of the display unit 14 to the user's actions.

[0105] 〔others〕 The above embodiment is merely an example, and various modifications are possible.

[0106] For example, in each of the above-described embodiments, the visor 141 having optical transparency is used to allow the user to view the real space, but the present invention is not limited to this. For example, the visor 141 having light blocking properties may be used to allow the user to view the image of the space 40 captured by the camera 154. That is, the CPU 11 may display on the display unit 14 the image of the space 40 captured by the camera 154 and the virtual image 30 superimposed on the image of the space 40. With such a configuration, MR in which the virtual image 30 is blended with the real space can also be realized.

[0107] Moreover, by using a pre-generated image of a virtual space instead of an image of a real space captured by camera 154, a VR can be realized that allows the user to experience being in the virtual space. In this VR as well, the user's visual recognition area 41 is specified, and a portion of the virtual space that is inside the visual recognition area 41 and a virtual image 30 whose display position is determined inside the visual recognition area 41 are displayed. Thus, as in each of the above embodiments, a display operation can be applied that displays instruction image 31 in a different display mode for each instructor.

[0108] The wearable terminal device 10 is not limited to the one having the ring-shaped main body 10a illustrated in Fig. 1, and may have any structure as long as it has a display unit that is visible to the user when worn. For example, it may have a structure that covers the entire head like a helmet. Also, it may have a frame that is hung on the ears like glasses, and various devices may be built into the frame.

[0109] The virtual image 30 does not necessarily have to be stationary in the space 40, but may move within the space 40 along a predetermined trajectory.

[0110] Although the example in which a user's gesture is detected and accepted as an input operation has been described, the present invention is not limited to this. For example, an input operation may be accepted by a controller that the user holds in his / her hand or wears on his / her body.

[0111] An example has been described in which an instruction to display the instruction image 31 is given from the external device 20 to the wearable terminal device 10 while a voice call is being performed via voice data communication, but this is not limited to this, and an instruction to display the instruction image 31 may be given without accompanying voice data communication.

[0112] Although an example of a voice call between the wearable terminal device 10 and the external device 20 has been described, this is not limiting, and a video call may be possible. In this case, a web camera for capturing an image of the remote operator may be provided in the external device 20, and image data captured by the web camera may be transmitted to the wearable terminal device 10 and displayed on the display unit 14.

[0113] The instruction image 31 is not limited to an image displayed for instructing the user of the wearable terminal device 10. The instruction image 31 includes any virtual image 30 displayed on the wearable terminal device 10 based on an instruction from an instructor (including the user himself / herself) for the purpose of being visually recognized by the user of the wearable terminal device 10.

[0114] In addition, the specific details of the configurations and controls shown in the above embodiments can be appropriately changed without departing from the spirit of the present disclosure. Also, the configurations and controls shown in the above embodiments can be appropriately combined without departing from the spirit of the present disclosure. [Industrial Applicability]

[0115] The present disclosure can be used in a wearable terminal device, a program, and a display method. [Explanation of symbols]

[0116] 1 Display System 10 Wearable terminal device 10a Main body 11 CPU (Processor) 12 RAM 13 Storage section 131 Programs 132 Virtual Image Data 14 Display section 141 Visor (display part) 142 Laser Scanner 15 Sensor section 151 Accelerometer 152 Angular velocity sensor 153 Depth Sensor 154 Camera 155 Eye Tracker 16 Communications Department 17. Mike 18 Speakers 19 Bus 20 External equipment 21 CPU 22 RAM 23 Memory section 231 Programs 24 Operation display section 25 Communications Department 26 Bus 30 Virtual Images 301 Feature Bar 302 Window shape change button 303 Close button 31, 31a~31c Instruction images 40 space 41 Visibility Zone 42 Instructor screen 51 Virtual Line 52 Pointer 61 User Icon 62 Arrow Icon 63 Pen input icon 64 Image Icons 65 Color Palettes 66 Notification display 80 Information processing device 81 CPU 82 RAM 83 Storage section 831 Programs 84 Operation display section 85 Communications Department 86 Bus A User (instructor) B, C Remote instructor (instructor) N Network

Claims

1. A wearable terminal device that is worn by a user, At least one processor; The at least one processor: Displaying an instruction image located within the space on the display unit based on an instruction from an instructor; displaying, on the display unit, a plurality of identification signs corresponding to a plurality of the instructors and a plurality of the instruction images based on instructions from the plurality of the instructors in a different color for each instructor who has given an instruction, and such that the identification signs and the instruction images corresponding to the same instructor are in the same color; A wearable terminal device that, when the distance between the user's position in the space and the display position of the instruction image is less than a predetermined reference distance, adds a different character or symbol to the display of the instruction image for each instructor who made the instruction.

2. A wearable terminal device that is worn and used by a user, At least one processor; The at least one processor: Displaying an instruction image located within the space on the display unit based on an instruction from an instructor; displaying, on the display unit, a plurality of identification signs corresponding to a plurality of the instructors and a plurality of the instruction images based on instructions from the plurality of the instructors in a different color for each instructor who has given an instruction, and such that the identification signs and the instruction images corresponding to the same instructor are in the same color; When a predetermined operation is performed on the instruction image, a different character or symbol is added to the display of the instruction image for each instructor who performed the instruction.

3. The display unit includes a light-transmitting display member, The wearable terminal device according to claim 1 or 2, wherein the at least one processor displays the instruction image on a display surface of the display member so that the instruction image is visible in the space viewed through the display member.

4. A camera is provided for photographing the space, The wearable terminal device according to claim 1 , wherein the at least one processor causes the display unit to display an image of the space captured by the camera and the instruction image superimposed on the image of the space.

5. The wearable terminal device according to claim 1 , wherein the plurality of instructors includes the user.

6. a communication unit for performing voice data communication with an external device used by the instructor other than the user, The wearable terminal device according to claim 1 , wherein the at least one processor causes the display unit to display the instruction image during execution of the voice data communication via the communication unit.

7. The at least one processor: Associating different display modes with the plurality of instructors participating in the voice call through the voice data communication; The wearable terminal device according to claim 6 , wherein the instruction image is displayed on the display unit in a display mode associated with an instructor who has issued the instruction.

8. The wearable terminal device according to claim 6 , wherein a display mode of the plurality of instruction images based on the instructions of the plurality of instructors is determined independently of a display mode of the plurality of instruction images in the external device.

9. A wearable terminal device according to any one of claims 1 to 8, wherein, when the instruction is given by the user and the instruction image is already being displayed based on an instruction from another instructor, the at least one processor causes the display unit to display a new instruction image in a display mode different from the display mode of the instruction image already being displayed, regardless of whether or not the user has performed an operation to specify the display mode.

10. the at least one processor receives, from the instructor, an operation of selecting one display mode from a plurality of display modes as selection candidates, and causes the instruction image to be displayed on the display unit in the selected display mode; The wearable terminal device according to any one of claims 1 to 9, wherein the multiple display modes of the selection candidates are display modes excluding the display mode of the instruction image that is already displayed when the instructor makes the selection.

11. The at least one processor: accepting an operation of selecting one display mode from a plurality of display modes as selection candidates from the instructor, and displaying the instruction image on the display unit in the selected display mode; A wearable terminal device according to any one of claims 1 to 9, wherein when the instructor attempts to select a display mode of the instruction image that is already displayed when the instructor makes the selection, a predetermined notification display is displayed on the display unit.

12. The wearable terminal device according to any one of claims 1 to 11, wherein the at least one processor changes the display mode of the instruction image being displayed so that the multiple instruction images based on instructions from the multiple instructors have different display modes for each instructor who gave the instructions, after displaying the instruction image on the display unit in an arbitrary display mode.

13. The wearable terminal device according to claim 1 , wherein the characters include a name, title, or abbreviation of the instructor who has given the instruction related to the display of the instruction image.

14. The wearable terminal device according to any one of claims 1 to 13, wherein the instruction image includes a document image in a predetermined file format.

15. The wearable terminal device according to claim 1 , wherein the instruction image includes an image of a virtual object.

16. The wearable terminal device according to claim 15 , wherein the virtual object includes an object that represents a trajectory of a pen input.

17. The wearable terminal device according to claim 15 or 16, wherein the virtual object includes an arrow-shaped object whose position and orientation are specified.

18. A computer capable of controlling a display unit of a wearable terminal device that is worn by a user, A process of displaying an instruction image located within a space on the display unit based on an instruction from an instructor; a process of displaying, on the display unit, a plurality of identification signs corresponding to a plurality of the instructors and a plurality of the instruction images based on instructions from the plurality of the instructors in a different color for each instructor who has given an instruction, and in a manner that the identification signs and the instruction images corresponding to the same instructor are in the same color; a process of adding a different character or symbol to the display of the instruction image for each instructor when a distance between the position of the user in the space and a display position of the instruction image is equal to or smaller than a predetermined reference distance; A program to execute.

19. A computer capable of controlling a display unit of a wearable terminal device worn by a user, A process of displaying an instruction image located within a space on the display unit based on an instruction from an instructor; a process of displaying, on the display unit, a plurality of identification signs corresponding to a plurality of the instructors and a plurality of the instruction images based on instructions from the plurality of the instructors in a different color for each instructor who has given an instruction, and in a manner that the identification signs and the instruction images corresponding to the same instructor are in the same color; When a predetermined operation is performed on the instruction image, a process of adding a different character or symbol to the display of the instruction image for each person who performed the instruction; A program to execute.

20. A display method for a wearable terminal device worn by a user, comprising: Displaying an instruction image located within the space on the display unit based on an instruction from an instructor; displaying, on the display unit, a plurality of identification signs corresponding to a plurality of the instructors and a plurality of the instruction images based on instructions from the plurality of the instructors in a different color for each instructor who has given an instruction, and such that the identification signs and the instruction images corresponding to the same instructor are in the same color; A display method in which, when the distance between the user's position in the space and the display position of the instruction image is less than a predetermined reference distance, a different character or symbol is added to the display of the instruction image for each instructor who made the instruction.

21. A display method for a wearable terminal device worn by a user, comprising: Displaying an instruction image located within the space on the display unit based on an instruction from an instructor; displaying, on the display unit, a plurality of identification signs corresponding to a plurality of the instructors and a plurality of the instruction images based on instructions from the plurality of the instructors in a different color for each instructor who has given an instruction, and such that the identification signs and the instruction images corresponding to the same instructor are in the same color; When a predetermined operation is performed on the instruction image, a different character or symbol is added to the display of the instruction image for each person who has performed the instruction.

Images