Wearable terminal device, program and image processing method

The wearable terminal device enhances user interaction with virtual images by enabling capture and manipulation through gesture-based operations, addressing the limitations of conventional interfaces in VR and MR technologies.

JP2024164109AActive Publication Date: 2024-11-26KYOCERA CORP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024138504
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26
Estimated Expiration
2041-06-25

AI Technical Summary

Technical Problem

Conventional wearable terminal devices for VR and MR technologies lack user-friendly interfaces for capturing and manipulating virtual images within the user's field of view, limiting the interactivity and convenience of virtual image management.

Method used

A wearable terminal device equipped with a camera and processors that allow users to specify a capture area through gesture operations, store captured images, and perform various manipulations such as moving, duplicating, and extracting information from virtual images, enhancing user interaction and convenience.

Benefits of technology

Enables intuitive and efficient capture, manipulation, and display of virtual images, improving user experience by allowing users to interact naturally with virtual content and extract relevant information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164109000001_ABST
    Figure 2024164109000001_ABST
Patent Text Reader

Abstract

SOLUTION: A wearable terminal device worn and used by a user comprises a camera that captures a space as a user's visible area and at least one processor. The at least one processor identifies a part of the visible area in the space captured by the camera as a capture area based on a first gesture operation by the user, and stores the capture image corresponding to the capture area in a storage unit.EFFECT: A portion of a visible area desired by the user can be stored as a capture image in the storage unit, thus enhancing convenience for the user.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a wearable terminal device, a program, and an image processing method. [Background technology]

[0002] Conventionally, VR (Virtual Reality), MR (Mixed Reality), and AR (Augmented Reality) are known as technologies that allow a user to experience virtual images and / or virtual spaces using a wearable terminal device that the user wears on the head. A wearable terminal device has a display unit that covers the user's field of vision when worn by the user. By displaying virtual images and / or virtual spaces on this display unit according to the user's position and orientation, a visual effect is realized that makes it seem as if these images and / or virtual spaces are actually present (for example, Patent Document 1 and Patent Document 2).

[0003] MR is a technology that allows users to experience mixed reality, which is a fusion of real space and virtual images, by displaying a virtual image in a specific location in the real space while allowing the user to view the real space. VR is a technology that allows users to experience as if they are in a virtual space by allowing them to view a virtual space instead of the real space in MR.

[0004] The virtual image displayed in VR and MR has a fixed display position in the space in which the user is located, and is displayed on the display unit and viewed by the user when the display position is within the user's visual field. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 2019 / 0087021 [Patent Document 2] US Patent Application Publication No. 2019 / 0340822 Summary of the Invention

[0006] A wearable terminal device according to the present disclosure is a wearable terminal device worn by a user and includes a camera that captures an image of a space as a visual recognition area of ​​the user, and at least one processor. The at least one processor identifies a part of the visual recognition area in the space captured by the camera as a capture area based on a first gesture operation of the user, and stores a capture image corresponding to the capture area in a storage unit.

[0007] The program of the present disclosure is executed by a computer provided in a wearable terminal device that is worn by a user and has a camera that captures an image of a space as the user's visual area. The program causes the computer to execute a process of specifying a part of the visual area in the space captured by the camera as a capture area based on a first gesture operation of the user, and a process of storing a capture image corresponding to the capture area in a storage unit.

[0008] The image processing method disclosed herein is an image processing method executed by a computer provided in a wearable terminal device that is worn by a user and has a camera that captures an image of a space as the user's visual area. In the image processing method, a part of the visual area in the space captured by the camera is identified as a capture area based on a first gesture operation of the user, and a capture image corresponding to the capture area is stored in a storage unit. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a schematic perspective view showing a configuration of a wearable terminal device. [Diagram 2] 1A to 1C are diagrams illustrating an example of a visual recognition area and a virtual image visually recognized by a user wearing a wearable terminal device. [Diagram 3] FIG. 1 is a diagram illustrating a visual recognition area in space. [Figure 4] FIG. 2 is a block diagram showing a main functional configuration of the wearable terminal device. [Diagram 5] FIG. 13 is a diagram showing an example of a first gesture operation for specifying a capture region. [Figure 6] FIG. 13 is a diagram showing an example of a first gesture operation for specifying a capture region. [Figure 7] FIG. 2 is a diagram showing an image of the entire space captured by a camera. [Figure 8] 13A to 13C are diagrams illustrating a third gesture operation for displaying a menu virtual image. [Figure 9] FIG. 13 is a diagram showing an example of a visual recognition area in a state in which a captured virtual image is displayed. [Figure 10] FIG. 13 is a diagram illustrating an example of a method for displaying a captured virtual image. [Figure 11] FIG. 13 is a diagram illustrating an example of a method for displaying a captured virtual image. [Figure 12] 1A and 1B are diagrams illustrating a captured virtual image with a highlighted outer frame and a method of moving the captured virtual image. [Figure 13] FIG. 13 illustrates a method for extending a captured virtual image. [Figure 14] 11A-11C are diagrams illustrating the operation of removing a virtual image region from a captured virtual image. [Figure 15] 11A-11C are diagrams illustrating the operation of removing a spatial image region from a captured virtual image. [Figure 16] 11A and 11B are diagrams illustrating an operation of duplicating a virtual image included in a captured virtual image. [Figure 17] 11A and 11B are diagrams illustrating an operation of duplicating a virtual image included in a captured virtual image. [Figure 18] FIG. 13 is a diagram showing a movement method designation button for designating a movement method of a captured virtual image. [Figure 19] FIG. 13 is a diagram showing the operation of moving a virtual image within the frame of a captured virtual image. [Figure 20] 13A and 13B are diagrams illustrating an operation of moving a virtual image outside the frame of a captured virtual image. [Figure 21] 11A and 11B are diagrams illustrating an operation of extracting and displaying person information from a captured virtual image. [Figure 22] FIG. 13 is a diagram showing another example of the extraction information virtual image. [Figure 23] 13A and 13B are diagrams illustrating an operation of extracting and displaying information about an article from a captured virtual image. [Figure 24] 13A and 13B are diagrams illustrating an operation of extracting and displaying location information from a captured virtual image. [Diagram 25] 13A and 13B are diagrams illustrating another example of an operation for extracting and displaying information from a captured virtual image. [Figure 26] 13A to 13C are diagrams illustrating an operation of extracting and displaying character information from a captured virtual image. [Figure 27] 13A to 13C are diagrams illustrating operations when various processes are executed on extracted character information. [Figure 28] 13A to 13C are diagrams illustrating an operation of extracting and displaying two-dimensional code information from a captured virtual image. [Figure 29] 13A and 13B are diagrams illustrating an operation of displaying position information when a capture image is generated. [Diagram 30] 13A and 13B are diagrams illustrating an operation of displaying user information when a capture image is generated. [Diagram 31] FIG. 13 is a diagram showing a display operation when capture is prohibited. [Diagram 32] 13A to 13C are diagrams illustrating a display operation when a captured image is stored in association with audio. [Diagram 33] FIG. 13 is a diagram showing a captured virtual image in a case where audio is associated with the captured image. [Diagram 34] FIG. 13 illustrates a captured virtual image displayed on the surface of another object. [Diagram 35] FIG. 13 is a diagram showing a captured virtual image that moves together with a display object. [Diagram 36] FIG. 13 shows a captured virtual image rotating with a display object. [Figure 37] 13 is a flowchart showing a control procedure of a captured virtual image display process. [Figure 38] 13 is a flowchart showing a control procedure of a captured virtual image display process. [Figure 39] FIG. 11 is a schematic diagram showing the configuration of a display system according to a second embodiment. [Diagram 40] FIG. 2 is a block diagram showing a main functional configuration of an external device. [Diagram 41] FIG. 11 is a diagram showing the visual recognition area of ​​a wearable terminal device during screen sharing. [Diagram 42] 13 is a diagram showing a dialog image displayed on an instructor screen of an external device during screen sharing. FIG. [Diagram 43] 11A and 11B are diagrams showing a dialogue virtual image displayed in the viewing area of ​​a wearable terminal device during screen sharing. [Diagram 44] FIG. 13 is a schematic diagram showing the configuration of a display system according to a third embodiment. [Diagram 45] 1 is a block diagram showing a main functional configuration of an information processing device; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described with reference to the drawings. However, for the sake of convenience, each of the drawings referred to below shows only the main components necessary for explaining the embodiments in a simplified manner. Therefore, the wearable terminal device 10, the external device 20, and the information processing device 80 of the present disclosure may include any components not shown in each of the drawings referred to.

[0011] [First embodiment] As shown in FIG. 1, the wearable terminal device 10 includes a main body 10a, a visor 141 (display member) attached to the main body 10a, and the like.

[0012] The main body 10a is an annular member whose circumference is adjustable. Various devices such as a depth sensor 153 and a camera 154 are built in the main body 10a. When the main body 10a is worn on the head, the user's field of vision is covered by a visor 141.

[0013] The visor 141 is optically transparent. The user can view the real space through the visor 141. An image such as a virtual image is projected and displayed on a display surface of the visor 141 facing the user's eyes from a laser scanner 142 (see FIG. 4) built into the main body 10a. The user views the virtual image by light reflected from the display surface. At this time, the user also views the real space through the visor 141, so a visual effect is obtained as if the virtual image exists in the real space.

[0014] As shown in FIG. 2, in a state where the virtual image 30 (first virtual image) is displayed, the user visually recognizes the virtual image 30 facing a predetermined direction at a predetermined position in the space 40. In this embodiment, the space 40 is a real space visually recognized by the user through a visor 141. Since the virtual image 30 is projected onto the visor 141 having optical transparency, it is visually recognized as a semi-transparent image overlapping the real space. In FIG. 2, a planar window screen is illustrated as the virtual image 30, but the present invention is not limited thereto, and the virtual image 30 may be, for example, an object such as an arrow, or may be various three-dimensional images (three-dimensional virtual objects). When the virtual image 30 is a window screen, the virtual image 30 has a front surface (first surface) and a back surface (second surface), and necessary information is displayed on the front surface, and usually, no information is displayed on the back surface.

[0015] The wearable terminal device 10 detects the visual recognition area 41 of the user based on the position and orientation of the user in the space 40 (in other words, the position and orientation of the wearable terminal device 10). As shown in FIG. 3, the visual recognition area 41 is an area in the space 40 located in front of the user U wearing the wearable terminal device 10. For example, the visual recognition area 41 is an area within a predetermined angular range in the left-right direction and the up-down direction from the front of the user U. In this case, the shape of the cut edge when a solid corresponding to the shape of the visual recognition area 41 is cut by a plane perpendicular to the front direction of the user U is rectangular. Note that the shape of the visual recognition area 41 may be determined so that the shape of the cut edge is other than rectangular (for example, circular or elliptical). The shape of the visual recognition area 41 (for example, the angular range in the left-right direction and the up-down direction from the front) can be specified by, for example, the following method.

[0016] In the wearable terminal device 10, the field of view is adjusted (hereinafter, referred to as calibration) in a predetermined procedure at a predetermined timing such as the initial startup. This calibration specifies the range that the user can see, and thereafter, the virtual image 30 is displayed within this range. The shape of the visible range specified by this calibration can be the shape of the visible area 41.

[0017] Furthermore, the calibration is not limited to being performed according to the above-mentioned predetermined procedure, and may be performed automatically during normal operation of the wearable terminal device 10. For example, when the user does not react to a display that should be reacted to, the range in which the display is being performed may be considered to be outside the range of the user's field of view, and the field of view (and the shape of the visual recognition area 41) may be adjusted. Also, a trial display may be performed at a position that is determined to be outside the range of the field of view, and when the user reacts to the display, the range in which the display is being performed may be considered to be within the range of the user's field of view, and the field of view (and the shape of the visual recognition area 41) may be adjusted.

[0018] The shape of the visible region 41 may be determined and fixed in advance at the time of shipment, etc., without being based on the result of adjusting the field of view. For example, the shape of the visible region 41 may be determined to be the maximum displayable range in terms of the optical design of the display unit 14.

[0019] The virtual image 30 is generated in a state where the display position and orientation in the space 40 are determined in response to a predetermined operation by the user. The wearable terminal device 10 projects and displays, on the visor 141, the virtual image 30 whose display position is determined to be within the visible region 41, among the generated virtual images 30. In FIG. 2, the visible region 41 is indicated by a dashed line.

[0020] The display position and orientation of the virtual image 30 on the visor 141 are updated in real time according to the change in the user's viewing area 41. That is, the display position and orientation of the virtual image 30 change according to the change in the viewing area 41 so that the user recognizes that "the virtual image 30 is located in the space 40 at the set position and orientation." For example, when the user moves from the front side of the virtual image 30 to the back side, the shape (angle) of the virtual image 30 displayed according to this movement gradually changes. Also, when the user turns toward the direction of the virtual image 30 after going around to the back side of the virtual image 30, the back side of the virtual image 30 is displayed so that the back side of the virtual image 30 is visible. Also, according to the change in the viewing area 41, the virtual image 30 whose display position is out of the viewing area 41 is no longer displayed, and if there is a virtual image 30 whose display position is in the viewing area 41, the virtual image 30 is newly displayed.

[0021] 2, when a user holds out his / her hand (or finger) in front, the direction in which the hand is extended is detected by the wearable terminal device 10, and a virtual line 411 extending in that direction and a pointer 412 are displayed on the display surface of the visor 141 and visually recognized by the user. The pointer 412 is displayed at the intersection of the virtual line 411 and the virtual image 30. If the virtual line 411 does not intersect with the virtual image 30, the pointer 412 may be displayed at the intersection of the virtual line 411 and a wall surface or the like of the space 40. If the distance between the user's hand and the virtual image 30 is within a predetermined reference distance, the display of the virtual line 411 may be omitted, and the pointer 412 may be directly displayed at a position corresponding to the position of the user's fingertip.

[0022] By changing the direction in which the user extends his / her hand, the direction of the virtual line 411 and the position of the pointer 412 can be adjusted. By performing a predetermined gesture in a state in which the pointer 412 is adjusted so as to be positioned on a predetermined operation target (for example, the function bar 31, the window shape change button 32, the close button 33, etc.) included in the virtual image 30, the gesture is detected by the wearable terminal device 10, and a predetermined operation can be performed on the operation target. For example, by performing a gesture to select the operation target (for example, a gesture to pinch the fingertips) in a state in which the pointer 412 is aligned with the close button 33, the virtual image 30 can be closed (deleted). Also, by performing a gesture to select the operation target in a state in which the pointer 412 is aligned with the function bar 31, and performing a gesture to move the hand back and forth and left and right while in the selected state, the virtual image 30 can be moved in the depth direction and left and right directions. The operations on the virtual image 30 are not limited to these.

[0023] In this way, the wearable terminal device 10 of the present embodiment realizes a visual effect as if the virtual image 30 exists in real space, and can accept a user's operation on the virtual image 30 and reflect it in the display of the virtual image 30. That is, the wearable terminal device 10 of the present embodiment provides MR.

[0024] Next, the functional configuration of the wearable terminal device 10 will be described with reference to FIG. The wearable terminal device 10 includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, a display unit 14, a sensor unit 15, a communication unit 16, a microphone 17, a speaker 18, and the like, and these units are connected via a bus 19. All of the components shown in Fig. 4 except for the visor 141 of the display unit 14 are built into the main body unit 10a and operate using power supplied from a battery also built into the main body unit 10a.

[0025] The CPU 11 is a processor that performs various calculation processes and controls the overall operation of each part of the wearable terminal device 10. The CPU 11 performs various control operations by reading and executing a program 131 stored in the storage unit 13. The CPU 11 executes, for example, a visual recognition area detection process and a display control process by executing the program 131. Of these, the visual recognition area detection process is a process for detecting a visual recognition area 41 of a user in a space 40. Moreover, the display control process is a process for displaying, on the display unit 14, a virtual image 30 whose position in the space 40 is determined and whose position is determined within the visual recognition area 41.

[0026] 4 illustrates a single CPU 11, but is not limited to this. Two or more processors such as CPUs may be provided, and the processing executed by the CPU 11 of the present embodiment may be shared and executed by these two or more processors.

[0027] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0028] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer. The storage unit 13 stores a program 131 executed by the CPU 11, various setting data, and the like. The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. As the storage unit 13, for example, a non-volatile storage device such as an SSD (Solid State Drive) equipped with a flash memory is used.

[0029] The data stored in the storage unit 13 includes virtual image data 132 related to the virtual image 30. The virtual image data 132 includes data related to the display content of the virtual image 30 (e.g., image data), data on the display position, and data on the orientation.

[0030] The display unit 14 has a visor 141, a laser scanner 142, and an optical system that guides light output from the laser scanner 142 to the display surface of the visor 141. The laser scanner 142 irradiates the optical system with pulsed laser light, which is controlled to be turned on / off for each pixel, while scanning the optical system in a predetermined direction according to a control signal from the CPU 11. The laser light incident on the optical system forms a display screen consisting of a two-dimensional pixel matrix on the display surface of the visor 141. The method of the laser scanner 142 is not particularly limited, but for example, a method of scanning the laser light by operating a mirror using MEMS (Micro Electro Mechanical Systems) can be used. The laser scanner 142 has three light-emitting units that emit laser light of, for example, RGB colors. The display unit 14 can perform color display by projecting light from these light-emitting units onto the visor 141.

[0031] The sensor unit 15 includes an acceleration sensor 151, an angular velocity sensor 152, a depth sensor 153, a camera 154, and an eye tracker 155. The sensor unit 15 may further include a sensor not shown in FIG.

[0032] The acceleration sensor 151 detects acceleration and outputs the detection result to the CPU 11. From the detection result by the acceleration sensor 151, the translational motion of the wearable terminal device 10 in three orthogonal axial directions can be detected.

[0033] The angular velocity sensor 152 (gyro sensor) detects the angular velocity and outputs the detection result to the CPU 11. From the detection result by the angular velocity sensor 152, the rotational motion of the wearable terminal device 10 can be detected.

[0034] The depth sensor 153 is an infrared camera that detects the distance to a subject by a ToF (Time of Flight) method, and outputs the distance detection result to the CPU 11. The depth sensor 153 is provided on the front surface of the main body 10a so as to be able to capture an image of the visible area 41. By repeatedly performing measurements by the depth sensor 153 each time the position and orientation of the user change in the space 40 and synthesizing the results, it is possible to perform three-dimensional mapping of the entire space 40 (i.e., to obtain a three-dimensional structure).

[0035] The camera 154 captures an image of the space 40 using a group of RGB image pickup elements, acquires color image data as the captured image, and outputs the color image data to the CPU 11. The camera 154 is provided on the front side of the main body 10a so as to capture an image of the space 40 as the visual recognition area 41. The image of the space 40 captured by the camera 154 is used to detect the position and orientation of the wearable terminal device 10, and is also used to transmit the image of the space 40 captured by the communication unit 16 to an external device and display the visual recognition area 41 of the user of the wearable terminal device 10 on the external device. The image of the space 40 captured by the camera 154 is also used as an image of the visual recognition area 41 when the visual recognition area 41 is stored as a captured image as described later.

[0036] The eye tracker 155 detects the line of sight of the user and outputs the detection result to the CPU 11. The method of detecting the line of sight is not particularly limited, but for example, a method of photographing the reflection point of near-infrared light on the user's eye with an eye tracking camera and analyzing the photographed result and the image photographed by the camera 154 to identify the object that the user is viewing can be used. A part of the configuration of the eye tracker 155 may be provided on the periphery of the visor 141, etc.

[0037] The communication unit 16 is a communication module having an antenna, a modulation / demodulation circuit, a signal processing circuit, etc. The communication unit 16 transmits and receives data to and from an external device via wireless communication in accordance with a predetermined communication protocol. The communication unit 16 can also communicate audio data with the external device. That is, the communication unit 16 transmits audio data collected by the microphone 17 to the external device, and receives audio data transmitted from the external device to output audio from the speaker 18.

[0038] The microphone 17 converts sounds such as the user's voice into electrical signals and outputs them to the CPU 11 .

[0039] The speaker 18 converts the input audio data into mechanical vibrations and outputs the vibrations as sound.

[0040] In the wearable terminal device 10 configured as above, the CPU 11 performs the following control operations.

[0041] The CPU 11 performs three-dimensional mapping of the space 40 based on distance data to the subject input from the depth sensor 153. The CPU 11 repeats this three-dimensional mapping every time the position and orientation of the user changes, and updates the result each time. The CPU 11 also performs three-dimensional mapping for each continuous space 40. Therefore, when the user moves between multiple rooms separated by walls or the like, the CPU 11 recognizes each room as one space 40 and performs three-dimensional mapping separately for each room.

[0042] The CPU 11 detects the visual recognition area 41 of the user in the space 40. In detail, the CPU 11 specifies the position and orientation of the user (wearable terminal device 10) in the space 40 based on the detection results by the acceleration sensor 151, the angular velocity sensor 152, the depth sensor 153, the camera 154, and the eye tracker 155, and the accumulated results of the three-dimensional mapping. Then, the CPU 11 detects (specifies) the visual recognition area 41 based on the specified position and orientation and the predetermined shape of the visual recognition area 41. The CPU 11 also continuously detects the position and orientation of the user in real time, and updates the visual recognition area 41 in conjunction with changes in the position and orientation of the user. The detection of the visual recognition area 41 may be performed using some of the detection results by the acceleration sensor 151, the angular velocity sensor 152, the depth sensor 153, the camera 154, and the eye tracker 155.

[0043] CPU 11 generates virtual image data 132 related to virtual image 30 in response to a user operation. That is, when CPU 11 detects a predetermined operation (gesture) instructing generation of virtual image 30, CPU 11 specifies the display content (e.g., image data), display position, and orientation of the virtual image, and generates virtual image data 132 including data representing the specified results.

[0044] The CPU 11 displays the virtual image 30, the display position of which is determined within the visible area 41, on the display unit 14. The CPU 11 identifies the virtual image 30 based on the information on the display position included in the virtual image data 132, and generates image data of the display screen to be displayed on the display unit 14 based on the positional relationship between the visible area 41 and the display position of the virtual image 30 at that time. The CPU 11 causes the laser scanner 142 to perform a scanning operation based on this image data, and forms a display screen including the virtual image on the display surface of the visor 141. That is, the CPU 11 displays the virtual image 30 on the display surface of the visor 141 so that the virtual image 30 is visible in the space 40 visible through the visor 141. The CPU 11 continuously performs this display control process, thereby updating the display content by the display unit 14 in real time in accordance with the user's movement (changes in the visible area 41). If the wearable terminal device 10 is set to retain the virtual image data 132 even when the power is turned off, the next time the wearable terminal device 10 is started up, the existing virtual image data 132 is loaded, and if there is a virtual image 30 within the visible area 41, it is displayed on the display unit 14.

[0045] Note that the virtual image data 132 may be generated based on instruction data acquired from an external device via the communication unit 16, and the virtual image 30 may be displayed based on the virtual image data 132. Alternatively, the virtual image data 132 itself may be acquired from an external device via the communication unit 16, and the virtual image 30 may be displayed based on the virtual image data 132. For example, an image of the camera 154 of the wearable terminal device 10 may be displayed on an external device operated by a remote instructor, and an instruction to display the virtual image 30 may be received from the external device, and the instructed virtual image 30 may be displayed on the display unit 14 of the wearable terminal device 10. This allows, for example, an operation in which the virtual image 30 showing the work content is displayed near a work target object and the remote instructor instructs the user of the wearable terminal device 10 to perform the work.

[0046] CPU 11 detects the position and orientation of the user's hand (and / or finger) based on the images captured by depth sensor 153 and camera 154, and displays a virtual line 411 extending in the detected direction and a pointer 412 on display unit 14. CPU 11 also detects a gesture of the user's hand (and / or finger) based on the images captured by depth sensor 153 and camera 154, and executes processing according to the content of the detected gesture and the position of pointer 412 at that time.

[0047] Next, the operation of the wearable terminal device 10 will be described, focusing on the operation of capturing the visible area 41.

[0048] The wearable terminal device 10 includes the camera 154, and can capture the user's visual recognition area 41 at that time by having the camera 154 capture an image of the space 40 at a timing according to the user's operation and storing the captured image. However, a simple method of simply storing an image of the entire visual recognition area 41 captured by the camera 154 does not necessarily allow the user to utilize the captured image for a desired purpose, and is inconvenient. For this reason, there has been a demand for an improvement in the user interface that takes user convenience into consideration for the capture function of the visual recognition area 41 in the wearable terminal device 10.

[0049] In response to this, the wearable terminal device 10 of the present disclosure is equipped with various functions related to capturing the visual recognition area 41. Below, operations related to these functions and processes executed by the CPU 11 to realize these operations will be described.

[0050] As shown in FIG. 5 and FIG. 6, the CPU 11 of the wearable terminal device 10 of the present disclosure specifies a part of the visual recognition area 41 in the space 40 photographed by the camera 154 as a capture area R (see FIG. 6 and FIG. 7) based on the first gesture operation of the user, and stores a capture image C corresponding to the specified capture area R in the storage unit 13. This allows a part of the visual recognition area 41 desired by the user to be stored in the storage unit 13 as the capture image C, thereby improving user convenience. The capture image C includes an image of a part of the visual recognition area 41 that corresponds to the capture area R. In addition, when a virtual image 30 is displayed in the visual recognition area 41 and the virtual image 30 is included in the capture area R, the virtual image 30 is also reflected in the capture image C. Therefore, the capture image C is an image obtained by simply cutting out a part of the field of view that the user is viewing through the visor 141 when the capture area R is specified.

[0051] The first gesture operation for identifying the capture area R may be a gesture operation of moving the user's hand so that the trajectory of a predetermined part of the user's hand surrounds a part of the visual recognition area 41, as shown in FIG. 5. In this case, the area surrounded by the trajectory of the predetermined part of the user's hand can be identified as the capture area R. The first gesture operation may be, for example, a gesture operation of holding up at least one finger and surrounding the capture area R with the trajectory of the fingertip. By holding up the finger, it is possible to distinguish the gesture operation from other gesture operations and reduce false detection. The first gesture operation may also be an operation using both hands. Alternatively, instead of the user's hand or finger, the area surrounded by the trajectory of the pointer 412 may be identified as the capture area R. In FIG. 5, a range including the person 44 and a part of the virtual image 30 in the visual recognition area 41 is identified as the capture area R.

[0052] The first gesture operation for specifying the capture region R may be a gesture operation for moving a capture frame r of a preset size to a desired position in the visual recognition region 41 to confirm the position, as shown in FIG. 6. The operation for moving the capture frame r may be an operation for moving the hand (or pointer) with the position of the user's finger (or pointer 412) aligned with the capture frame r. The operation for confirming the position of the capture frame r may be an operation for tapping the air with the user's hand (finger). Here, the operation for tapping the air may be an operation of moving the hand (finger) away from the user and then moving it closer to the user, which is repeated twice. The size of the capture frame r may be changed by a gesture operation for moving the hand while pinching a part of the capture frame r with the fingers (or selecting the part with the pointer 412). Alternatively, two or more capture frames r of different sizes may be displayed, and the user may select one of the capture frames r. When the position of the capture frame r is confirmed, the area surrounded by the capture frame r is specified as the capture region R.

[0053] When the capture area R is specified, the CPU 11 stores in the storage unit 13 the captured image D (see FIG. 7) of the entire space 40 captured by the camera 154 at that time. The CPU 11 also stores in the storage unit 13 a composite image E (corresponding to the visual recognition area 41 shown in the upper diagram of FIG. 5 and the upper diagram of FIG. 6) obtained by combining the captured image D with the virtual image 30 displayed on the display unit 14. The CPU 11 also extracts a partial image corresponding to the capture area R from the composite image E and stores it in the storage unit 13 as a capture image C. In this way, when at least a part of the virtual image 30 as the first virtual image is included in the capture area, the CPU 11 stores in the storage unit 13 the capture image C including at least a part of the virtual image 30. This makes it possible to store the user's field of vision including the virtual image 30 as it is as the capture image C.

[0054] When performing the first gesture operation, the user first performs a predetermined operation to cause the wearable terminal device 10 to transition to a state in which the first gesture operation can be accepted. For example, as shown in FIG. 8, the user performs a predetermined third gesture operation to cause the display unit 14 to display a menu virtual image 61 (third virtual image). Then, in the menu virtual image 61, the user selects a capture operation start button 611 or 612 for starting the acceptance of the first gesture operation. The third gesture operation for displaying the menu virtual image 61 may be a gesture operation of moving both hands to have a predetermined positional relationship. For example, as shown in the upper left of FIG. 8, it may be a gesture operation of pointing the wrist of the left hand with the fingers of the right hand. When the CPU 11 detects this third gesture operation, it causes the display unit 14 to display the menu virtual image 61. Also, the third gesture operation may be a gesture operation of moving the hand (or the pointer 412) in a predetermined direction from outside the visible region 41 to inside the visible region 41, as shown in the upper right of FIG. 8. When the CPU 11 detects this third gesture operation, the menu virtual image 61 slides in so as to follow the hand movement (from the left side in the example of FIG. 8) and is displayed on the display unit 14. In this way, the CPU 11 displays the menu virtual image 61 as a third virtual image for starting to accept the first gesture operation in response to the user's third gesture operation on the display unit 14. This makes it possible to easily start the capture operation.

[0055] When a gesture operation to select the capture operation start button 611 of the menu virtual image 61 is performed, the CPU 11 accepts a first gesture operation to surround the capture area R with a fingertip or the like, as shown in Fig. 5. When a gesture operation to select the capture operation start button 612 is performed, the CPU 11 accepts a first gesture operation to specify the capture area R by a capture frame r, as shown in Fig. 6. In this way, the CPU 11 specifies the capture area R in different methods depending on the type of the first gesture operation, and specifies the type of the first gesture operation to be accepted depending on the operation on the menu virtual image 61 as the third virtual image. This allows the user to select a desired first gesture operation to specify the capture area R.

[0056] It should be noted that the state may be changed to one in which the first gesture operation is accepted without going through the menu virtual image 61. For example, the state may be changed to one in which the first gesture operation is accepted in response to the operation of tapping the air with the user's hand (finger) as described above. Also, an icon for starting capture may be provided on the function bar 31 of the virtual image 30, and the state may be changed to one in which the first gesture operation is accepted in response to the operation of selecting the icon.

[0057] As shown in FIG. 9, the capture image C stored in the storage unit 13 can be displayed on the display unit 14 as a capture virtual image 50 (second virtual image). When the CPU 11 stores the capture image C in the storage unit 13, the CPU 11 may display the capture image C on the display unit 14 as a capture virtual image 50 (second virtual image). This allows the capture virtual image 50 to be automatically displayed without the user performing a particular gesture operation. Furthermore, the CPU 11 may display the capture image C on the display unit 14 as the capture virtual image 50 (second virtual image) in response to the user's second gesture operation. This allows the user to display the capture virtual image 50 at a desired position at an intended timing.

[0058] 9, when the virtual image 30 is displayed as the first virtual image before displaying the captured virtual image 50 as the second virtual image, the CPU 11 may display the captured virtual image 50 at a position closer to the user than the virtual image 30 in the space 40. This allows the captured virtual image 50 to be displayed in a manner that allows the entire captured virtual image 50 to be viewed.

[0059] The second gesture operation in the case where the captured virtual image 50 is displayed in response to the second gesture operation is not particularly limited as long as it corresponds to the movement of the hand or fingers. The second gesture operation can be, for example, a specific movement of the hand (left hand and / or right hand), a specific movement of the fingers, opening and closing of the fingers, or a combination of these. The second gesture operation may be a movement of the pointer 412. The captured virtual image 50 may be displayed in response to a plurality of different types of second gesture operations. When the second gesture operation is performed, the CPU 11 displays the captured virtual image 50 at a relative position with respect to the user that is determined in advance in response to the type of the second gesture operation. This relative position can be, for example, near the palm or fingers, or on the user's line of sight. FIG. 10 shows an example in which the captured virtual image 50 is displayed near the fingertips of the left hand in response to the second gesture operation of pointing the wrist of the left hand with the fingers of the right hand. In this way, the CPU 11 displays the capture virtual image 50 as the second virtual image at a predetermined relative position to the user in response to the second gesture operation, and the relative position is set in advance for each type of the second gesture operation. This allows the capture virtual image 50 to be displayed at a desired position with a simple operation.

[0060] The method of displaying the capture virtual image 50 is not limited to the above, and may be a method of selecting one of one or more capture images C stored in the storage unit 13 to be displayed as the capture virtual image 50, as shown in FIG. 11. In the example of FIG. 11, as shown in the upper diagram, a gesture operation of moving a hand (or a pointer 412) from the outside (right side) of the visual recognition area 41 into the visual recognition area 41 causes a list area L of the capture images C to slide in and be displayed from the right end of the visual recognition area 41 so as to follow the movement of the hand. Next, as shown in the lower diagram of FIG. 11, a gesture operation of placing a finger on one capture image C in the list area L (or selecting it with the pointer 412) and dragging it into the visual recognition area 41 causes the capture image C to be displayed on the display unit 14 as the capture virtual image 50. When the list area L contains one capture image C, the list area L may be erased in response to the drag operation. When the list area L includes two or more capture images C, the list area L may be displayed even after the drag operation, so that the user can continue to perform drag operations on other capture images C. The image showing the list area L may be determined based on a predetermined condition. For example, the CPU 11 may display a specific image associated with a user ID used when logging in to the wearable terminal device 10 as the image showing the list area L.

[0061] 12, the CPU 11 may display the outer frame of the captured virtual image 50 as the second virtual image in a predetermined highlighted manner. This makes it possible to easily view the captured virtual image 50. One example of the highlighted manner is to make the outer frame of the captured virtual image 50 thicker than the outer frame of the virtual image 30 as shown in FIG. 12, but is not limited to this.

[0062] 12, the CPU 11 moves the captured virtual image 50 in response to a fourth gesture operation by the user with respect to the captured virtual image 50 as the displayed second virtual image. This allows the user to arbitrarily change the display position of the captured virtual image 50. The fourth gesture operation is not particularly limited, and may be an operation of moving a hand while selecting a predetermined portion (e.g., the upper function bar) of the captured virtual image 50, or an arbitrary portion of the captured virtual image 50.

[0063] As shown in FIG. 13, the user can expand the display area of ​​the capture virtual image 50 by performing a predetermined fifth gesture operation. The fifth gesture operation may be, for example, an operation of moving a hand while pinching a part of the outer frame of the capture virtual image 50 with fingers (or selecting it with the pointer 412). In response to the expansion of the capture virtual image 50, the range of the capture area R reflected in the capture virtual image 50 is expanded. The expanded capture virtual image 50 includes the initial display area 50a before the expansion and the expanded expanded area 50b. The image of the expanded area 50b may be extracted from the composite image E of the entire visual recognition area 41 stored when the capture image C was generated. In other words, a part of the composite image E corresponding to the expanded capture area R is extracted to generate a new capture image C, and the expanded capture virtual image 50 is displayed based on the new capture image C. Note that the capture virtual image 50 can also be reduced in size in response to the fifth gesture operation. In this way, the CPU 11 expands or reduces the capture virtual image 50 in response to the user's fifth gesture operation on the captured virtual image 50 as the displayed second virtual image, and expands or reduces the range of the capture region R reflected in the captured virtual image 50 in response to the expansion or contraction. This makes it possible to change the capture range even after the captured virtual image 50 is displayed.

[0064] As shown in the upper diagram of FIG. 14, when the captured virtual image 50 includes a virtual image area 51 corresponding to the virtual image 30 and a spatial image area 52 corresponding to the background space 40, the virtual image area 51 can be selectively deleted as shown in the lower diagram. For convenience, the virtual image area 51 in the captured virtual image 50 may be referred to as the "virtual image 30 included in the captured virtual image 50" below. The virtual image area 51 is deleted in response to a sixth gesture operation by the user. The sixth gesture operation is not particularly limited, and may be, for example, an operation of double tapping or long pressing the virtual image area 51. In the captured virtual image 50 after the virtual image area 51 is deleted, the background space 40 is displayed in the range where the virtual image area 51 was displayed before the deletion. The image of the space 40 is extracted from the captured image D (see FIG. 7) of the space 40 stored when the captured image C was generated. In this way, the CPU 11 deletes the virtual image 30 as the first virtual image included in the capture virtual image 50 in response to the sixth gesture operation of the user on the captured virtual image 50 as the displayed second virtual image, and displays an image of the space 40 corresponding to the deleted area in the deleted area. This allows the virtual image 30 to be deleted from the capture virtual image 50 even after the capture virtual image 50 is displayed. In this case, the CPU 11 may display the virtual image 30 included in the capture virtual image 50 in a predetermined highlighted manner. This makes it easier to visually recognize the virtual image area 51 to be deleted. An example of the highlighted manner is to thicken the outer frame of the virtual image area 51 as shown in the upper diagram of FIG. 14, but is not limited thereto.

[0065] As shown in the upper diagram of FIG. 15, in the case where the captured virtual image 50 includes a virtual image area 51 and a spatial image area 52, as shown in the lower diagram, the spatial image area 52 can be selectively deleted from the captured virtual image 50. The deletion of the virtual image area 51 is performed in response to a seventh gesture operation by the user. The seventh gesture operation is not particularly limited, and may be, for example, an operation of double tapping or long pressing the spatial image area 52. The captured virtual image 50 after the spatial image area 52 is deleted may include only the virtual image area 51. The image of the virtual image area 51 may be extracted from the captured image C before the deletion. Also, the image of the virtual image area 51 may be extracted from the virtual image data 132 stored in the storage unit 13. That is, an ID for identifying the virtual image 30 may be stored when the captured image C is stored, and the image data of the virtual image area 51 may be identified and acquired from the virtual image data 132 by referring to the ID. In this manner, the CPU 11 deletes the portion of the capture virtual image 50 other than the virtual image 30 in response to the user's seventh gesture operation on the capture virtual image 50 as the displayed second virtual image. This allows the background space 40 to be deleted from the capture virtual image 50 even after the capture virtual image 50 is displayed. In this case, the CPU 11 may display the portion (spatial image area 52) other than the virtual image 30 included in the capture virtual image 50 in a predetermined emphasized manner. This makes it possible to easily visually recognize the spatial image area 52 to be deleted. An example of the emphasized manner is to thicken the outer frame of the spatial image area 52 as shown in the upper diagram of FIG. 15, but is not limited thereto.

[0066] 16, in response to a tenth gesture operation by the user on the captured virtual image 50 as the displayed second virtual image, the CPU 11 may copy at least a part of the captured virtual image 50 to display a virtual image 34 as a fifth virtual image. This allows a part of the captured virtual image 50 to be handled as a separate virtual image 34, thereby improving user convenience.

[0067] From another perspective, as shown in the upper diagram of FIG. 16, when a part of the virtual image 30 as the first virtual image is included in the captured virtual image 50 as the second virtual image, the CPU 11 may copy the part of the virtual image 30 and display the virtual image 34 as the fourth virtual image in response to the eighth gesture operation of the user on the displayed virtual image 30. This makes it possible to extract a part of the virtual image 30 included in the captured virtual image 50 and handle it as a separate virtual image 34, thereby improving user convenience. Here, the virtual image area 51 in the original captured virtual image 50 may be displayed in a predetermined suppression mode after copying. This makes it possible to clearly show the copied object. The suppression mode is not particularly limited, and may be, for example, a mode in which the image is made semi-transparent. The image of the virtual image 34 (part of the virtual image 30) to be copied may be extracted from the virtual image data 132 stored in the storage unit 13. That is, the ID of the virtual image 30 may be stored when the captured image C is stored, and part of the image data of the virtual image 30 may be acquired from the virtual image data 132 by referring to the ID. The tenth and eighth gesture operations are not particularly limited, but may be, for example, an operation of double-tapping the spatial image area 52 or an operation of pressing and holding the spatial image area 52.

[0068] As shown in the upper diagram of FIG. 17, when the eighth gesture operation is performed on a virtual image area 51 of the captured virtual image 50 that corresponds to a part of the virtual image 30, the entire virtual image 30 may be restored and displayed as a virtual image 35 as shown in the lower diagram of FIG. 17. Therefore, the contents of the virtual image 35 are the same as the contents of the virtual image 30. In this case, the image of the virtual image 35 to be copied may be extracted from the virtual image data 132 stored in the storage unit 13. That is, the ID of the virtual image 30 may be stored when the captured image C is stored, and the image data of the entire virtual image 30 may be obtained from the virtual image data 132 by referring to the ID. Note that, while displaying the copied virtual image 35, a part of the virtual image 30 (the virtual image area 51) that was inside the captured virtual image 50 may be deleted. This provides a visual effect of moving the virtual image 30 from inside the frame of the captured virtual image 50 to outside the frame.

[0069] In addition, when copying part or all of the virtual images 30 contained in the captured virtual image 50 as shown in Figures 16 and 17, only the virtual images 30 for which copying is permitted may be copied, and copying may not be performed on the virtual images 30 for which copying is not permitted.

[0070] As shown in FIGS. 18 to 20, when a virtual image area 51 (at least a part of the virtual image 30) is included in a captured virtual image 50, the virtual image area 51 may be able to be moved within and / or outside the frame of the captured virtual image 50. When performing such a movement, the user performs a predetermined gesture operation (for example, an operation of pressing and holding inside the captured virtual image 50) to display movement method designation buttons 91 to 93 for designating a movement method of the virtual image area 51, as shown in FIG. 18. The movement method designation button 91 is a button for moving the virtual image area 51 within the frame of the captured virtual image 50, the movement method designation button 92 is a button for moving the virtual image area 51 within and / or outside the frame of the captured virtual image 50, and the movement method designation button 93 is a button for moving the virtual image area 51 outside the frame of the captured virtual image 50.

[0071] When the user performs a ninth gesture operation of selecting the movement method designation button 91, the CPU 11 accepts the start of an operation of moving the virtual image area 51 (at least a part of the virtual image 30) within the frame of the captured virtual image 50. In this state, as shown in FIG. 19, when the user performs a gesture operation of dragging the virtual image area 51 within the frame of the captured virtual image 50, the virtual image area 51 moves within the frame of the captured virtual image 50 so as to follow the drag. Here, since the outer shape of the captured virtual image 50 does not change, the size of the virtual image area 51 displayed within the captured virtual image 50 (the display range of the virtual image 30) is expanded or reduced with the movement of the virtual image area 51. In response to the movement of the virtual image area 51, a shadow 51a may be displayed in the area where the virtual image area 51 was located before the movement.

[0072] When the user performs a ninth gesture operation of selecting the movement method designation button 93, the CPU 11 accepts the start of an operation of moving the virtual image area 51 (a part of the virtual image 30) outside the frame of the captured virtual image 50. In this state, as shown in FIG. 20, when the user performs a gesture operation of dragging the virtual image area 51 outside the frame of the captured virtual image 50, a virtual image 35 obtained by duplicating the entire virtual image 30 is displayed outside the frame of the captured virtual image 50. Alternatively, as in the virtual image 34 in FIG. 16, a part of the captured virtual image 50 corresponding to the virtual image area 51 may be moved directly outside the frame. In addition, the virtual image area 51 that was within the frame of the captured virtual image 50 may be deleted in accordance with the display of the virtual image 35 (or the virtual image 34).

[0073] When the user performs a ninth gesture operation of selecting the movement method designation button 92, the CPU 11 accepts the start of both the movement of the virtual image area 51 within the frame of the captured virtual image 50 and the movement of the virtual image area 51 outside the frame of the captured virtual image 50. That is, when the user performs a gesture operation of dragging the virtual image area 51 within the frame of the captured virtual image 50, the CPU 11 moves the virtual image area 51 within the frame of the captured virtual image 50 as shown in FIG. 19, and when the user performs a gesture operation of dragging the virtual image area 51 outside the frame of the captured virtual image 50, the CPU 11 displays the virtual image 35 (or virtual image 34) outside the frame of the captured virtual image 50 as shown in FIG. 20.

[0074] In this way, in response to the user's ninth gesture operation on captured virtual image 50 as the displayed second virtual image, CPU 11 accepts the start of an operation to move virtual image 30 as the first virtual image included in captured virtual image 50 by any of a plurality of moving methods, the plurality of moving methods including a method of expanding or reducing the display range of virtual image 30 within captured virtual image 50, and a method of moving virtual image 30 outside captured virtual image 50 and displaying it as virtual image 34. This allows virtual image 30 within captured virtual image 50 to be moved in a desired manner.

[0075] In the wearable terminal device 10 of the present disclosure, information can be extracted from the captured virtual image 50 and displayed. That is, when an extraction target from which information can be extracted is included in the captured virtual image 50 as the second virtual image, the CPU 11 extracts information from the extraction target and displays it on the display unit 14. The information extraction target may be at least one of a person, an object, a place, text, and code information. This allows easy access to information that can be extracted from the person, object, place, text, code information, and the like included in the captured virtual image 50. Hereinafter, various aspects related to the extraction of information will be described with reference to FIGS. 21 to 30.

[0076] As shown in FIG. 21, when the captured virtual image 50 includes an image of a person 44 and information can be extracted from the image of the person 44, an extracted information virtual image 62 (sixth virtual image) including the extracted information is displayed. The extracted information virtual image 62 may be displayed when a predetermined operation is performed by the user while the captured virtual image 50 is displayed, or may be displayed automatically together with the display of the captured virtual image 50. The extracted information virtual image 62 displays information such as a facial photograph 621, name, affiliation, and contact ID of the person 44. When the facial photograph 621 cannot be acquired, a message indicating that it cannot be acquired may be displayed. In response to a gesture operation to select the facial photograph 621, an operation (e.g., a call operation) for accessing the person 44 may be started. The contact ID is a code or number used to contact the person 44, and may be, for example, a telephone number. The method for extracting information about person 44 from captured virtual image 50 is not particularly limited, but for example, a method may be used in which characteristics of the image of person 44 are analyzed, a person matching the results of the characteristic analysis is identified by referring to a database in which information about multiple people is registered in advance, and information about that person is obtained from the database.

[0077] In addition, the extracted information virtual image 62 may display an icon 622 (sign) of an application program (hereinafter, referred to as an app) to be executed to contact the person 44. The app is predefined according to the type of information extraction target (here, a person). When a gesture operation to select the icon 622 is performed, an app corresponding to the icon 622 (here, a phone book app) is executed. In this manner, when an extraction target from which information can be extracted is included in the captured virtual image 50 as the second virtual image, the CPU 11 may display the icon 622 as a sign for starting an application predefined according to the type of the extraction target on the display unit 14. This allows an appropriate app according to the type of the extraction target to be easily started. Note that the display of the icon 622 may be omitted, and an app for contacting the person 44 may be started in response to a gesture operation to select the face photo 621 or a contact ID (such as a phone number) to start a call operation or the like.

[0078] In FIG. 21, various information and an icon 622 are displayed on the front (first surface) of the extracted information virtual image 62, and no information is displayed on the back (second surface). However, this is not limited to this embodiment, and for example, as shown in FIG. 22, an icon 622 may be displayed on the back of the extracted information virtual image 62. That is, the CPU 11 displays the extracted information virtual image 62 as a sixth virtual image including information extracted from the extraction target on the display unit 14, and displays the icon 622 as a sign on at least one of the first surface of the extracted information virtual image 62 and the second surface opposite to the first surface. This allows the icon 622 to be displayed in an easily accessible position according to the display position and orientation of the extracted information virtual image 62.

[0079] As shown in FIG. 23, when the captured virtual image 50 includes an image of the item 45 (here, a mask) and information can be extracted from the image of the item 45, an extracted information virtual image 63 (sixth virtual image) including the extracted information is displayed. The extracted information virtual image 63 displays an image 631 of the item 45 acquired from the Web (e.g., an EC site), information such as the product name, manufacturer, contact ID, and price of the item 45. When the image 631 cannot be acquired, a message indicating that the image cannot be acquired may be displayed. In response to a gesture operation to select the image 631, a website (e.g., an EC site) where information on the item 45 can be accessed may be displayed. The contact ID is a code or number used to acquire information on or purchase the item 45, and may be, for example, a telephone number or a URL. The method of extracting information on the item 45 from the captured virtual image 50 is not particularly limited, but for example, a method of identifying the item 45 in a similar manner to the above-mentioned method of identifying the person 44 and acquiring information on the item 45 from a database may be used.

[0080] Furthermore, the extracted information virtual image 63 may display an icon 632 (sign) of an application to be executed for accessing information on the item 45. The application is determined in advance according to the type of information to be extracted (an item in this case). When a gesture operation to select the icon 632 is performed, an application corresponding to the icon 632 (a browser application in this case) is executed. Note that the display of the icon 632 may be omitted, and an application (such as a browser application) for accessing information on the item 45 may be started in response to a gesture operation to select the image 631 of the item 45 or a contact ID (such as a URL), and an EC site where the item 45 can be purchased may be displayed, for example.

[0081] As shown in FIG. 24, when it is possible to extract information on a place from an image of the space 40 in the background of the captured virtual image 50, an extracted information virtual image 64 (sixth virtual image) including the extracted information is displayed. The extracted information virtual image 64 displays information such as an image 641 representing the place, the name of the place, information on the building if the place is a building, and a contact ID. For example, when the place is a building, the image 641 may be a logo of a company that owns the building. When the image 641 cannot be acquired, a message indicating that the image 641 cannot be acquired may be displayed. In response to a gesture operation to select the image 641, a map of the place may be displayed. The displayed map may be a standard map if the place is outside the building, and may be an indoor map if the place is inside the building. The contact ID is a code or number used to access information on the place, and may be, for example, a telephone number or a URL. The method of extracting information on the place from the captured virtual image 50 is not particularly limited, but for example, a method of identifying the place in a similar manner to the above-mentioned method of identifying the person 44 and acquiring information on the place from a database may be used.

[0082] Furthermore, the extracted information virtual image 64 may display an icon 642 (sign) of an application to be executed to access location information. The application is determined in advance according to the type of information to be extracted (location in this case). When a gesture operation to select the icon 642 is performed, an application corresponding to the icon 642 (map application in this case) is executed. Note that the display of the icon 642 may be omitted, and an application (map application, etc.) to access location information may be started in response to a gesture operation to select the location image 641 or a contact ID (URL, etc.), and a map showing the location of the location, for example, may be displayed.

[0083] The operation when information can be extracted from the captured virtual image 50 is not limited to the above. For example, as shown on the left side of Fig. 25, when person information can be extracted from the captured virtual image 50, an icon 622 of an application (e.g., a calling application) that is pre-associated with the "person" from which information is to be extracted may be automatically displayed. In addition, the application may be executed in response to a gesture operation to select the icon 622, and a virtual image 623 of the application may be displayed in a state where, for example, a call with the extracted person has started.

[0084] 25, when information about an item can be extracted from the captured virtual image 50, an icon 632 of an app (e.g., a browser app) that is pre-associated with the "item" from which information is to be extracted may be automatically displayed. In addition, the app may be executed in response to a gesture operation to select the icon 632, and a virtual image 633 of the app including, for example, a website from which the extracted information about the item can be accessed may be displayed.

[0085] 25, when location information can be extracted from the captured virtual image 50, an icon 642 of an app (e.g., a map app) that is pre-associated with the "location" from which information is to be extracted may be automatically displayed. In addition, the app may be executed in response to a gesture operation to select the icon 642, and a virtual image 643 of the app including a map showing the position of the extracted location may be displayed.

[0086] In FIG. 25, the display of the icons 622, 632, and 642 may be skipped, and the virtual images 623, 633, and 643 of the applications may be displayed directly.

[0087] As shown in FIG. 26, when the captured virtual image 50 includes an image of the character 46 and information can be extracted from the image of the character 46, an extracted information virtual image 65 (sixth virtual image) including the extracted information is displayed. The extracted information virtual image 65 displays the content of the character 46 extracted by OCR (Optical Character Recognition) or the like, and operation buttons 651 to 653 for executing various processes on the extracted character. The size of the character 46 displayed in the extracted information virtual image 65 may be larger than the size of the character 46 in the captured virtual image 50. Also, the size of the character 46 displayed in the extracted information virtual image 65 may be set in advance.

[0088] As shown in Fig. 27, when a gesture operation is performed to select one of the operation buttons 651 to 653, a process corresponding to the operation button 651 to 653 is executed for the extracted character 46. When a gesture operation is performed to select the operation button 651 displaying "Copy", an icon 661 of an application (for example, an editor application) that is preset as an application for editing text data is automatically displayed, as shown on the left side of Fig. 27. Furthermore, the application is executed in response to the gesture operation to select the icon 661, and a virtual image 67 of the application is displayed in a state in which the character 46 can be edited.

[0089] Also, when a gesture operation is performed to select the operation button 652 displaying "Translation: Automatic → EG", an icon 662 of an application (for example, an editor application) that is preset as an application for editing text data is automatically displayed as shown in the center of FIG. 27. The icon 662 may be the same as the icon 661. Also, when a gesture operation is performed to select the icon 662, the extracted characters 46 are translated according to a predetermined translation setting. Here, the setting is that the language of the extracted characters 46 is automatically determined and translated into English. Alternatively, a translation setting related to the translation destination language or the like may be made according to an eleventh gesture operation by the user on a setting button not shown. In this case, the characters 46 are translated according to the translation setting by the user. Then, the application corresponding to the icon 662 is executed, and a virtual image 68 of the application is displayed in a state in which the translated characters can be edited. In this way, the CPU 11 translates the information extracted from the characters 46 according to a predetermined translation setting or according to a translation setting according to the eleventh gesture operation by the user, and displays it on the display unit 14. This makes it possible to easily translate and display the extracted characters 46.

[0090] Furthermore, when a gesture operation is performed to select the operation button 653 labeled "Specify Application," an icon 663 of the application (here, a browser application) designated by the user is automatically displayed, as shown on the right side of FIG. 27. The application may be designated and set in advance by the user, or the user may select the application each time from among application options displayed in response to the selection of the operation button 653. The application is executed in response to the gesture operation to select the icon 663, and a virtual image 69 of the application that performs processing related to the information of the extracted characters 46 is displayed. In the example of FIG. 27, a virtual image 69 of the browser application in a state in which the search results for the extracted characters 46 are displayed is displayed.

[0091] In FIG. 27, the display of the icons 661 to 663 may be skipped and the virtual images 67 to 69 of the applications may be displayed directly.

[0092] As shown in FIG. 28, when the captured virtual image 50 includes an image of a two-dimensional code 47 (code information) and information can be extracted from the image of the two-dimensional code 47, an extracted information virtual image 71 (sixth virtual image) including the extracted information is displayed. The extracted information virtual image 71 displays information obtained by decoding the two-dimensional code 47 and information related to the information. In the example of FIG. 71, information on "K Corporation" is extracted from the two-dimensional code 47, and the company's address, telephone number, and URL information are displayed. In response to a gesture operation to select the telephone number or URL, an application (such as a calling application or a browser application) for accessing "K Corporation" may be launched. Note that the code information from which information can be extracted is not limited to the two-dimensional code 47, and may be a barcode or a code consisting of characters and symbols.

[0093] The captured virtual image 50 may include position information of the wearable terminal device 10 when the captured image C was generated. In this case, the CPU 11 acquires position information of the wearable terminal device 10 when the first gesture operation described above is performed, and stores the acquired position information in the storage unit 13 in association with the captured image C. This allows the position at which the captured image C was generated to be easily referenced at any timing after capture. The position information may be acquired from a positioning satellite of a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System), or may be various local position information. The local position information may be acquired from a signal transmitted from, for example, a wireless LAN access point, a beacon station, and a local 5G base station. In addition, when the captured virtual image 50 is displayed based on the captured image C, a virtual image 72 including information related to the position at which the captured image C was generated may be displayed as shown in FIG. 29. In addition, instead of (or in addition to) the character information as shown in FIG. 29, a map indicating the capture position may be displayed. In addition to the position information, information on the date and time when the captured image C was generated may be acquired and displayed on the virtual image 72.

[0094] The captured virtual image 50 may include information for identifying a user who was using the wearable terminal device 10 when the captured image C was generated. In this case, the CPU 11 identifies the user when the first gesture operation was performed, and stores user information related to the identified user in the storage unit 13 in association with the captured image C. This makes it possible to easily refer to the operator when the captured image C was generated at any timing after capture. The method for identifying the user is not particularly limited, and may be obtained from login information of the user when the captured image C was generated, for example. In addition, when the captured virtual image 50 is displayed based on the captured image C, a virtual image 73 including information related to the user and the date and time when the captured image C was generated may be displayed as shown in FIG. 30. Note that, as information related to the user, an icon capable of displaying a face image of the user may be superimposed on the captured virtual image 50 or displayed near the captured virtual image 50.

[0095] Because the wearable terminal device 10 can be used for various purposes in various places, it may not be appropriate to generate and store the captured image C depending on the location when the capture operation is performed and the capture target included in the visible area 41. The wearable terminal device 10 may have a function to prevent the captured image C from being stored in the storage unit 13 in such a case. Hereinafter, prohibiting the generation of the captured image C and storing it in the storage unit 13 is also referred to as "prohibiting capture," and allowing the generation of the captured image C and storing it in the storage unit 13 is also referred to as "allowing capture."

[0096] For example, as shown in FIG. 31, the CPU 11 may determine whether or not a predetermined capture prohibited object is included in the capture area R, and when it is determined that the capture prohibited object is included in the capture area R, the CPU 11 may not store the capture image C in the storage unit 13. This makes it possible to restrict the generation of the capture image C including the capture prohibited object and the storage of the capture image C in the storage unit 13. In the example of FIG. 31, the capture area R specified by the first gesture operation of the user includes a person 44 that is set in advance as a capture prohibited object. In this case, even if the capture area R is specified, the capture image C is not stored in the storage unit 13. Note that the capture prohibited object is not limited to a person, and may be an object such as a work of art such as a painting, or real estate such as a building. The capture prohibited object may be an object protected by copyright. The method of determining whether or not the object is a capture prohibited object is not particularly limited, and may be, for example, a method of storing image features of the capture prohibited object in the storage unit 13 or an external server and comparing the image features of the capture prohibited object with the image included in the capture area R.

[0097] When capture is prohibited, for example, the display mode of the capture operation start buttons 611 and 612 may be changed in the menu virtual image 61 shown in the lower diagram of Fig. 8 to disable the operation. Alternatively, the capture operation start buttons 611 and 612 may be hidden.

[0098] When the capture area R includes a capture-prohibited object, if an authority who releases the capture prohibition permits capture, the prohibition may be lifted and the capture image C may be stored in the storage unit 13. For example, as shown in the lower diagram of FIG. 31, in response to a first gesture operation that specifies the capture area R, a dialogue virtual image 74 that asks the user whether or not to request permission for capture is displayed. When a gesture operation is performed on a decision button 741 of the dialogue virtual image 74, a signal requesting lifting of the capture prohibition is transmitted to an external device used by the authority. When a permission signal permitting lifting of the capture prohibition is received from the external device, the capture image C is stored in the storage unit 13. The subsequent display operation of the capture virtual image 50 is the same as that described above.

[0099] Furthermore, the authority level of the user operating the wearable terminal device 10 may be referenced based on the user ID used when logging in to the wearable terminal device 10, and whether or not to prohibit capture may be determined according to the authority level. The authority level may be determined based on, for example, a position, department, or qualification.

[0100] Also, whether or not capture is prohibited may be determined based on the current position of the wearable terminal device 10. In this case, the CPU 11 acquires position information of the own device when the first gesture operation is performed, and when the position indicated by the position information satisfies a predetermined prohibited position condition, the CPU 11 does not store the capture image C in the storage unit 13. This makes it possible to prohibit capture when the wearable terminal device 10 is located in a specific location. The position information may be acquired from a GNSS positioning satellite such as GPS, or may be various local position information. The local position information may be acquired from a signal transmitted from, for example, a wireless LAN access point, a beacon station, and a local 5G base station. When the acquired position information is within a predetermined prohibited area, it is determined that the prohibited position condition is satisfied.

[0101] Furthermore, the CPU 11 may not store the capture image C in the storage unit 13 when the communication unit 16 receives a specific signal. In this way, by transmitting a specific signal to the wearable terminal device 10, capture can be prohibited at any timing. The specific signal may be any signal that is set in advance as a signal for prohibiting capture. When a predetermined time has elapsed after the reception of the specific signal, the prohibition of capture may be lifted. Even when a specific signal is received, capture may be permitted when the visible region 41 does not include a virtual object such as the virtual image 30, but includes only the background space 40.

[0102] Furthermore, the CPU 11 may not store the capture image C in the storage unit 13 when a predetermined connection condition related to the connection state of the communication unit 16 to the communication network is satisfied. This enables capture prohibition control according to the connection state to the communication network. The connection condition related to the capture prohibition control may be arbitrarily determined. As one example, when connected to a public communication network, it may be determined that the above connection condition is satisfied and capture may be prohibited, and when connected to a private communication network (such as local 5G or wireless LAN), it may be determined that the above connection condition is not satisfied and capture may be permitted. As another example, when the wearable terminal device 10 is online, it may be determined that the above connection condition is satisfied and capture may be prohibited, and when offline, it may be determined that the above connection condition is not satisfied and capture may be permitted.

[0103] Furthermore, when at least a part of a virtual image (first virtual image) is included in the capture area R and the virtual image satisfies a predetermined prohibition condition, the CPU 11 may not store the capture image C in the storage unit 13. This makes it possible to prevent a virtual image that is not appropriate as a capture target from being stored as the capture image C. For example, when the virtual image is a screen of a specific application and a specific capture prohibition flag is set in the application, it is possible to prevent the virtual image of the application from being stored as the capture image C.

[0104] As described above, in each of the cases where capture is prohibited based on the current location, where capture is prohibited in response to receiving a specific signal, where capture is prohibited in response to satisfying a connection condition related to the connection state to the communication network, and where capture is prohibited in response to a virtual image included in the capture area R satisfying a prohibition condition, the prohibition of capture may be lifted if an authority permits it. In this case, as shown in the lower diagram of FIG. 31, a dialog virtual image 74 inquiring of the user as to whether or not to request permission for capture may be displayed. Also, the authority level of the user operating the wearable terminal device 10 may be referred to, and whether or not to prohibit capture may be determined in accordance with the authority level.

[0105] The capture image C may be stored in association with audio data. That is, when the CPU 11 stores the capture image C in the storage unit 13, the CPU 11 may acquire audio data and store the audio data in association with the capture image C in the storage unit 13. This allows audio information to be added to the capture image C. For example, when the capture area R is specified in response to the first gesture operation as shown in the upper diagram of FIG. 32, a dialogue virtual image 75 inquiring of the user as to whether or not to record audio may be displayed as shown in the lower diagram of FIG. 32. When a gesture operation is performed to operate the decision button 751 of the dialogue virtual image 75, audio data is acquired (recorded) by the microphone 17, and is stored in association with the capture image C of the capture area R in the storage unit 13. Note that the timing of recording is not limited to this, and for example, recording may be performed when the capture virtual image 50 is displayed. When the capture virtual image 50 including the capture image C associated with audio data is displayed, a play button 53 for playing the audio data may also be displayed as shown in FIG. 33. In response to a gesture operation of selecting the play button 53, the sound of the audio data associated with the capture image C is output from the speaker 18. The play button 53 may be displayed behind the capture virtual image 50.

[0106] In the above, the captured virtual image 50 is displayed alone in the space 40, but the present invention is not limited to this, and the captured virtual image 50 may be displayed so as to satisfy a predetermined positional relationship with another object (display target). That is, the CPU 11 may display the captured virtual image 50 as a second virtual image on the surface of a predetermined display target located in the space 40, or at a position that satisfies a predetermined positional relationship with the display target. The display target may be an object, a person, or any virtual image other than the captured virtual image 50 in the space 40. This allows the captured virtual image 50 to be displayed in association with other objects, people, virtual images (virtual objects, etc.), etc.

[0107] 34, a captured virtual image 50 may be displayed in a position corresponding to the outer surface of a spherical object 48 located in the space 40, in accordance with the shape of the outer surface. Also, the contour of the surface of an object or virtual image located in the space 40 may be specified, and a captured virtual image 50 having an outer shape that matches the contour may be displayed on the surface of the object or virtual image. Also, the display position of the captured virtual image 50 is not limited to the surface of the object, etc., and may be a position in the vicinity of the object, etc.

[0108] Furthermore, the CPU 11 may move the display position of the capture virtual image 50 as the second virtual image in response to the movement of the display target in the space 40. This allows the relationship between the capture virtual image 50 and other objects, people, virtual images, etc. to be dynamically expressed. For example, as shown in FIG. 35, in a case where the capture virtual image 50 is displayed at a position above the head of a person 44 as a display target, the capture virtual image 50 may be moved in response to the movement of the person 44. Furthermore, as shown in FIG. 35, the CPU 11 may display the capture virtual image 50 so that the capture virtual image 50 becomes larger as the distance between the user and the display target decreases. This allows the size of the capture virtual image 50 to express a sense of perspective, so that the capture virtual image 50 appears to move more naturally in the space 40.

[0109] Furthermore, the CPU 11 may change the orientation of the captured virtual image 50 as the second virtual image so as to follow the change in the orientation of the display object. For example, in the upper diagram of FIG. 36, the captured virtual image 50 is displayed directly above an arrow-shaped three-dimensional virtual object 54 (virtual image) as the display object, with the surface facing the direction of the arrow. Here, when the orientation of the virtual object 54 is changed as shown in the lower diagram, the orientation of the captured virtual image 50 is changed to match the orientation of the virtual object 54. Here, only the states before and after the orientation change are shown, but when the captured virtual image 50 rotates from the state shown in the upper diagram of FIG. 36 to the state shown in the lower diagram, the captured virtual image 50 may be rotated to follow the rotation.

[0110] Next, a capture virtual image display process for performing various operations related to the display of the above-mentioned capture virtual image 50 will be described with reference to the flowcharts of Fig. 37 and Fig. 38. Representative processes for performing representative operations are illustrated here. As explained for each operation above, the operations and processes related to the display of the capture virtual image 50 are not limited to these.

[0111] As shown in FIG. 37, when the capture virtual image display process is started, the CPU 11 displays the menu virtual image 61 on the display unit 14 in response to the user's third gesture operation (step S101). The CPU 11 determines whether or not the designation method of the capture area R has been designated (step S102). For example, the CPU 11 determines that the designation method of the capture area R has been designated when an operation of selecting the capture operation start button 611 or 612 in the menu virtual image 61 of FIG. 8 has been performed. When it is determined that the designation method of the capture area R has not been designated ("NO" in step S102), the CPU 11 executes step S102 again. When it is determined that the designation method of the capture area R has been designated ("YES" in step S102), the CPU 11 accepts the designation of the capture area R by the designated designation method (step S103), and determines whether or not the capture area R has been designated (step S104). If it is determined that the capture region R has not been specified ("NO" in step S104), the CPU 11 executes step S104 again.

[0112] When it is determined that the capture area R has been specified ("YES" in step S104), the CPU 11 determines whether or not the capture is prohibited (step S105). As exemplified above, capture is prohibited when the capture area R includes a capture prohibited object, when capture is prohibited based on the current position, when capture is prohibited in response to receiving a specific signal, when capture is prohibited in response to satisfying a connection condition related to the connection state to the communication network, and when capture is prohibited in response to a virtual image included in the capture area R satisfying a prohibition condition.

[0113] If it is determined that the capture is prohibited ("YES" in step S105), the CPU 11 transmits a signal requesting the lifting of the capture prohibition to an external device operated by an authorized person in response to a user's instruction, and determines whether or not an authorization signal permitting the lifting of the capture prohibition has been received from the external device (step S106). If the authorization signal has not been received within a predetermined period ("NO" in step S106), the CPU 11 terminates the capture virtual image display process. If the authorization signal has been received within a predetermined period ("YES" in step S106), or if it is determined in step S105 that the capture is not prohibited ("NO" in step S105), the CPU 11 generates a capture image C of the capture area R and stores it in the storage unit 13 (step S107).

[0114] The CPU 11 determines whether or not a second gesture operation for displaying the capture image C as the capture virtual image 50 has been performed (step S108), and if it is determined that the second gesture operation has not been performed ("NO" in step S108), executes step S108 again. If it is determined that the second gesture operation has been performed ("YES" in step S108), the CPU 11 causes the display unit 14 to display the capture virtual image 50 including the capture image C (step S109).

[0115] The CPU 11 determines whether the captured virtual image 50 includes an information extraction target (step S110). If it is determined that the information extraction target is included ("YES" in step S110), the CPU 11 displays an extracted information virtual image including the extracted information and an icon of a predetermined application on the display unit 14 (step S111). The CPU 11 determines whether a gesture operation for selecting an icon has been performed (step S112), and if it is determined that the gesture operation has been performed ("YES" in step S112), the CPU 11 executes the application corresponding to the icon and displays a virtual image of the application on the display unit 14 (step S113). When step S113 is completed, when it is determined that the information extraction target is not included in step S110 ("NO" in step S110), or when it is determined that a gesture operation for selecting an icon has not been performed in step S112 ("NO" in step S112), the CPU 11 ends the captured virtual image display process.

[0116] Second Embodiment Next, the configuration of a display system 1 according to a second embodiment will be described. As shown in Fig. 39, the display system 1 according to the second embodiment differs from the first embodiment in that it includes a wearable terminal device 10 and a plurality of external devices 20. Below, the differences from the first embodiment will be described, and a description of the commonalities will be omitted.

[0117] As shown in FIG. 39, the wearable terminal device 10 and the multiple external devices 20 included in the display system 1 are communicatively connected via a network N. The network N can be, for example, the Internet, but is not limited to this. The display system 1 may include multiple wearable terminal devices 10. The display system 1 may include only one external device 20. For example, in this embodiment, a user performing a predetermined task wears the wearable terminal device 10. A remote instructor who gives instructions to the user wearing the wearable terminal device 10 from a remote location via the wearable terminal device 10 operates the external device 20.

[0118] As shown in FIG. 40, the external device 20 includes a CPU 21, a RAM 22, a memory unit 23, an operation display unit 24, a communication unit 25, a microphone 26, a speaker 27, etc., and each of these units is connected by a bus 28.

[0119] The CPU 21 is a processor that performs various types of arithmetic processing and generally controls the operations of each part of the external device 20. The CPU 21 reads and executes a program 231 stored in the storage unit 23 to perform various control operations.

[0120] The RAM 22 provides a working memory space for the CPU 21 and stores temporary data.

[0121] The storage unit 23 is a non-transitory recording medium readable by the CPU 21 as a computer. The storage unit 23 stores a program 231 executed by the CPU 21, various setting data, and the like. The program 231 is stored in the storage unit 23 in the form of a computer-readable program code. As the storage unit 23, for example, a non-volatile storage device such as an SSD equipped with a flash memory or an HDD (Hard Disk Drive) is used.

[0122] The operation display unit 24 includes a display device such as a liquid crystal display, and an input device such as a mouse and a keyboard. The operation display unit 24 performs various displays such as the operation status and processing results of the display system 1 on the display device. The displays include, for example, an instructor screen 42 (see FIG. 42) including an image of the visual recognition area 41 captured by the camera 154 of the wearable terminal device 10. The operation display unit 24 also converts the user's input operation on the input device into an operation signal and outputs the operation signal to the CPU 21.

[0123] The communication unit 25 transmits and receives data to and from the wearable terminal device 10 in accordance with a predetermined communication protocol. The communication unit 25 can also perform voice data communication with the wearable terminal device 10. That is, the communication unit 25 transmits voice data collected by the microphone 26 to the wearable terminal device 10, and receives voice data transmitted from the wearable terminal device 10 to output voice from the speaker 27. The communication unit 25 may be capable of communicating with devices other than the wearable terminal device 10.

[0124] The microphone 26 converts sounds such as the voice of a remote instructor into electrical signals and outputs them to the CPU 21 .

[0125] The speaker 27 converts input audio data into mechanical vibrations and outputs the vibrations as sound.

[0126] In the display system 1 of the present embodiment, various data can be shared and a collaborative work can be performed by performing bidirectional data communication between the wearable terminal device 10 and one or more external devices 20. For example, data of an image captured by the camera 154 of the wearable terminal device 10 and data of the displayed virtual image 30 are transmitted to the external device 20 and displayed as an instructor screen 42 on the operation display unit 24, so that the remote instructor can recognize in real time what the user of the wearable terminal device 10 is viewing through the visor 141. In addition, voices collected by the microphone 17 of the wearable terminal device 10 and the microphone 26 of the external device 20 are transmitted by bidirectional voice data communication, so that a voice call can be performed. Thus, the period during which the voice data communication is being performed by the wearable terminal device 10 and the external device 20 includes the period during which the user of the wearable terminal device 10 and the remote instructor are engaged in a voice call. The remote instructor can provide instructions and assistance to the user of the wearable terminal device 10 through voice communication while viewing the real-time camera image on the instructor screen 42.

[0127] When the above-described captured virtual image 50 is displayed on the display unit 14 of the wearable terminal device 10, the display of the captured virtual image 50 can be reflected as it is on the instructor screen 42 of the external device 20. However, there are cases where it is desirable not to display the captured virtual image 50 on the instructor screen 42 of the external device 20, such as when the captured virtual image 50 contains confidential information. In this regard, in the display system 1 of the present disclosure, it is possible to set in advance whether or not to reflect the display of the captured virtual image 50 on the instructor screen 42. Data related to the setting may be stored in the storage unit 13 of the wearable terminal device 10 or in the storage unit 23 of the external device 20. In addition, even if the setting is set so that the captured virtual image 50 is not reflected on the instructor screen 42 of the external device 20, the display of the captured virtual image 50 may be reflected on the instructor screen 42 if the user of the wearable terminal device 10 allows it.

[0128] For example, it is assumed that a user A is logged in to the wearable terminal device 10, a user B is logged in to the external device 20, and screen sharing is being performed between the wearable terminal device 10 and the external device 20. It is also assumed that a setting is made such that the display of a captured virtual image 50 in the visible area 41 of the wearable terminal device 10 is not reflected on the instructor screen 42 of the external device 20. In this state, when the captured virtual image 50 is displayed on the wearable terminal device 10 as shown in Fig. 41, an area corresponding to the captured virtual image 50 is displayed on the instructor screen 42 of the external device 20 as shown in Fig. 42, but the content of the captured virtual image 50 is not displayed.

[0129] 42, a dialogue image 76 for inquiring of the remote instructor as to whether or not to request permission to display the captured virtual image 50 may be displayed on the instructor screen 42. When an operation for selecting a decision button 761 on the dialogue image 76 is performed, a request signal for requesting permission to display the captured virtual image 50 is transmitted to the wearable terminal device 10.

[0130] 43, in response to receiving the request signal, the wearable terminal device 10 displays on the display unit 14 a dialog virtual image 77 inquiring whether or not to permit display of a captured image. When a gesture operation is performed to select the decision button 771 of the dialog virtual image 77, a permission signal permitting display of the captured virtual image 50 is transmitted to the external device 20. On the instructor screen 42 of the external device 20, when the permission signal is received, the content of the captured virtual image 50 is displayed.

[0131] Third embodiment Next, a configuration of a display system 1 according to a third embodiment will be described. The third embodiment differs from the first embodiment in that an external information processing device 80 executes part of the processing executed by the CPU 11 of the wearable terminal device 10 in the first embodiment. Below, differences from the first embodiment will be described, and commonalities will be omitted. The third embodiment may be combined with the second embodiment.

[0132] As shown in FIG. 44, the display system 1 includes a wearable terminal device 10 and an information processing device 80 (server) communicatively connected to the wearable terminal device 10. At least a part of the communication path between the wearable terminal device 10 and the information processing device 80 may be wireless communication. The hardware configuration of the wearable terminal device 10 may be the same as that of the first embodiment, but a processor for performing the same processing as that performed by the information processing device 80 may be omitted. In addition, when this embodiment is combined with the second embodiment, the information processing device 80 may be connected to a network N.

[0133] As shown in FIG. 45, an information processing device 80 includes a CPU 81, a RAM 82, a storage unit 83, an operation display unit 84, a communication unit 85, and the like, and these units are connected to each other via a bus 86.

[0134] The CPU 81 is a processor that performs various types of arithmetic processing and generally controls the operation of each unit of the information processing device 80. The CPU 81 reads out and executes a program 831 stored in the storage unit 83 to perform various control operations.

[0135] The RAM 82 provides a working memory space for the CPU 81 and stores temporary data.

[0136] The storage unit 83 is a non-transitory recording medium readable by the CPU 81 as a computer. The storage unit 83 stores a program 831 executed by the CPU 81, various setting data, and the like. The program 831 is stored in the storage unit 83 in the form of a computer-readable program code. As the storage unit 83, for example, an SSD equipped with a flash memory, or a non-volatile storage device such as an HDD is used.

[0137] The operation display unit 84 includes a display device such as a liquid crystal display, and an input device such as a mouse and a keyboard. The operation display unit 84 displays various information such as the operation status and processing results of the display system 1 on the display device. Here, the operation status of the display system 1 may include real-time images captured by the camera 154 of the wearable terminal device 10. The operation display unit 84 also converts the user's input operation on the input device into an operation signal and outputs the operation signal to the CPU 21.

[0138] The communication unit 85 transmits and receives data by communicating with the wearable terminal device 10. For example, the communication unit 85 receives data including a part or all of the detection results by the sensor unit 15 of the wearable terminal device 10, and information related to a user's operation (gesture) detected by the wearable terminal device 10. The communication unit 85 may also be capable of communicating with devices other than the wearable terminal device 10.

[0139] In the display system 1 having such a configuration, the CPU 81 of the information processing device 80 executes at least a part of the processing executed by the CPU 11 of the wearable terminal device 10 in the first embodiment. For example, the CPU 81 may perform three-dimensional mapping of the space 40 based on the detection result by the depth sensor 153. The CPU 81 may also detect the user's visual recognition area 41 in the space 40 based on the detection result by each part of the sensor unit 15. The CPU 81 may also generate virtual image data 132 related to the virtual image 30 in response to the operation of the user of the wearable terminal device 10. The CPU 81 may also detect the position and orientation of the user's hand (and / or fingers) based on the captured image by the depth sensor 153 and the camera 154. The CPU 81 may also execute processing related to the generation of the capture image C and the display of the capture virtual image 50.

[0140] The above-mentioned processing results by the CPU 81 are transmitted to the wearable terminal device 10 via the communication unit 85. The CPU 11 of the wearable terminal device 10 operates each unit (e.g., the display unit 14) of the wearable terminal device 10 based on the received processing results. The CPU 81 may also transmit a control signal to the wearable terminal device 10 to perform display control of the display unit 14 of the wearable terminal device 10.

[0141] In this way, by executing at least a part of the processing in the information processing device 80, it is possible to simplify the device configuration of the wearable terminal device 10 and reduce the manufacturing cost. Furthermore, by using a higher performance information processing device 80, it is possible to increase the speed and accuracy of various processes related to MR. Therefore, it is possible to increase the accuracy of 3D mapping of the space 40, improve the display quality of the display unit 14, and increase the response speed of the display unit 14 to the user's actions.

[0142] 〔others〕 The above embodiment is merely an example, and various modifications are possible.

[0143] For example, in each of the above-described embodiments, the visor 141 having optical transparency is used to allow the user to view the real space, but the present invention is not limited to this. For example, the visor 141 having light blocking properties may be used to allow the user to view the image of the space 40 captured by the camera 154. That is, the CPU 11 may display on the display unit 14 the image of the space 40 captured by the camera 154 and the virtual image 30 superimposed on the image of the space 40. With such a configuration, MR in which the virtual image 30 is blended with the real space can also be realized.

[0144] Moreover, by using a pre-generated image of a virtual space instead of an image of a real space captured by the camera 154, a VR that allows the user to experience being in the virtual space can be realized. In this VR as well, the user's visual recognition area 41 is specified, and a portion of the virtual space that is inside the visual recognition area 41 and a virtual image 30 whose display position is determined inside the visual recognition area 41 are displayed. Therefore, the background of the captured image C in this case is the virtual space.

[0145] The wearable terminal device 10 is not limited to the one having the ring-shaped main body 10a illustrated in Fig. 1, and may have any structure as long as it has a display unit that is visible to the user when worn. For example, it may have a structure that covers the entire head like a helmet. Also, it may have a frame that is hung on the ears like glasses, and various devices may be built into the frame.

[0146] The various virtual images do not necessarily have to be stationary in the space 40, but may move within the space 40 on a predetermined trajectory.

[0147] Although an example in which a gesture of a user is detected and accepted as an input operation has been described, the present invention is not limited to this. For example, an input operation may be accepted by a controller that the user holds in his / her hand or wears on his / her body.

[0148] Although an example of a voice call between the wearable terminal device 10 and the external device 20 has been described, this is not limiting, and a video call may be possible. In this case, a web camera for capturing an image of the remote operator may be provided in the external device 20, and image data captured by the web camera may be transmitted to the wearable terminal device 10 and displayed on the display unit 14.

[0149] In addition, the specific details of the configurations and controls shown in the above embodiments can be appropriately changed without departing from the spirit of the present disclosure. Also, the configurations and controls shown in the above embodiments can be appropriately combined without departing from the spirit of the present disclosure. [Industrial Applicability]

[0150] The present disclosure can be used in a wearable terminal device, a program, and an image processing method. [Explanation of symbols]

[0151] 1 Display System 10 Wearable terminal device 10a Main body 11 CPU (Processor) 12 RAM 13 Storage section 131 Programs 132 Virtual Image Data 14 Display section 141 Visor (display part) 142 Laser Scanner 15 Sensor section 151 Accelerometer 152 Angular velocity sensor 153 Depth Sensor 154 Camera 155 Eye Tracker 16 Communications Department 17. Mike 18 Speakers 19 Bus 20 External equipment 21 CPU 23 Memory section 231 Programs 24 Operation display section 30 Virtual Image (First Virtual Image) 31 Feature Bar 32 Window shape change button 33 Close button 34, 35 Virtual image (4th virtual image, 5th virtual image) 40 space 41 Visibility Zone 411 Virtual Line 412 Pointer 42 Instructor screen 44 People 45 Goods 46 characters 47 Two-dimensional code (code information) 48 Object 50 Capture Virtual Image (Second Virtual Image) 50a Initial display area 50b Extension Area 51 Virtual Image Area 52 Spatial Image Region 53 Play button 54 Virtual Objects 61 Menu Virtual Image (Third Virtual Image) 62-65, 71 Extraction information virtual image (6th virtual image) 622, 632, 642, 661~663 Icons (signs) 67~69, 72, 73, 623, 633, 643 Virtual images 74, 75, 77 Dialogue Virtual Images 76 Dialogue Images 80 Information processing device 81 CPU 83 Memory section 831 Programs 84 Operation display section C. Captured image D. Photographed image E. Composite image L List area N Network R Capture Area r Capture frame U User

Claims

1. A wearable terminal device that is worn by a user, A camera that captures a space as the user's visual recognition area; At least one processor; The at least one processor: Identifying a part of the visible area in the space photographed by the camera as a capture area based on a first gesture operation of the user; The wearable terminal device stores a capture image corresponding to the capture area in a storage unit.

2. A display unit having a light-transmitting display member, The wearable terminal device according to claim 1 , wherein the at least one processor causes the first virtual image to be displayed on a display surface of the display member so that the first virtual image is viewed in the space viewed through the display member.

3. A display unit is provided, The wearable terminal device according to claim 1 , wherein the at least one processor causes the display unit to display an image of the space captured by the camera and a first virtual image superimposed on the image of the space.

4. The wearable terminal device according to claim 2 , wherein the at least one processor is configured to, when storing the captured image in the storage unit, display the captured image on the display unit as a second virtual image.

5. The wearable terminal device according to claim 2 , wherein the at least one processor causes the display unit to display the captured image as a second virtual image in response to a second gesture operation by a user.

6. The at least one processor displays the second virtual image at a predetermined position relative to the user in response to the second gesture operation; The wearable terminal device according to claim 5 , wherein the relative position is set in advance for each type of the second gesture operation.

7. The wearable terminal device according to any one of claims 4 to 6, wherein the at least one processor displays the second virtual image in the space at a position closer to the user than the first virtual image.

8. The wearable terminal device according to any one of claims 4 to 7, wherein the at least one processor causes the display unit to display a third virtual image for starting to accept the first gesture operation in response to a third gesture operation by the user.

9. The at least one processor: Identifying the capture area by a method different from each other depending on a type of the first gesture operation; The wearable terminal device according to claim 8 , further comprising: a step of identifying a type of the first gesture operation to be accepted in response to an operation on the third virtual image.

10. The wearable terminal device according to any one of claims 4 to 9, wherein the at least one processor displays an outer frame of the second virtual image in a predetermined highlighted manner.

11. The wearable terminal device according to any one of claims 4 to 10, wherein the at least one processor moves the second virtual image in response to a fourth gesture operation by a user on the displayed second virtual image.

12. The wearable terminal device according to any one of claims 4 to 11, wherein the at least one processor expands or reduces the second virtual image in response to a user's fifth gesture operation on the displayed second virtual image, and expands or reduces the range of the capture area to be reflected in the second virtual image in response to the expansion or contraction.

13. A wearable terminal device according to any one of claims 4 to 12, wherein the at least one processor stores the capture image including at least a portion of the first virtual image in the memory unit when the capture area includes at least a portion of the first virtual image.

14. The wearable terminal device according to claim 13, wherein the at least one processor deletes the first virtual image included in the second virtual image in response to a sixth gesture operation by a user on the displayed second virtual image, and displays an image of the space corresponding to the deleted area in the deleted area.

15. The wearable terminal device according to claim 14 , wherein the at least one processor causes the first virtual image included in the second virtual image to be displayed in a predetermined highlighted manner.

16. The wearable terminal device according to claim 13 , wherein the at least one processor deletes a portion of the second virtual image other than the first virtual image in response to a seventh gesture operation by the user on the displayed second virtual image.

17. The wearable terminal device according to claim 16 , wherein the at least one processor displays a portion of the second virtual image other than the first virtual image in a predetermined highlighted manner.

18. The wearable terminal device according to claim 13, wherein when the second virtual image includes a portion of the first virtual image, the at least one processor duplicates the first virtual image and displays it as a fourth virtual image in response to an eighth gesture operation by the user on the displayed second virtual image.

19. the at least one processor accepts, in response to a ninth gesture operation by a user on the displayed second virtual image, a start of an operation of moving the first virtual image included in the second virtual image by one of a plurality of moving methods; The wearable terminal device according to claim 13 , wherein the plurality of movement methods include a method of expanding or reducing a display range of the first virtual image within the second virtual image, and a method of moving the first virtual image outside the second virtual image.

20. The wearable terminal device according to any one of claims 4 to 19, wherein the at least one processor replicates at least a portion of the second virtual image and displays it as a fifth virtual image in response to a user's tenth gesture operation on the displayed second virtual image.

21. The at least one processor, when the second virtual image includes an extraction object from which information can be extracted, extracts the information from the extraction object and displays it on the display unit. A wearable terminal device according to any one of claims 4 to 20.

22. The wearable terminal device according to claim 21 , wherein the at least one processor translates the extracted information according to a predetermined translation setting or according to a translation setting corresponding to an eleventh gesture operation of the user, and displays the translated information on the display unit.

23. The wearable terminal device according to any one of claims 4 to 22, wherein the at least one processor, when the second virtual image includes an extraction object from which information can be extracted, causes the display unit to display a sign for launching a predetermined application according to the type of the extraction object.

24. The at least one processor: displaying a sixth virtual image including the information extracted from the extraction target on the display unit; The wearable terminal device according to claim 23 , wherein the marker is displayed on at least one of a first surface of the sixth virtual image and a second surface opposite to the first surface.

25. The wearable terminal device according to any one of claims 21 to 24, wherein the extraction target is at least one of a person, an object, a place, text, and code information.

26. The at least one processor: acquiring position information of the own device when the first gesture operation is performed; The wearable terminal device according to any one of claims 1 to 25, wherein the acquired position information is stored in the storage unit in association with the captured image.

27. The at least one processor: Identifying the user when the first gesture operation is performed; The wearable terminal device according to any one of claims 1 to 26, wherein user information relating to the identified user is stored in the storage unit in association with the capture image.

28. The at least one processor: determining whether or not the capture area includes a predetermined object that cannot be captured; The wearable terminal device according to any one of claims 1 to 27, wherein when it is determined that the capture area includes the object that is prohibited from being captured, the captured image is not stored in the memory unit.

29. The at least one processor: acquiring position information of the own device when the first gesture operation is performed; The wearable terminal device according to any one of claims 1 to 28, wherein if the position indicated by the position information satisfies a predetermined prohibited position condition, the captured image is not stored in the memory unit.

30. A communication unit is provided for performing data communication with an external device, The wearable terminal device according to any one of claims 1 to 29, wherein the at least one processor does not store the captured image in the memory unit when the communication unit is receiving a specific signal.

31. A communication unit is provided for performing data communication with an external device, The communication unit performs the data communication via a predetermined communication network, The wearable terminal device according to any one of claims 1 to 29, wherein the at least one processor does not store the captured image in the memory unit when a predetermined connection condition related to the connection state of the communication unit to the communication network is satisfied.

32. A wearable terminal device as described in any one of claims 2 to 25, wherein the at least one processor does not store the capture image in the memory unit when the capture area includes at least a portion of the first virtual image and the first virtual image satisfies a predetermined prohibition condition.

33. Equipped with a microphone that converts voice into audio data, The at least one processor, when storing the captured image in the memory unit, acquires the audio data and associates it with the captured image to store it in the memory unit. A wearable terminal device according to any one of claims 1 to 32.

34. The wearable terminal device according to any one of claims 4 to 25, wherein the at least one processor displays the second virtual image on the surface of a specified display object located in the space, or at a position that satisfies a specified positional relationship with the display object.

35. The wearable terminal device according to claim 34 , wherein the at least one processor moves a display position of the second virtual image in response to a movement of the display object in the space.

36. The wearable terminal device according to claim 35 , wherein the at least one processor displays the second virtual image such that the second virtual image becomes larger as the distance between the user and the display target decreases.

37. The wearable terminal device according to any one of claims 34 to 36, wherein the at least one processor changes an orientation of the second virtual image to follow a change in orientation of the display object.

38. The wearable terminal device according to any one of claims 34 to 37, wherein the display target is an object, a person, or any virtual image other than the second virtual image in the space.

39. A computer is provided in a wearable terminal device that is worn by a user and has a camera that captures an image of a space as the user's visual recognition area. A process of identifying a part of the visible area in the space photographed by the camera as a capture area based on a first gesture operation of the user; A process of storing a capture image corresponding to the capture area in a storage unit; A program that executes the following.

40. An image processing method executed by a computer provided in a wearable terminal device that is worn by a user and has a camera that captures an image of a space as the user's visual recognition area, Identifying a part of the visible area in the space photographed by the camera as a capture area based on a first gesture operation of the user; The image processing method further comprises storing a capture image corresponding to the capture area in a storage unit.

Citation Information

Patent Citations

  • Display device, method for controlling display device, and program

    JP2016161734A

  • Human body gesture-based region and volume selection for hmd

    JP2016514298A

  • Picture-Taking Within Virtual Reality

    US20190377416A1

  • Information processing device and application execution method

    WO2014181380A1

  • Passive optical and inertial tracking in slim form-factor

    US20190087021A1