Wearable terminal, system, and display method

Abstract

To provide a wearable terminal, a system, and a display method that can easily adjust the display position of an image.SOLUTION: According to an embodiment, a wearable terminal includes: a projection device that projects a display image in front of a user to form a virtual image corresponding to the display image in front of the user; a camera that photographs an image the user is looking at; and a gesture sensor that analyzes the motion of the user's hands and fingers in the image photographed by the camera to discriminate the user's gesture. The virtual image is changed according to the user's gesture detected by the gesture sensor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a wearable terminal, a system, and a display method. [Background technology]

[0002] In manufacturing sites or factories with numerous manufacturing machines, the availability of these machines is a major factor affecting product production volume. Long periods of downtime due to preventable equipment breakdowns caused by inadequate regular maintenance and inspections, or due to inadequate response to sudden equipment malfunctions, result in lower availability and reduced product production. Therefore, minimizing downtime is essential. Because maintenance, inspection, and repair procedures vary for each piece of manufacturing equipment, workers often refer to manuals or checklists (collectively referred to as checklists) that outline the procedures for each step of the process. In recent years, wearable devices have become increasingly popular in manufacturing sites. For example, workers wear eyeglass-type wearable devices with checklists displayed on the lenses. This eliminates the need for workers to refer to paper checklists while working, allowing them to efficiently perform unfamiliar or complex tasks without stopping.

[0003] However, because checklists are required to ensure the accuracy of work completion at each stage, and although reference is made electronically on a screen, paper checklists are still prepared, and workers stop working at each stage to fill them out. This forces the manufacturing equipment to be stopped, leading to a decrease in production volume. Furthermore, after returning to the office, workers create work reports based on these checklists. Creating these work reports is cumbersome for workers.

[0004] Systems that use camera-equipped head-mounted displays to assist operators in their work are being developed, such as a medical equipment management system that assists operators in handling used and contaminated medical equipment such as endoscopes, scalpels, forceps, and other instruments.

[0005] This system comprises a head-mounted camera that captures the field of view of an operator handling medical equipment or instruments, a storage means that stores an image of the operator's field of view captured during exemplary work when handling medical equipment or instruments as a standard image, a first discrimination means that compares the image captured by the camera with the standard image read from the storage means and determines whether a specified work is being performed standardly based on the similarity between the two images, an information output means that outputs information indicating a warning or instruction based on the discrimination result by the discrimination means, and an output means that issues a warning or instruction to the operator based on the information indicating the warning or instruction. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-279193 Summary of the Invention [Problem to be solved by the invention]

[0007] This system automatically recognizes the operator's work by comparing images of the operator handling medical instruments captured by a camera with standard images prepared in advance. However, this method uses simple pattern matching between images, which results in low recognition accuracy, and requires complex image processing such as extracting feature parts from the images. This makes the automatic recognition process highly complex and takes a long time.

[0008] An object of the present invention is to provide a wearable terminal, a system, and a display method that allow the display position of an image to be easily adjusted. [Means for solving the problem]

[0009] According to an embodiment, a wearable device includes a display that displays a display image, a projection device that projects the display image in front of a user to form a virtual image corresponding to the display image in front of the user, a camera that captures an image in front of the user's line of sight, and a gesture sensor that analyzes the movement of the user's hands and fingers in the image captured by the camera and determines the user's gesture. The projection device includes a lens that projects divergent light emitted from the display in front of the user, and a first user operation unit. The first user operation unit is capable of adjusting the projection angle of the projection device by rotating it. The user can adjust the display position of the virtual image to match the shape or size of the user's head by rotating the first user operation unit while viewing the virtual image. The virtual image is changed in accordance with the user's gesture determined by the gesture sensor. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a perspective view illustrating an example of a wearable terminal according to an embodiment. [Figure 2] 1A and 1B are a front view and a cross-sectional view showing an example of a wearable terminal; [Figure 3] FIG. 10 is a diagram illustrating an example of position detection of a wearable device. [Figure 4] FIG. 1 is a diagram illustrating the principle of position detection of a wearable device. [Figure 5] 10A and 10B are diagrams illustrating signal waveforms when detecting the position of a wearable device. [Figure 6] FIG. 1 is a diagram illustrating an example of a system including a wearable terminal and an information management server. [Figure 7] FIG. 2 is a block diagram showing the electrical configuration of the wearable terminal. [Figure 8] FIG. 2 is a diagram illustrating an example of a sensor that detects the state of the device. [Figure 9] FIG. 10 is a diagram illustrating an example of a sensor that detects a user's behavior. [Figure 10]FIG. 1 is a schematic diagram illustrating an example of a usage environment of a system. [Figure 11] FIG. 2 is an exploded perspective view showing the structure of an example of a sensor used in the system to detect user behavior. [Figure 12] FIG. 10 is an exploded perspective view showing the structure of another example of a sensor for detecting user behavior used in the system. [Figure 13] 1A and 1B are diagrams illustrating an example of a work procedure displayed by the system and a work record created by the system. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described with reference to the drawings.

[0012] Wearable devices come in a variety of types, including head-mounted types (including glasses, goggles, and helmets, which are sometimes collectively referred to as glasses), wristband types, and pendant types, but here we will explain an embodiment of a glasses-type wearable device. Glasses-type wearable devices include types that allow you to see the scenery ahead through transparent lenses, and types known as head-mounted displays that obstruct your field of vision and prevent you from seeing the scenery. Here, we will explain the type that allows you to see the scenery as an example.

[0013] FIG. 1 is a perspective view of a glasses-type wearable terminal (hereinafter simply referred to as a wearable terminal) 10, FIG. 2(a) is a front view, and FIG. 2(b) is a view showing a cross-sectional structure seen from above.

[0014] The wearable device 10 has almost the same shape as regular eyeglasses, except that in this case, a projection device 12 is attached to the outside of the temple on the right eye side. Glasses 14 and 16 are fitted into the frame. The glass 14 on the left eye side is regular transparent glass so that the user can see the scenery. At least a portion of the glass 16 on the right eye side serves as a screen 16. The screen 16 allows the user to see the image projected by the projection device 12. The screen 16 is transparent when the projection device 12 is not projecting an image, and the user can see the scenery through the glass (screen) 16 on the right eye side.

[0015] The projection device 12 includes, as electronic components, a power supply unit 22 and a control unit 24. The power supply unit 22 may include a button battery, a rechargeable battery, a secondary battery capable of receiving power in a wireless manner, or the like. Alternatively, the power supply unit 22 may not be built-in, and power may be supplied to the projection device 12 from an external power source via a power line. The control unit 24 communicates with a server or other electronic devices via a network, which will be described later, to send and receive information. This communication may be wired or wireless. In the case of wireless communication, short-range wireless communication such as Bluetooth (registered trademark), ZigBee (registered trademark), or UWB, medium-range wireless communication such as WiFi (registered trademark), or long-range wireless communication such as 3G / 4G or WiMAX (registered trademark) may be used depending on the usage environment.

[0016] The projection device 12 further includes optical components such as a light source 28, a display unit 30, a prism 32, and a lens group 34. The light source 28 may be a dimmable white LED light source including multiple (e.g., three) LEDs (Light Emitting Diodes) that emit different light colors and whose output light intensity can be independently adjusted. The dimmable white LED light source allows the emitted light color to be changed according to the usage environment, even in environments where the wearable device 10 is used, such as clean rooms where orange-based lighting is often used, and produces clear projection images. Furthermore, the dimmable white LED light source can output display colors that are easy for users to see, which can prevent eye fatigue and associated migraines, which can be a nuisance for users, compared to display colors that are difficult for users to see.

[0017] The display unit 30 is, for example, a reflective LCD (Liquid Crystal Display) module, and displays predetermined text, images, etc. (hereinafter, what is displayed on the display unit 30 may be collectively referred to as a display image) based on display control by the control unit 24. Non-parallel light (divergent light, hereinafter also referred to as divergent light) emitted from the light source 28 is reflected by the half-mirror surface 32a of the prism 32 and illuminates the display image on the display unit 30. The reflected light from the display unit 30 passes through the half-mirror surface 32a as light corresponding to the display image (sometimes referred to as image light), is emitted from the emission surface 32c, and is projected onto the screen 16 via the lens group 34 as a projection image of a predetermined size.

[0018] The screen 16 has a front-side transparent refraction body 42, a Fresnel lens-type half-mirror surface 44, and a rear-side transparent refraction body 46. A portion of the image light that reaches the Fresnel lens-type half-mirror surface 44 is reflected by the Fresnel lens-type half-mirror surface 44, forming a virtual image (projected image) several meters away that corresponds to the image displayed on the display unit 30. Note that the screen 16 can also partially transmit the scenery in the line of sight of the user wearing the wearable device 10, and may be configured to display a scenery visible to the user on the screen 16 together with the projected image.

[0019] A portion of the image light (divergent light) emitted from the light source 28 and passing through the half mirror surface 32a is totally reflected by the total reflection surface 32b and refracted by the emission surface 32c to become leakage light 50, which is divergent light from the light source 28. The leakage light 50 passes through an opening or gap (guiding portion) 52 formed in front of the projection device 12 and is emitted in a direction different from that of the screen 16.

[0020] The wearable terminal 10 has a speaker 54A, an earphone jack 54B, a microphone jack 56, a slide switch 57, a rotary switch 58, and the like at predetermined positions on the projection device 12, for example, on the bottom surface. A hands-free microphone (not shown) is connected to the microphone jack 56 to collect the user's voice. The slide switch 57 can adjust, for example, the brightness and color tone of the image projected by the projection device 12. The rotary switch 58 can adjust, for example, the projection angle of the image. By allowing different adjustment amounts to be set by different operations, such as the slide switch 57 and the rotary switch 58, the user can adjust the projection image by touch while visually checking the projection image. For example, by operating the slide switch 57, a projection image with display brightness and color tone that suits the user's preference can be provided. By operating the rotary switch 58, the projection angle can be adjusted to display the image in an optimal position according to the shape and size of the user's head. Of course, the objects adjusted by the slide switch 57 and the rotary switch 58 may be reversed, the positions of the slide switch 57 and the rotary switch 58 may be reversed, or both may be assigned to two types of operation of a single operating member.

[0021] Selection using these switches 57 and 58 may be made by trial and error while viewing only the projected image, but to increase the efficiency of adjustment, a menu screen may be projected and adjustment may be made by selecting an item on that screen. When the display unit 30 displays the menu screen, the menu screen is projected onto the screen 16.

[0022] Furthermore, menu items may be selected by touch operation instead of by operating switches 57 and 58. For this reason, a touchpad 55 is also provided on the outside of projection device 12. When display unit 30 displays a menu or the like, a user can easily and efficiently input operations by touching a position on touchpad 55 that corresponds to the display position of an item in the menu.

[0023] A camera 59 is provided on the outside of the center of the front face, and is capable of capturing an image (either a still image or a video) in the direction of the user's line of sight. Although not shown, a camera may be provided inside the center of the front face (at a position corresponding to the position of camera 59) facing the user's face, capturing an image of the user's eyeballs and detecting the user's iris. The iris can be used for user authentication.

[0024] The state of the wearable terminal 10, i.e., the state of the user, can be detected by utilizing the leaked light 50 from the wearable terminal 10. The principle of detecting the state of the wearable terminal will be explained with reference to Figures 3, 4, and 5. Here, the state includes the position, movement of the position, etc.

[0025] An example of how a wearable device can be used is shown in Figure 3. For example, an arbitrary number of work spaces or product shelves A01 to Axy (x and y are both positive integers), B01 to Bxy, and C01 to Cxy are arranged in a work area 60 such as a parts yard in a factory, a product warehouse in a mail-order company, or a delivery department in a retail store. The work spaces or product shelves may be, for example, work tables in a factory, manufacturing equipment in a production line, desks in a school, seating positions in a conference room, etc.

[0026] At least one optical sensor 62-1 to 62-n (n is a positive integer) is arranged in work area 60. Optical sensors 62-1 to 62-n can individually detect the positions (x, y, z), number, changes in position (movement), and changes in orientation of wearable terminals 10-1 to 10-m (m is a positive integer) using the detection methods shown in FIGS. 4 and 5. By detecting the positions, number, movements, and changes in orientation of wearable terminals 10-1 to 10-m, the positions, movements, and other statuses of any number of users wearing wearable terminals 10-1 to 10-m can be recognized.

[0027] A user can freely move within the work area 60. The user performs a predetermined task in a work space 64, which is a predetermined position, such as a station (cart), a container equivalent thereto, or a movable table. Note that the work space 64 may not be movable, but may be a fixed desk or a seating position thereon.

[0028] As shown in FIGS. 3 and 4 , the detection system includes one or more wearable terminals 10 and one or more optical sensors 62. The optical sensors 62 have a function of detecting leaked light 50 and a communication function of transmitting the detection results to a server or the like. Like the communication function of the wearable terminal 10, this communication function may be either wired or wireless. In the case of wireless communication, any of short-range wireless communication such as Bluetooth, ZigBee, UWB, etc., medium-range wireless communication such as WiFi, etc., and long-range wireless communication such as 3G / 4G, WiMAX, etc. may be used depending on the usage environment. The embodiments described below include various units and modules with communication functions, and the communication functions of these units and modules may also be either wired or wireless. In the case of wireless communication, any of short-range wireless communication such as Bluetooth, ZigBee, UWB, etc., medium-range wireless communication such as WiFi, etc., and long-range wireless communication such as 3G / 4G, WiMAX, etc. may be used depending on the usage environment.

[0029] The wearable device 10 intermittently modulates the leaked light 50 using information including the device's identification information (hereinafter also referred to as device ID) so that the optical sensor 62 can identify the wearable device 10 from the leaked light 50. A typical example of a modulation method is a chopper modulation method that reduces the amount of light emitted to zero, but here a modulation method is adopted that can ensure a predetermined amount of light emission or more even when the amount of light emitted is low. This reduces the strain on the user's eyes. As a modulation method, for example, DSV (Digital Signal Velocity) By adopting a (Sum Value)-free modulation method (i.e., a modulation method that constantly calculates the DSV of the modulated signal and inserts appropriate bit inversion codes to zero the DC component), changes in the light emission intensity over a relatively long range can be suppressed, macroscopically eliminating changes in the light emission intensity, further reducing eye strain. Because the human eye can detect changes up to approximately 0.02 seconds, setting the reference frequency of the modulation to 10 Hz or higher, e.g., 20 Hz or higher, and more preferably 60 Hz or higher, also reduces eye strain. Meanwhile, because the LED used in the light source 28 has internal impedance and connection capacitance, a modulation frequency with good accuracy is less than 100 MHz, preferably 10 MHz or lower. Therefore, the modulation frequency of the light source 28 used in the detection system of the embodiment is preferably in the range of 10 Hz to 100 MHz, preferably 10 Hz to 10 MHz.

[0030] Since the leakage light 50 of divergent light from the light source 28 is used, the amount of light detected by the optical sensor 62 changes depending on the distance between the wearable terminal 10 and the optical sensor 62. By utilizing this phenomenon, it is possible to determine the distance between the wearable terminal 10 and the optical sensor 62 or the orientation of the wearable terminal 10 with respect to the optical sensor 62. Because the position (including the height) of the optical sensor 62 is fixed, once the distance between the optical sensor 62 and the wearable terminal 10 is known, the position (x, y, z) of the wearable terminal 10 can be detected.

[0031] Furthermore, since the system uses the leakage light 50 of divergent light from the light source 28, it is possible to detect the leakage light 50 over a relatively wide range. As a result, by simply installing a relatively small number of optical sensors 62-1 to 62-n, it is possible to detect the positions of the wearable terminals 10-1 to 10-m within the work area 60, the distance between the wearable terminal 10 and the optical sensor 62, the direction of the wearable terminals 10-1 to 10-m, or the orientation of the wearable terminal 10 relative to the optical sensor 62. This reduces the equipment costs required to install the detection system.

[0032] The light intensity information of leak light 50 detected by light sensor 62 is transmitted from light sensor 62 to a server (described later) at a predetermined timing. The server analyzes the information collected from light sensor 62. This makes it possible to detect the position and state of any wearable terminal 10-1 to 10-m, i.e., the user.

[0033] 4 is a schematic diagram illustrating a specific example of use of the system for recognizing wearable devices according to the embodiment. Assume a situation in which three users wearing wearable devices 10-1 to 10-3 are positioned around four optical sensors 62-1 to 60-4. Light leakage 50 from wearable devices 10-1 and 10-2 is detected by the optical sensors 62-1 to 60-4. The optical sensors 62-1 to 60-4 convert the amount of light leakage 50 detected by each sensor into analog-to-digital (AD) data, and transmit the converted data as light intensity information corresponding to the amount of light to a server at a predetermined timing, for example, via short-range wireless communication.

[0034] Suppose that wearable terminal 10-1 moves toward optical sensor 62-1 in response to the user's movement, while the orientation of wearable terminal 10-2 temporarily changes in response to an arbitrary movement of the user, such as a head shake (turning of the head). The change in detection information at this time is shown in FIG.

[0035] 5 shows an example in which an intermittent time-varying method is used as the modulation method for leaked light 50 of each of wearable terminals 10-1 to 10-3. That is, the ID modulation periods are shifted between wearable terminals 10-1 to 10-3.

[0036] 5(a), (b), and (c), ID modulation periods for the first to third wearable terminals 10-1 to 10-3 are set intermittently, and the remaining periods are non-modulation periods. Within each ID modulation period, the synchronization signal SYNC and the terminal ID of wearable terminal 10-1 to 10-3 form a pair (one-to-one correspondence), and this pair is repeated multiple times (a multiple of four times if there are four sensors as shown in FIG. 5).

[0037] At the same time that the first wearable terminal 10-1 enters the non-modulation period, the ID modulation period of the second wearable terminal 10-2 begins. Similarly, at the same time that the second wearable terminal 10-2 enters the non-modulation period, the ID modulation period of the third wearable terminal 10-3 begins.

[0038] During the ID modulation period of the second wearable terminal 10-2 and the ID modulation period of the third wearable terminal 10-3, the synchronization signal SYNC and the terminal IDs of the wearable terminals 10-2 and 10-3 are repeatedly modulated. In this way, the terminal ID of the wearable terminal 10 is included in the modulated signal, making it possible to detect the terminal ID.

[0039] In the above example, the modulation timing of each of the wearable terminals 10-1 to 10-3 is time-divided (intermittent). However, for example, all of the wearable terminals 10-1 to 10-3 may be continuously modulated, and the modulation reference frequency of each of the wearable terminals 10-1 to 10-3 may be changed. Furthermore, the frequency spectrum characteristics of each of the wearable terminals 10-1 to 10-3 during spectrum spreading may be changed.

[0040] As shown in FIGS. 5(d), (e), (f), and (g), the information communication period from the optical sensors 62-1 to 62-4 is divided into small intervals in each ID modulation period.

[0041] As shown in FIG. 4, initially, part of the leaked light from the wearable terminal 10-1 reaches the optical sensor 62-4. Therefore, initially, as shown in FIG. 5(k), the optical sensor 62-4 detects the leaked light from the wearable terminal 10-1. However, as the wearable terminal 10-1 moves toward the optical sensor 62-1, the amplitude of the modulated signal of the leaked light from the wearable terminal 10-1 detected by the optical sensor 62-4 decreases. On the other hand, as shown in FIG. 5(h), the amplitude of the modulated signal of the leaked light from the wearable terminal 10-1 detected by the optical sensor 62-1 increases over time. In this way, by comparing the temporal changes in the amplitude of the modulated signals detected by the optical sensors 62-1 to 62-n, it is possible to detect the temporal change in the position (movement state) of the wearable terminals 10-1 to 10-m that are the detection targets.

[0042] On the other hand, since wearable terminal 10-2 is initially facing optical sensor 62-3, the amplitude of the modulated signal obtained from the leaked light is greater at optical sensor 62-3 than at optical sensor 62-2. After this, for example, suppose the second user turns their head and temporarily faces toward optical sensor 62-2. Then, as shown in FIG. 5(i), the detection output of wearable terminal 10-2 output by optical sensor 62-2 temporarily increases and then decreases. On the other hand, the detection output of wearable terminal 10-2 output by optical sensor 62-3 temporarily decreases and then increases, as shown in FIG. 5(j).

[0043] In this way, by comparing the change over time in the amplitude of the modulated signal detected by optical sensor 62, it is possible to estimate the change over time in the orientation of wearable terminals 10-1 to 10-m that are the detection targets.

[0044] The above detection examples are examples where the user's movements are movements or head movements. However, various other user actions may be used. For example, the user's hand movements or twisting of the upper body (body) may temporarily block leaked light. In this case, a temporary decrease in the amplitude of the modulated signal occurs in common at the same time period in all of the optical sensors 62-1 to 60-4. In this way, by comparing the correlation of the changes in the amplitude of the modulated signals of all of the optical sensors 62-1 to 60-4, different behavioral patterns of the user can be identified.

[0045] By using the above method, it is possible to not only detect the user's actions but also recognize the user's intentions.

[0046] Note that beacons may also be used as a method for detecting the position (x, y, z) of the wearable device 10. In the above example, light modulated with device identification information is emitted from a large number of wearable devices 10 to a large number of optical sensors 60, and the information received by the large number of optical sensors 60 is compared and processed to detect the position and state of the wearable device 10. However, if a large number of position information transmitters are placed within the work area 60 and the transmitters transmit beacons corresponding to their positions via short-range wireless communication such as RF-ID with a reach of several meters, the wearable device 10 that receives the beacons can be considered to be in approximately the same position as the transmitter. Furthermore, the position of the wearable device can also be detected using GPS. Position detection does not need to be based on only one method; using multiple methods in combination can improve detection accuracy.

[0047] FIG. 6 shows an example of an overall system using a wearable device. Here, a system constructed at a manufacturing site of a manufacturing factory is described. A plurality of wearable devices 10, a plurality of optical sensors 62 shown in FIG. 3, one or more administrator terminals 104, a plurality of manufacturing devices 106, one or more cameras 114, and an information management server 116 are connected to a network 102. The network 102 may be, for example, a unit such as a factory building, a department, a building floor, or a company's sales office; it may be a network for each factory, building, or company; or it may be the Internet. If there are multiple manufacturing sites within a factory, the networks for each site in FIG. 6 may constitute a LAN, and the multiple LANs may be connected to a network for the entire factory. The network 102 may be a wireless network or a wired network.

[0048] Although there are many workers in a manufacturing site, not all workers need to wear the wearable device 10. Therefore, it is not necessary to prepare enough wearable devices 10 for all workers; instead, a predetermined number of wearable devices 10 are prepared, and the necessary workers wear the shared wearable device that is available. The system must identify the user when the user puts on the wearable device. This is necessary to display the work procedures of a specific manufacturing device to the user who uses that manufacturing device and to create a work report based on the user's behavior. There are various ways to identify the user, but the user may enter their own ID and device ID from a terminal (not shown) when putting on or taking off the wearable device 10. Input is not limited to key input, but may also be voice input from a microphone or scanning a barcode. Furthermore, since user behavior when putting on a device is likely to reflect user habits, the user's behavior at this time may be detected and estimated. Features indicating the user's behavior can be obtained from the acceleration and angular velocity of the wearable device 10, the movements of the user's face, hands, and fingers, and environmental sounds collected by the microphone. For example, the wearing status of the wearable device can be determined based on the friction sound between the temples and the skin or hair when the wearable device 10 is put on or taken off the face.

[0049] At least one manager who supervises the workers is assigned to the manufacturing site, and the manager uses a manager terminal 104. The manager terminal 104 may have the same configuration as the wearable terminal 10, but since the manager does not need to travel, the same configuration as a regular PC or tablet is sufficient, and a detailed description of the manager terminal 104 will be omitted.

[0050] Each of the manufacturing devices 106 is equipped with a device status sensor 108 and a user behavior sensor 110. These sensors 108 and 110 also have communication capabilities and are connected to the network 102.

[0051] The camera 114 constantly captures video of users in the manufacturing site. By analyzing these images, the user's behavior can be estimated. For example, a standard image for each user is stored in advance, and the user can be identified by comparing an image of the user putting on or taking off the wearable terminal 10 with the standard image. If it is not possible to install enough cameras to simultaneously cover the entire manufacturing site, it is sufficient to install several cameras 114 with variable angles that can capture a wide range of users with a single camera.

[0052] The information management server 116 includes a control unit 118, a communication unit 120, a location management unit 122, a user behavior management unit 124, a device status management unit 126, etc. The communication functions of the sensors 108 and 110, the communication function of the administrator terminal 104, the communication function of the camera 114, and the communication function of the communication unit 120 may be wired or wireless, similar to the communication function of the wearable terminal 10. In the case of wireless communication, short-range wireless communication such as Bluetooth, ZigBee, or UWB, medium-range wireless communication such as WiFi, or long-range wireless communication such as 3G / 4G or WiMAX may be used depending on the usage environment.

[0053] The location management unit 122 periodically collects information about the locations of the wearable terminal 10 and the administrator terminal 104 based on the outputs of the optical sensor 62 and various sensors of the wearable terminal 10 and the administrator terminal 104. Furthermore, the location management unit 122 identifies the user of the wearable terminal 10 or the administrator terminal 104, and manages the terminal ID, user ID, and location of the wearable terminal 10 or the administrator terminal 104.

[0054] The user behavior management unit 124 periodically collects information about the behavior and status of the user of the wearable terminal 10 based on the outputs of the optical sensor 62, various sensors of the wearable terminal 10, and the user behavior sensor 110 of the manufacturing equipment 106, and manages the terminal ID, user ID, and behavior and status of the wearable terminal 10. The equipment status management unit 126 periodically collects and manages information about the status of the manufacturing equipment based on the outputs of the equipment status sensor 108 of the manufacturing equipment 106. When the status of the equipment changes, the equipment status sensor 108 may notify the equipment status management unit 126, and information about the status of the manufacturing equipment may be collected.

[0055] When the information management server 116 detects an abnormality in an equipment in the equipment status management unit 126, it notifies the manager terminal 104 of the location information and status of the corresponding manufacturing equipment. At the same time, it judges the status of the workers, extracts candidate workers who can most efficiently deal with the equipment where the abnormality has occurred, and presents them to the manager terminal 104.

[0056] In this embodiment, a work checklist is automatically generated and presented to the user, and when the user performs a task, the corresponding item in the checklist is automatically checked. To this end, the information management server 116 integrates and processes information obtained from multiple sensors 108, 110 connected to the network 102 or various sensors of the terminals 10, 104, and automatically estimates / recognizes the actions of each worker. The information management server 116 then generates a work procedure (checklist) based on the results and supports automatic input (automatic entry) of the corresponding items in the checklist. When automatic input (automatic entry) of the final item in the work checklist is complete, the information management server 116 automatically generates a work report.

[0057] The contents of the work checklist vary depending on the manufacturing equipment requiring maintenance. Furthermore, the contents of the work checklist vary depending on the location of the malfunction within the manufacturing equipment. Therefore, the information management server 116 collects information related to the manufacturing equipment obtained from the equipment status sensor 108 associated with the manufacturing equipment requiring maintenance, and automatically estimates / recognizes the location of the malfunction in the manufacturing equipment. The information management server 116 then automatically identifies the wearable terminal 10 worn by the worker performing the maintenance, and displays the maintenance details in the form of a work checklist on the terminal 10.

[0058] 7 shows an example of the electrical configuration of the wearable terminal 10. The wearable terminal 10 includes a CPU 140, a system controller 142, a main memory 144, a storage device 146, a microphone 148, a speaker 54, a projection processing unit 150 (which controls the light source 28 and the display unit 30), a camera 59, a wireless communication device 152, a motion sensor 154, a gaze detection sensor 156, a gesture sensor 158, a touchpad 55, a vibrator 68, a position information receiver 159, a GPS module 155, and the like.

[0059] The CPU 140 is a processor that controls the operation of various modules within the wearable terminal 10 and executes computer programs loaded into the main memory 144 from a storage device 146, which is a non-volatile semiconductor memory such as an SSD or a flash array. These programs include an operating system (OS) and various application programs. The CPU 140 executes the various application programs and communicates with the information management server 116 via the network 102 using the wireless communication device 152, thereby performing various processes, such as: inputting voice using the microphone 148 and sending the voice data to the information management server 116; capturing an image using the camera 59 and sending the image data to the information management server 116; sending input data from the motion sensor 154, the gaze detection sensor 156, the gesture sensor 158, the touchpad 55, and the position information receiver 159 to the information management server 116; playing sounds from the stereo earphones or speaker 54 (not shown) connected to the earphone jack 54B; and vibrating the vibrator 68. The speaker 54 is assumed to be a monaural speaker, but if a stereo speaker is required, a speaker may also be provided on the temple on the left eye side, although this is not shown in FIGS.

[0060] The system controller 142 is a device that connects the local bus of the CPU 140 to various components. The microphone 148 is connected to the microphone jack 56 and collects the user's voice or environmental sounds. By recognizing the user's voice or analyzing the environmental sounds, the user's behavior can be estimated and the user can be identified. For example, a standard voice for each user can be stored in advance and the user can be identified by comparing the voice produced by the wearer with the standard voice. Furthermore, by analyzing the environmental sounds, the wearer's work location can be identified. The speaker 54 outputs an alarm or the like to alert the user. The projection processing unit 150 outputs an image signal to the display unit 30 and lights the light source 28 to project the image from the display unit 30 onto the screen 16. This image can include not only still images but also videos. The wireless communication device 152, for example, has a wireless LAN function and wirelessly connects the wearable device 10 to the access point 112.

[0061] The motion sensor 154 is a sensor that combines a triaxial acceleration sensor, a triaxial gyro sensor, and a triaxial geomagnetic sensor. It detects the head movement of the user using the wearable device 10 and, as a result, determines the direction in which the face is facing. The worker's state may be detected using the microphone 148, a barometer, etc. The worker's state includes not only whether the worker is moving or taking a break, but also the type of work and progress of the work. Using the movement detected by the motion sensor 154, altitude determined by atmospheric pressure, etc., it is possible to determine whether the feature values ​​obtained from these detection results match the feature values ​​of each process previously acquired from the worker, etc., and thereby determine which of multiple processes is being performed or completed. Furthermore, it is also possible to determine which of multiple processes is being performed or completed by determining whether the feature values ​​of environmental sounds input from the microphone 148 match the feature values ​​of environmental sounds characteristic of each work process previously acquired.

[0062] The gaze detection sensor 156 is located inside the center of the eyeglass frame, facing the user's face, and captures the user's eyeballs to detect gaze movement. Furthermore, the gaze detection sensor 156 may be capable of detecting the user's iris. The gesture sensor 158 is a sensor that identifies gestures based on finger movements. Specifically, it is a sensor that identifies the user's gestures by analyzing finger movements on the touchpad 55 provided on the projection device 12 and hand and finger movements in images captured by the camera 59. The vibrator 68 vibrates the temples of the wearable device 10 by vibrating the projection device 12, thereby conveying information to the user. The location information receiver 159 receives beacons containing location information transmitted by multiple location information transmitters 113 located within the area of ​​the LAN 102 using short-range wireless communication such as RF-ID. Because this is short-range wireless communication, the transmitter and receiver (wearable device) can be considered to be located at approximately the same position. The GPS module 155 detects the position (x, y, z) of the wearable device 10. By combining this detection result with the detection result of the position information receiver 159 and the detection result of the optical sensor 62 in Figure 3, the user's position and any changes therein can be detected more accurately.

[0063] The display unit 30 displays instructions and incoming calls from the manager terminal 104 and the information management server 116, and the working status of the worker detected by the motion sensor 154, etc. This display image is displayed on the screen 16 by the projection processing unit 150.

[0064] A microphone 148 and a speaker 54 allow voice communication with the outside world.

[0065] The administrator terminal 104 may have the same configuration as the wearable terminal 10, or may be a regular personal computer or tablet. The electrical configuration of a regular personal computer or tablet is equivalent to that of the wearable terminal 10 omitting the projection processing unit 150, camera 59, motion sensor 154, gaze detection sensor 156, gesture sensor 158, etc. The position of the administrator terminal 104 is detected by GPS.

[0066] An example of an equipment status sensor 108 attached to a manufacturing apparatus 106 will be described with reference to FIG. 8. FIG. 8(a) shows the attachment position relative to the apparatus, and FIG. 8(b) shows the configuration of the sensor 108. Conventionally, whenever a malfunction occurred in a manufacturing apparatus, an operator inspected the malfunctioning part of the manufacturing apparatus, repaired it, and investigated the cause of the malfunction. This extended the maintenance time (downtime of the manufacturing apparatus) of the manufacturing apparatus, resulting in a decrease in product productivity. In this embodiment, the information management server 116 collects and integrates information related to the equipment status obtained from the equipment status sensors 108 connected to the network 102, and automatically estimates or recognizes the malfunctioning part of the manufacturing apparatus. As a result, automatic diagnosis of the malfunctioning part in the manufacturing apparatus becomes possible, significantly shortening the maintenance time (downtime of the manufacturing apparatus) of the manufacturing apparatus and preventing a decrease in product productivity.

[0067] The equipment status sensor 108 is composed of an acceleration sensor 108a and a wireless communication device 108b, and an acceleration signal detected by the acceleration sensor 108a is transmitted to the information management server 116 via the wireless communication device 108b and the network 102. The equipment status sensor 108 is provided with an adhesive or fixing portion, allowing it to be easily attached to existing manufacturing equipment. An adhesive layer may be formed on the adhesive portion in advance, or an adhesive may be applied during bonding. Alternatively, the equipment status sensor 108 may be attached to existing manufacturing equipment by screwing the fixing portion to the manufacturing equipment.

[0068] Automatic diagnosis of defective parts of manufacturing equipment requires automatic collection of the operating status of each part of the manufacturing equipment. Therefore, purchasing or replacing new manufacturing equipment requires a huge amount of money. However, in this embodiment, it is only necessary to add and fix sensor terminals, which can be purchased very inexpensively, to each part of existing manufacturing equipment. This makes it possible to add an automatic diagnosis environment for defective parts at a very low cost while maintaining the existing facility environment.

[0069] As shown in Figure 8(a), the device status sensor 108 is fixed to, for example, a part of the moving belt 136, or a part of the movable arm 134 or movable shaft 132 that holds the article. If a place that should move under normal conditions becomes stationary, it can be determined that a malfunction has occurred in that moving part.

[0070] The control unit 118 in the information management server 116 stores in advance a maintenance procedure manual, which is a procedure manual for repair, maintenance, and inspection methods for each location where a problem occurs in various manufacturing equipment, and generates an optimal work checklist based on the results of the automatic diagnosis described above.

[0071] 8, an acceleration detection method was described as an example of the equipment status sensor 108. However, any physical or chemical quantity, such as temperature or current flow, may be used to extract defective parts. Furthermore, defective parts in manufacturing equipment may be automatically diagnosed by comparing images captured by a camera or environmental sounds collected by a microphone.

[0072] 8, when a manufacturing device with a malfunction is detected, the information management server 116 automatically selects a worker to perform maintenance on the manufacturing device, and displays the maintenance procedure or a work checklist derived therefrom on the worker terminal 103 worn by the worker. The information management server 116 selects, for example, a worker who (i) is located near the manufacturing device with the malfunction, (ii) can interrupt the work being performed, and (iii) can perform the maintenance work. This makes it possible to minimize the time lost by the worker when traveling.

[0073] In this embodiment, as shown in FIG. 9, an optical sensor 106a similar to the optical sensor 62 shown in FIG. 4 and a wireless communication device 106b are attached to a portion of the manufacturing equipment 106 to efficiently locate a worker who is near the manufacturing equipment where a malfunction has occurred. As described with reference to FIG. 5, the leaked light 50 emitted from the wearable device 10 contains the terminal ID information of the terminal 10. Therefore, when the information contained in the leaked light 50 detected by the optical sensor 106a is transmitted to the information management server 116 via the wireless communication device 106b and the network 102, the information management server 116 can identify which wearable device 10, i.e., which worker, is located near the manufacturing equipment where the malfunction has occurred. Based on this information, the information management server 116 selects a worker who will perform maintenance on the manufacturing equipment and transmits a work checklist to the wearable device 10 of the corresponding worker. As shown in FIG. 13(a), the work checklist is displayed on the screen 16 of the terminal 10. For the sake of convenience, the illustration is simplified in FIG. 13(a), but the actual work checklist is as follows:

[0074] Put your luggage in the cart Close the valve Turn off the on / off switch Turn off the first light switch Turn off the third light switch In this way, leak light 50 emitted from the wearable device 10 is detected / collected / aggregated in real time, making it possible to very easily and accurately identify workers who are near the manufacturing equipment 106 that is the target of maintenance. This saves the worker's travel time, shortens the maintenance period, and prevents a decline in manufacturing efficiency.

[0075] Another example of a method for recognizing a worker located near a target manufacturing device is to use a camera 114 that is installed in advance near the manufacturing device 106. An imaging device 114a in the camera 114 captures an image of the vicinity of the manufacturing device 106, and the image is transmitted to the information management server 116 via the wireless communication device 14b and the network 102. The information management server 116 may analyze the received image and automatically identify the worker appearing in the image.

[0076] An example of a worker performing work according to a work checklist will be described with reference to FIG. 10. When a work checklist like the one shown in FIG. 13(a) is displayed on the screen 16, the worker begins work. If the text is too small to read if the entire checklist is displayed on the screen 16 at once, only one or a few steps may be displayed, and the work progress may be automatically recognized as the work is completed. The displayed checklist may be updated in real time as each step is completed. A worker at the work site shown in FIG. 10(a) places a package 162 in a cart 164, closes a valve 170 (or rotates the handle 170 by a specified angle), turns off an on / off switch 172, and turns off a first light switch 176 and a third light switch 180, according to the work checklist. In this embodiment, the worker's actions are automatically recognized and identified in real time by a user behavior sensor 110 attached to the manufacturing equipment 106, and the work completion time is automatically entered into the work checklist (see FIG. 13(b)). When the final work step is completed, a work report is automatically generated in the information management server 116, and its contents are displayed on the manager terminal 104. The work report (Fig. 13(b)) is the work checklist (Fig. 13(a)) with the completion time entered.

[0077] Any detection technology or a combination thereof can be used as a method for automatically recognizing / identifying worker behavior using the user behavior sensor 110. For example, automatic recognition / identification of worker behavior may be performed by analyzing images of the worker's behavior captured by the camera 114 or 59. However, when analyzing images captured by the camera 114 for automatic recognition / identification of worker behavior, the worker's behavior may be hidden by shadows in the images depending on the situation. Alternatively, voice recognition technology may be used. By having the worker utter a specific voice each time they complete each item on the work checklist (maintenance work procedure) displayed on the wearable device 10, the microphone 148 can detect the specific voice input and automatically recognize / identify the worker's behavior. Alternatively, automatic recognition / identification of worker behavior may be performed by detecting environmental sounds generated when performing a specific task using the microphone 148 or a microphone built into the equipment status sensor 108. Another method for automatic recognition / identification of worker behavior is to recognize predetermined worker gestures. As a method for recognizing the worker's gestures, images of the worker's actions captured by cameras 59 and 114 may be analyzed, or the results of detecting leakage light 50 emitted from the wearable terminal 10 using optical sensors 62 installed at multiple locations or optical sensor 106a attached to the manufacturing equipment 106 may be compared.

[0078] A pair of light-emitting unit 166a and light-receiving unit 166b is installed at the opening of cart 164, and detects light blocking when luggage 162 passes through the opening of cart 164, automatically detecting whether luggage 162 is being placed in or removed from cart 164. Figure 10(b) shows the signal characteristics detected by light-receiving unit 166b when luggage is being placed in or removed from cart 164. The vertical axis represents the amount of light detected by the light-receiving unit, and the horizontal axis represents the passage of time. While luggage 162 passes through the opening of cart 164, the amount of light detected by the light-receiving unit decreases. The method of detecting whether luggage 162 is being placed in or removed from cart 164 is not limited to the above-mentioned method using light, and any other method can be used.

[0079] This section describes an example of automatic real-time recognition / identification of actions other than loading and unloading cargo, such as "closing a valve," "turning off an on / off switch," and "turning off a light switch." Generally, maintenance (maintenance, inspection, and repair) of manufacturing equipment requires a worker to directly contact a designated location within the equipment. Taking advantage of this feature, this embodiment detects the worker's contact with a designated location within the equipment and reflects this in the automatic recognition / identification of the worker's actions. This method is very simple to detect and ensures high accuracy of automatic recognition / identification. In the example shown in FIG. 10( a), a contact sensor 168 is attached to a valve 170, and transparent contact sensors are also attached to an on / off switch 172 and a light switch board 174. The light switch board 174 includes first, second, and third light switches 176, 178, and 180.

[0080] A contact sensor, which is an example of the user behavior sensor 110, includes a wireless communication function (e.g., near-field wireless communication) and a function for detecting the contact status of a worker. Any contact-detectable element, such as a piezoelectric element, a photointerrupter, or an acceleration sensor (gyro sensor), can be used to detect the contact status. This contact sensor can be attached to existing manufacturing equipment and other facilities, and is very inexpensive. Therefore, by attaching this contact sensor (user behavior sensor 110) to existing manufacturing equipment, a near-field wireless communication network environment can be added at very low cost while maintaining the existing infrastructure.

[0081] Examples of the user behavior sensor 110 are shown in Figures 11 and 12. Figure 11 shows a user behavior sensor 110 attached to an on / off switch 172, which is an existing infrastructure, or a user behavior sensor 110 (contact sensor 168) attached to a valve 170, and Figure 12 shows a user behavior sensor 110 attached to a lighting switch board 174, which is an existing infrastructure.

[0082] As shown in FIG. 11 , the user behavior sensor 110 includes an adhesive layer 202 at the bottom, on which a control and communication circuit 204 and a solar cell 206 are sequentially formed. A transparent conductive layer 208, a transparent intermediate layer 210, a transparent conductive layer 212, and a transparent uneven layer 214 are sequentially layered on the solar cell 206. The control and communication circuit 204 has a wireless communication function (near-field wireless communication) and a worker's contact detection function. The solar cell 206 is used as the driving power source for both functions. If a battery were used as the driving power source, it would require time and effort to replace the battery. Furthermore, if an external power source connected by an electric wire were used as the driving power source, the electric wire would interfere with the worker's contact. However, the solar cell 206 eliminates the time and effort required for battery replacement, enabling the user behavior sensor 110 to be used for a long period of time without interfering with the worker's contact.

[0083] By stacking the control and communication circuit 204 that performs near field communication and control under the solar cell 206, the power generation efficiency of the solar cell 206 is increased and the planar size of the user behavior sensor 110 is reduced.

[0084] To use the solar cell 206, ambient light must be able to irradiate it. On the other hand, it is desirable to place the portion that detects user contact on the surface of the user activity sensor 110. To simultaneously satisfy both requirements, we made the contact detection portion transparent and adopted a capacitance-based detection method. To detect the worker's contact or pressure using changes in capacitance, a structure can be adopted in which a transparent and flexible transparent intermediate layer 210 (e.g., a sheet made of a transparent organic material) is sandwiched between two transparent conductive layers 208 and 212 (e.g., transparent organic material sheets). An AC voltage 216 is applied between the two transparent conductive layers 208 and 212 to resonate the transparent conductive layers 208 and 212. When the worker touches the surface of the user activity sensor 110, the capacitance changes, which changes the AC resonance state. The worker's contact can be detected by detecting this change in the AC resonance state. However, the detection method is not limited to the capacitance type, and any element that can detect contact or pressure and allows at least a portion of the ambient light to be irradiated onto the solar cell 206 within the user behavior sensor 110 may be used.

[0085] As an example, minute irregularities are provided on the transparent layer on the surface of the user behavior sensor 110. This is to prevent the surface from slipping, but if Braille information is recorded on these irregularities, it will be easier for visually impaired people to use.

[0086] The user behavior sensor 110 can be fixed to a part of an existing manufacturing device using any fixing method, such as screws, but direct bonding or adhesion / attachment can save space. This bonding or adhesion / attachment method is not limited to direct bonding using an adhesive, and can also be a method using an adhesive sheet or adhesive tape. For the on / off switch 172 and the light switch board 174, an adhesive layer 202 with double-sided tape properties can be used, and for the bulb 170, an adhesive layer 202 made of a transparent adhesive tape layer can be used.

[0087] FIG. 12 shows a user behavior sensor 110 attached to a lighting switch board 174. In the lighting switch board 174, the first, second, and third light switches 176, 178, and 180 have characters such as "Light 1," "Light 2," and "Light 3" written on their surfaces. Therefore, it is desirable that these characters remain visible even when the user behavior sensor 110 is attached. Therefore, it is desirable that the tops of the light switches 176, 178, and 180 are transparent. Furthermore, it is necessary to detect the contact status of each of the multiple light switches 176, 178, and 180 independently. Meanwhile, the lighting switch board 174 has a surplus space 182 where the light switches 176, 178, and 180 are not installed. To accommodate this situation, the user behavior sensor 110 shown in FIG. 12 includes a transparent sheet 208, a transparent intermediate layer 210, a transparent sheet 212, and a transparent uneven layer 214 sequentially stacked on an adhesive layer 202. Transparent sheets 208 and 212 correspond to the transparent conductive layers 208 and 212 in FIG. 11 . Transparent sheet 208 includes three transparent conductive regions 208a, 208b, and 208c, and transparent sheet 210 includes three transparent conductive regions 210a, 210b, and 210c. Transparent conductive regions 208a and 210a are located at the position of first light switch 176, transparent conductive regions 208b and 210b are located at the position of second light switch 178, and transparent conductive regions 208c and 210c are located at the position of third light switch 180. AC voltage 216 is applied between transparent sheets 208 and 212. By dividing the transparent sheet into three regions corresponding to the three light switches in this way, the contact status of each of the three light switches can be detected independently. Braille information can also be formed on transparent uneven layer 214 on the surface.

[0088] A control circuit 204a and a communication circuit 204b are formed on the transparent uneven layer 214 in the spare space 182 where no light switch is installed, and a solar cell 206 is formed on top of them. Because the solar cell 206 is located at the top, power generation efficiency is high. In addition, because the control circuit 204a, the communication circuit 204b, and the solar cell 206 are positioned vertically, the planar size of the user behavior sensor 110 is reduced.

[0089] According to the embodiment, the status of the wearable device and the manufacturing equipment is detected, and based on the detection results, work procedures are displayed on the wearable device of a worker who is able to perform the work near the manufacturing equipment that requires maintenance, inspection, repair, etc., thereby providing meaningful information to the worker. Furthermore, based on the detection results of the wearable device and the status of the manufacturing equipment, the completion of each step of a series of work is determined and a work report recording the completion of the work is automatically created, thereby significantly reducing the worker's effort. Furthermore, by attaching a contact sensor to the manufacturing equipment to detect the completion of the work, it is possible to accurately estimate / recognize the worker's behavior in a very simple and inexpensive manner without modifying existing manufacturing equipment.

[0090] In the above description, maintenance of a manufacturing device is used as an example. However, this embodiment is not limited to this, and the user's behavior corresponding to another purpose may be monitored and the corresponding work content may be displayed. Furthermore, as a means for monitoring the user's behavior, a contact sensor is provided at a location where the user may come into contact, but this is not limiting, and other sensors may also be used.

[0091] Although an eyeglass-type wearable terminal has been described as an embodiment, the present invention can also be applied to other head-mounted types such as goggle-type and helmet-type, as well as wristband-type and pendant-type terminals. For example, if the terminal is a helmet-type or goggle-type, the projection device 12 and camera 59 can be attached to the helmet or goggles, and normal eyeglass wearers can also use it. Furthermore, if the terminal is a helmet-type, the speaker 54 can be attached inside the helmet, allowing for clearer sound to be heard, and the microphone can be attached to the helmet and its position can be adjusted, improving the microphone's sound collection capability.

[0092] The types of sensors that detect the state of the manufacturing equipment and the wearable terminal are not limited to those described above, and various sensors can be used as appropriate.

[0093] The present invention can be applied to wearable devices other than head-mounted devices, as well as to electronic devices that are not wearable but are small, lightweight, portable, and always with the user, such as notebook computers, tablet computers, and smartphones.

[0094] The division of functions between the wearable terminal and the information management server is not limited to the above description, and some of the functions described as the functions of the wearable terminal may be realized as functions of the information management server, or some of the functions described as the functions of the information management server may be realized as functions of the wearable terminal.

[0095] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0096] 10...wearable terminal, 12...projection device, 16...screen, 54...speaker, 55...patch pad, 59...camera, 62...optical sensor, 104...administrator terminal, 106...manufacturing equipment, 108...equipment status sensor, 110...user behavior sensor, 114...camera, 116...information management server, 148...microphone, 152...wireless communication device.

Claims

1. a display for displaying a display image; a projection device that projects the display image in front of a user to form a virtual image corresponding to the display image in front of the user; A camera that captures an image of the user's line of sight; a gesture sensor that analyzes the movement of the user's hands and fingers in the image captured by the camera and determines the gesture of the user; A wearable terminal comprising: the projection device includes a lens that projects divergent light emitted from the display in front of the user; and a first user operation unit; the first user operation unit is capable of adjusting a projection angle of the projection device by a rotation operation; The user can adjust the display position of the virtual image to match the shape or size of the user's head by rotating the first user operation unit while viewing the virtual image, The virtual image is changed in response to a gesture of the user determined by the gesture sensor. Wearable device.

2. A server for storing images; a wearable terminal including: a display connected to the server and displaying a display image based on an image transmitted from the server; a projection device that projects the display image in front of a user to form a virtual image corresponding to the display image in front of the user; a camera that captures an image in front of the user's line of sight; and a gesture sensor that analyzes the movement of the user's hands and fingers in the image captured by the camera and identifies a gesture of the user; A system comprising: the projection device includes a lens that projects divergent light emitted from the display in front of the user; and a first user operation unit; the first user operation unit is capable of adjusting a projection angle of the projection device by a rotation operation; The user can adjust the display position of the virtual image to match the shape or size of the user's head by rotating the first user operation unit while viewing the virtual image, The virtual image is changed in response to a gesture of the user determined by the gesture sensor. system.

3. A display method for a wearable terminal including: a display that displays a display image; a projection device that projects the display image in front of a user to form a virtual image corresponding to the display image in front of the user; a camera that captures an image in front of the user's line of sight; and a gesture sensor that analyzes the movement of the user's hands and fingers in the image captured by the camera and identifies a gesture of the user, the projection device includes a lens that projects divergent light emitted from the display in front of the user; and a first user operation unit; the first user operation unit is capable of adjusting a projection angle of the projection device by a rotation operation; The user can adjust the display position of the virtual image to match the shape or size of the user's head by rotating the first user operation unit while viewing the virtual image, The virtual image is changed in response to a gesture of the user determined by the gesture sensor. Display method.

Images