Wearable devices, systems, and display methods

Abstract

To provide a wearable terminal, a system, and a display method that can easily adjust the display position of an image.SOLUTION: According to an embodiment, a wearable terminal includes: a projection device that projects a display image in front of a user to form a virtual image corresponding to the display image in front of the user; a camera that photographs an image the user is looking at; and a gesture sensor that analyzes the motion of the user's hands and fingers in the image photographed by the camera to discriminate the user's gesture. The virtual image is changed according to the user's gesture detected by the gesture sensor.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to wearable terminals, systems, and display methods.

Background Art

[0002] In a manufacturing site or factory having a large number of manufacturing devices, the operating rate of the manufacturing devices is a major factor affecting the production volume of products. Due to the lack of regular maintenance and inspection, the failure of manufacturing devices that could have been avoided originally or the inefficiency in dealing with sudden malfunctions in manufacturing devices may cause the manufacturing devices to be inoperable for a long time, leading to a decrease in the operating rate and the production volume of products. Therefore, it is required to shorten the downtime of manufacturing devices as much as possible. Since the maintenance, inspection, and repair operations are different for each manufacturing device, workers may refer to a manual or checklist (hereinafter collectively referred to as a checklist) that describes the work procedures at each stage of the work. In recent years, wearable terminals have been widely used in manufacturing sites. For example, an operator wears a glasses-type wearable terminal at the manufacturing site, and the checklist is displayed on the lens surface. As a result, the operator does not need to refer to a paper checklist during the work and can efficiently perform unfamiliar or complex work without taking a break.

[0003] However, since it is necessary to check the checklist to grasp the certainty of the completion of the work at each stage, the reference is electronically performed on the screen. Still, a paper checklist is prepared, and when the work at each stage is completed, the operator takes a break from the work and fills it in. For this reason, the manufacturing device is forced to stop, leading to a decrease in the production volume. Furthermore, after returning to the office, the operator creates a work report based on this checklist. Creating this work report is cumbersome for the operator.

[0004] Systems are being developed that use head-mounted displays with cameras to assist operators in their work. One example is a medical device management system that assists operators handling used and contaminated medical devices such as endoscopes, scalpels, forceps, and other instruments.

[0005] This system comprises a head-mounted camera that captures the field of view of an operator handling medical equipment or instruments; a storage means that stores images of the operator's field of view captured during exemplary work when handling medical equipment or instruments as standard images; a first determination means that compares the image captured by the camera with the standard image read from the storage means and determines whether a predetermined task is being performed in a standard manner based on the similarity between the two images; an information output means that outputs information indicating a warning or instruction based on the determination result by the determination means; and an output means that gives a warning or instruction to the operator based on the information indicating a warning or instruction. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2009-279193 [Overview of the project] [Problems that the invention aims to solve]

[0007] This system automatically recognizes operator actions by comparing images of medical devices being handled, captured by a camera, with pre-prepared standard images. However, this method has low recognition accuracy with simple pattern matching between images, requiring complex image processing such as feature extraction. As a result, the automatic recognition process becomes highly complex and takes a considerable amount of time.

[0008] The object of the present invention is to provide a wearable terminal, system, and display method that allows for easy adjustment of the image display position. [Means for solving the problem]

[0009] According to one embodiment, the wearable terminal comprises a display unit that displays an image, a projection device used to project the display image in front of the user and to form a virtual image corresponding to the display image in front of the user, and a camera that captures an image in the user's line of sight. The projection device includes a lens that projects divergent light obtained from the display unit in front of the user, and a user operation unit including a slide switch or a rotary switch. The camera is installed at a first position on the outer surface of the projection device. The user operation unit is installed at a second position on the outer surface of the projection device. The projection angle of the virtual image is adjustable based on the user's operation of the user operation unit. The user can adjust the projection angle by blind-touching the user operation unit while visually observing the virtual image, and by adjusting the projection angle, the display position of the virtual image can be adjusted to match the shape or size of the user's head. When the user's actions are recognized based on the movements of the user's hands and fingers in the image captured by the camera, the virtual image is changed to reflect the recognized user actions. The above content is a list showing the work procedures to be performed by the user. When the completion of a task is recognized as an action by the user, the list is updated, and the updated list represents the completion of the task. [Brief explanation of the drawing]

[0010] [Figure 1] This is a perspective view showing an example of a wearable device according to the embodiment. [Figure 2] This figure shows a front view and a cross-sectional view of an example of a wearable device. [Figure 3] This figure shows an example of location detection for a wearable device. [Figure 4] This diagram shows the principle of location detection for wearable devices. [Figure 5] This figure shows the signal waveform during location detection of a wearable device. [Figure 6] This figure shows an example of a system that includes a wearable device and an information management server. [Figure 7] This is a block diagram showing the electrical configuration of a wearable device. [Figure 8]This figure shows an example of a sensor that detects the status of a device. [Figure 9] This figure shows an example of a sensor that detects user behavior. [Figure 10] This is a schematic diagram showing an example of the system's operating environment. [Figure 11] This is an exploded perspective view showing the structure of an example of a sensor used in a system to detect user behavior. [Figure 12] This is an exploded perspective view showing the structure of another example of a sensor used in a system to detect user behavior. [Figure 13] This diagram shows an example of the work procedure displayed by the system and the work record created by the system. [Modes for carrying out the invention]

[0011] The embodiments will be described below with reference to the drawings.

[0012] Wearable devices come in various forms, including head-mounted (including glasses, goggles, helmets, etc., although these are sometimes collectively referred to as glasses), wristbands, and pendants. Here, we will describe an embodiment of a glasses-type wearable device. Glasses-type wearable devices include types that allow the user to see the scenery in front of their eyes through transparent lenses, and types called head-mounted displays that obstruct the view and prevent the user from seeing the scenery. Here, we will describe the type that allows the user to see the scenery as an example.

[0013] Figure 1 is a perspective view of a glasses-type wearable device (hereinafter simply referred to as a wearable device) 10, Figure 2(a) is a front view, and Figure 2(b) is a diagram showing the cross-sectional structure viewed from above.

[0014] The wearable terminal 10 has almost the same shape as ordinary glasses. On the other hand, here, the projection device 12 is attached to the outside of the temple on the right-eye side. Glasses 14 and 16 are fitted into the frame. The glass 14 on the left-eye side is ordinary transparent glass so that the user can see the scenery. At least a part of the glass 16 on the right-eye side is the screen 16. The screen 16 enables the user to view the image projected by the projection device 12. The screen 16 is transparent when the projection device 12 is not projecting an image, and the user can see the scenery through the glass (screen) 16 on the right-eye side.

[0015] The projection device 12 includes a power supply unit 22 and a control unit 24 as electronic components. The power supply unit 22 can include a button-type battery, a rechargeable battery, a secondary battery capable of non-contact power supply, etc. Alternatively, the power supply may not be built-in, and power may be supplied to the projection device 12 from an external power source via a power line. The control unit 24 communicates with a server or other electronic devices via a network described later and transmits and receives information. This communication may be either wired or wireless. In the case of wireless, depending on the usage environment, any of short-range wireless communications such as Bluetooth (registered trademark), ZigBee (registered trademark), UWB, medium-range wireless communications such as WiFi (registered trademark), or long-range wireless communications such as 3G / 4G, WiMAX (registered trademark) may be used.

[0016] The projection device 12 further includes, as optical components, a light source 28, a display unit 30, a prism 32, a lens group 34, etc. The light source 28 may be a dimming type white LED light source including a plurality of, for example, three LEDs (Light Emitting Diodes) with different emission colors from each other and capable of independently changing the output light amount of each. According to the dimming type white LED light source, even when the usage environment of the wearable terminal 10 is, for example, in a clean room where orange-based lighting is often used, the emission color can be changed according to the usage environment, and a clear projected image can be obtained. Further, according to the dimming type white LED light source, a display color that is easy for the user to see can be output, and compared with the case of outputting a display color that is difficult for the user to see, it is possible to avoid the occurrence of factors that are troublesome for the user, such as eye fatigue and accompanying migraines.

[0017] The display unit 30 is, for example, a reflective LCD (Liquid Crystal Display) module, and displays a predetermined text, image, etc. (hereinafter, what is displayed on the display unit 30 may also be collectively referred to as a display image) based on display control by the control unit 24. The non-parallel light (divergent light, hereinafter may also be referred to as divergent light) emitted from the light source 28 is reflected by the half mirror surface 32a of the prism 32 and illuminates the display image of the display unit 30. The reflected light of the display unit 30 passes through the half mirror surface 32a as light corresponding to the display image (sometimes referred to as image light), is emitted from the emission surface 32c, and is projected onto the screen 16 as a projected image of a predetermined size through the lens group 34.

[0018] The screen 16 has a front transparent refractive body 42, a Fresnel lens-shaped half mirror surface 44, and a rear transparent refractive body 46. A part of the image light reaching the Fresnel lens-shaped half mirror surface 44 is reflected by the Fresnel lens-shaped half mirror surface 44 to form a virtual image (projected image) corresponding to the display image of the display unit 30 several meters ahead. Note that the screen 16 can also partially transmit the scenery in front of the line of sight of the user wearing the wearable terminal 10, and the screen 16 may be configured to display a scenery visible to the user together with the projected image.

[0019] A portion of the image light (divergent light) emitted from the light source 28 and passing through the half-mirror surface 32a is totally reflected by the totally reflective surface 32b and refracted by the emission surface 32c to become leak light 50, which is divergent light from the light source 28. The leak light 50 is emitted in a direction different from the screen 16 through an opening or gap (guidance section) 52 formed on the front of the projection device 12.

[0020] The wearable terminal 10 has a speaker 54A, an earphone jack 54B, a microphone jack 56, a slide switch 57, and a rotary switch 58, etc., at a predetermined location on the projection device 12, for example, on the bottom surface. A hands-free microphone (not shown) is connected to the microphone jack 56 to collect the user's voice. The slide switch 57 can adjust, for example, the brightness and color tone of the projected image of the projection device 12. The rotary switch 58 can adjust, for example, the projection angle of the projected image. By allowing different adjustment amounts to be set by different operations, such as the slide switch 57 and the rotary switch 58, the user can adjust the projected image by touch while visually observing the projected image. For example, by operating the slide switch 57, a projected image with display brightness and color tone that suits the user's preference can be provided. By operating the rotary switch 58, the projection angle can be adjusted to display the image in the optimal position according to the shape and size of the user's head. It goes without saying that the objects to be adjusted by the slide switch 57 and the rotary switch 58 may be reversed, the positions of the slide switch 57 and the rotary switch 58 may be reversed, and both may be assigned to two types of operations of a single operating member.

[0021] The selections made using these switches 57 and 58 can be made by trial and error while only looking at the projected image, but to improve the efficiency of the adjustment, the menu screen may be projected and the items selected on that screen may be used for adjustment. When the display unit 30 displays the menu screen, the menu screen is projected onto the screen 16.

[0022] Furthermore, menu items can be selected not only by operating switches 57 and 58, but also by touch operation. For this reason, a touchpad 55 is also provided on the outside of the projection device 12. The display unit 30 displays menus, etc., and the user can easily and efficiently input operations by touching the corresponding position on the touchpad 55 that corresponds to the display position of the items in the menu.

[0023] A camera 59 is located on the outside of the central front section, capable of capturing images (still images or videos) in the user's line of sight. Although not shown in the diagram, a camera may also be placed on the inside of the central front section (corresponding to the position of camera 59) facing the user's face to capture the user's eyeballs and detect the user's iris. The iris can be used for user authentication.

[0024] By utilizing the light leakage 50 from the wearable device 10, the state of the wearable device 10, i.e., the user's state, can be detected. The principle of detecting the state of the wearable device will be explained with reference to Figures 3, 4, and 5. Here, the state includes position and movement of position, etc.

[0025] Figure 3 shows an example of how wearable devices can be used. For example, any number of workspaces or product shelves A01~Axy (where x and y are both positive integers), B01~Bxy, and C01~Cxy are arranged within a work area 60 such as a factory parts yard, a mail-order company's product warehouse, or a retail company's delivery department. The workspaces or product shelves may be, for example, factory work tables, manufacturing equipment on a production line, school desks, or seating positions in a conference room.

[0026] At least one optical sensor 62-1 to 62-n (where n is a positive integer) is placed in the work area 60. The optical sensors 62-1 to 62-n can individually detect the position (x, y, z), number, changes in position (movement), and changes in orientation of wearable terminals 10-1 to 10-m (where m is a positive integer) using the detection methods shown in Figures 4 and 5. By detecting the position, number, movement, and changes in orientation of the wearable terminals 10-1 to 10-m, the position, movement, and other status of any number of users wearing the wearable terminals 10-1 to 10-m can be recognized.

[0027] The user can move freely within the work area 60. The user performs predetermined tasks in predetermined locations, such as a work space 64 which is a station (cart), a similar storage device, or a movable table. The work space 64 may not be movable and may be a fixed desk or seating position.

[0028] As shown in Figures 3 and 4, the detection system includes one or more wearable terminals 10 and one or more optical sensors 62. The optical sensors 62 have a function to detect leaked light 50 and a communication function to transmit the detection result to a server or the like. This communication function, like the communication function of the wearable terminal 10, may be either wired or wireless. In the case of wireless, depending on the usage environment, any of the following may be used: short-range wireless communication such as Bluetooth, ZigBee, UWB, medium-range wireless communication such as WiFi, or long-range wireless communication such as 3G / 4G, WiMAX. The embodiments described below have various units and modules with communication functions, and similarly, the communication functions of these units and modules may also be either wired or wireless. In the case of wireless, depending on the usage environment, any of the following may be used: short-range wireless communication such as Bluetooth, ZigBee, UWB, medium-range wireless communication such as WiFi, or long-range wireless communication such as 3G / 4G, WiMAX.

[0029] To enable the wearable terminal 10 to be identified from the leaked light 50 received by the light sensor 62, the wearable terminal 10 intermittently modulates the leaked light 50 using information including terminal identification information (Identification, sometimes referred to as terminal ID). A typical example of a modulation method is the chopper type modulation method, which reduces the amount of light emitted to zero, but here, a modulation method is adopted that can ensure an amount of light emitted above a predetermined amount even when the amount of light emitted is low. This reduces the strain on the user's eyes. As a modulation method, for example, DSV (Digital If a Sum Value-free modulation scheme is adopted (i.e., a modulation scheme that constantly calculates the DSV of the modulation signal and allows the insertion of bit-inverted codes as appropriate to make the DC component zero), changes in the amount of light emitted over a relatively long range are suppressed, and the amount of light emitted can always be kept at zero macroscopically, further reducing the burden on the user's eyes. Since the human eye can perceive changes of up to about 0.02 seconds, setting the reference frequency of the above modulation to 10 Hz or higher, for example 20 Hz or higher, and more preferably 60 Hz or higher, also reduces the burden on the user's eyes. On the other hand, since the LED used in the light source 28 has internal impedance and connection capacitance, a modulation frequency that is accurate is preferably less than 100 MHz, and more preferably 10 MHz or lower. Therefore, the modulation frequency of the light source 28 used in the detection system of the embodiment is preferably in the range of 10 Hz to 100 MHz, and more preferably 10 Hz to 10 MHz.

[0030] Since the light leakage 50 from the light source 28 is utilized, the amount of light detected by the light sensor 62 changes according to the distance between the wearable terminal 10 and the light sensor 62. By utilizing this phenomenon, the distance between the wearable terminal 10 and the light sensor 62 or the orientation of the wearable terminal 10 relative to the light sensor 62 can be determined. Since the position (including height) of the light sensor 62 is fixed, if the distance between the light sensor 62 and the wearable terminal 10 is known, the position (x, y, z) of the wearable terminal 10 can be detected.

[0031] Furthermore, since the system utilizes the stray light 50 emitted from the light source 28, the stray light 50 can be detected over a relatively wide area. As a result, by installing only a relatively small number of optical sensors 62-1 to 62-n, the positions of wearable terminals 10-1 to 10-m within the work area 60, the distance between the wearable terminal 10 and the optical sensor 62, the direction of the wearable terminals 10-1 to 10-m, or the orientation of the wearable terminal 10 relative to the optical sensor 62 can be detected. This reduces the equipment costs required to install the detection system.

[0032] The light intensity information of the leaked light 50 detected by the light sensor 62 is transmitted from the light sensor 62 to a server described later at a predetermined timing. The server analyzes the information collected from the light sensor 62. This makes it possible to detect the location and status of any wearable device 10-1 to 10-m, i.e., the user.

[0033] Figure 4 is a schematic diagram illustrating a specific use case of a system for recognizing wearable terminals according to an embodiment. It assumes a situation where three users, each wearing wearable terminals 10-1 to 10-3, are positioned around four optical sensors 62-1 to 60-4. Leaked light 50 from wearable terminals 10-1 and 10-2 is detected by optical sensors 62-1 to 60-4. Each of the optical sensors 62-1 to 60-4 performs an analog-to-digital (AD) conversion of the detected amount of leaked light 50 and transmits the corresponding light intensity information to a server at a predetermined timing, for example, via short-range wireless communication.

[0034] Suppose that the wearable terminal 10-1 moves towards the light sensor 62-1 in response to the user's movement, while the orientation of the wearable terminal 10-2 temporarily changes in response to an arbitrary action by the user, such as head movement (turning the head). Figure 5 shows the changes in detection information at this time.

[0035] Figure 5 shows an example where an intermittent time-varying modulation scheme is used for the leakage light 50 of each wearable terminal 10-1 to 10-3. In other words, the ID modulation period is staggered for each of the wearable terminals 10-1 to 10-3.

[0036] As shown in Figures 5(a), (b), and (c), for the first to third wearable terminals 10-1 to 10-3, an intermittent ID modulation period is set for the wearable terminal, and the rest of the time is an unmodulated period. Within each ID modulation period, the synchronization signal SYNC and the terminal IDs of the wearable terminals 10-1 to 10-3 form a pair (corresponding one-to-one), and this pair is repeated multiple times (a multiple of 4 times in the case of 4 sensors as shown in Figure 5).

[0037] As soon as the first wearable device 10-1 enters its unmodulated period, the ID modulation period of the second wearable device 10-2 begins. Similarly, as soon as the second wearable device 10-2 enters its unmodulated period, the ID modulation period of the third wearable device 10-3 begins.

[0038] During the ID modulation period of the second wearable terminal 10-2 and the ID modulation period of the third wearable terminal 10-3, the synchronization signal SYNC and the terminal IDs of wearable terminals 10-2 and 10-3 are repeatedly modulated. In this way, by embedding the terminal ID of wearable terminal 10 into the modulated signal, the terminal ID can be detected.

[0039] In the example above, the modulation timing of each wearable device 10-1 to 10-3 is time-division (intermittent). However, for example, all wearable devices 10-1 to 10-3 may be continuously modulated, and the modulation reference frequencies of each wearable device 10-1 to 10-3 may be changed. Furthermore, the frequency spectral characteristics of each device during spread spectrum may also be changed.

[0040] As shown in Figures 5(d), (e), (f), and (g), the information communication period from optical sensors 62-1 to 62-4 is finely divided into each ID modulation period.

[0041] As shown in Figure 4, initially, some of the leaked light from the wearable device 10-1 reaches the light sensor 62-4. Therefore, initially, as shown in Figure 5(k), the light sensor 62-4 detects the leaked light from the wearable device 10-1. However, as the wearable device 10-1 moves toward the light sensor 62-1, the amplitude of the modulated signal of the leaked light from the wearable device 10-1 detected by the light sensor 62-4 decreases. On the other hand, as shown in Figure 5(h), the amplitude of the modulated signal of the leaked light from the wearable device 10-1 detected by the light sensor 62-1 increases over time. In this way, by comparing the time changes in the modulated signal amplitudes detected by the light sensors 62-1 to 62-n, the time change (movement state) of the wearable devices 10-1 to 10-m that are the target of detection can be detected.

[0042] On the other hand, initially, since the wearable device 10-2 is facing the light sensor 62-3, the amplitude of the modulated signal obtained from the leaked light is greater at the light sensor 62-3 than at the light sensor 62-2. After this, suppose, for example, a second user turns their head and temporarily faces the light sensor 62-2. Then, as shown in Figure 5(i), the detection output of the wearable device 10-2 output by the light sensor 62-2 temporarily increases and then decreases. On the other hand, as shown in Figure 5(j), the detection output of the wearable device 10-2 output by the light sensor 62-3 temporarily decreases and then increases.

[0043] In this way, by comparing the time changes in the modulation signal amplitude detected by the optical sensor 62, the time changes in the orientation of the wearable terminal 10-1 to 10-m, which is the target of detection, can also be estimated.

[0044] The above detection example is an example where the user's movement is movement or head shaking. However, it is not limited to these, and various other user actions may be used. For example, the user's hand movements or torso twisting may temporarily block the leaked light. In this case, a temporary decrease in the modulation signal amplitude occurs in all light sensors 62-1 to 60-4 at the same time. By comparing the relationships of the changes in the modulation signal amplitude of all light sensors 62-1 to 60-4 in this way, different user behavior patterns can be identified.

[0045] By utilizing the methods described above, it becomes possible not only to detect user behavior but also to recognize user intent.

[0046] Furthermore, beacons may be used as a method for detecting the position (x, y, z) of the wearable terminal 10. In the example described above, a large number of wearable terminals 10 emitted modulated light from a large number of light sensors 60, and the large number of light sensors 60 compared and processed the received information to detect the position and state of the wearable terminals 10. However, if a large number of location information transmitters are placed in the work area 60, and beacons corresponding to their placement are transmitted from the transmitters using short-range wireless communication such as RF-ID with a range of several meters, the wearable terminal 10 that receives the beacon can be considered to be in approximately the same position as the transmitter. Moreover, the position of the wearable terminal can also be detected using GPS. Position detection does not need to be based on only one method; detection accuracy can be improved by using multiple methods in combination.

[0047] Figure 6 shows an example of an entire system using wearable devices. Here, we describe a system built on the manufacturing floor of a factory. Multiple wearable devices 10, multiple optical sensors 62 as shown in Figure 3, one or more administrator terminals 104, multiple manufacturing devices 106, one or more cameras 114, and an information management server 116 are connected to a network 102. The network 102 may be, for example, a factory building, department, floor of a building, or a company's sales office, or it may be a network for each factory, building, or company, or it may be the internet. If there are multiple manufacturing sites within the factory, the networks for each site in Figure 6 may form a LAN, and multiple LANs may be connected to the factory-wide network. The network 102 may be a wireless network or a wired network.

[0048] Although there are many workers on the manufacturing floor, not all workers need to wear a wearable device 10. Therefore, it is not necessary to provide a wearable device 10 for every worker; only a predetermined number needs to be provided, and workers who need one can wear an available shared wearable device. The system needs to identify the user when they put on the wearable device. This is because it is necessary to display the work procedures for a specific manufacturing device to the user using that device, and to create work reports based on the user's actions. There are various methods of identification, but the user may enter their own ID and device ID from a device (not shown) when putting on or taking off the wearable device 10. Input is not limited to keyboard input; it can also be voice input from a microphone or scanning input using a barcode. Furthermore, since the user's habits tend to be revealed when putting on the device, the user's actions at this time may be detected and estimated. Feature quantities indicating the user's actions can be obtained from the acceleration and angular velocity of the wearable device 10, the movements of the user's face, hands, and fingers, and ambient sounds picked up by the microphone. For example, the wearing status of the wearable device 10 can be determined based on the friction sound between the temples and the skin or hair when the wearable device 10 is placed on or removed from the face.

[0049] At least one supervisor is assigned to the manufacturing site to oversee the workers, and the supervisor uses the supervisor terminal 104. The supervisor terminal 104 may have the same configuration as the wearable terminal 10, but since the supervisor does not need to move, it can have the same configuration as a regular PC or tablet, and a detailed explanation of the supervisor terminal 104 is omitted.

[0050] Each of the manufacturing devices 106 is equipped with a device status sensor 108 and a user behavior sensor 110. These sensors 108 and 110 also have communication capabilities and are connected to the network 102.

[0051] Camera 114 continuously records video of users within the manufacturing area. By analyzing these images, user behavior can be estimated. For example, by pre-storing a standard image for each user and comparing images of users attaching or removing the wearable device 10 with the standard image, the user wearing the device can be identified. If it is not possible to deploy enough cameras to cover the entire manufacturing area simultaneously, several cameras 114 with variable angles that can capture a wide range of users can be deployed.

[0052] The information management server 116 includes a control unit 118, a communication unit 120, a location management unit 122, a user behavior management unit 124, a device status management unit 126, etc. The communication functions of sensors 108 and 110, the administrator terminal 104, the camera 114, and the communication unit 120 may be wired or wireless, similar to the communication functions of the wearable terminal 10. In the case of wireless communication, depending on the usage environment, any of the following may be used: short-range wireless communication such as Bluetooth, ZigBee, or UWB; medium-range wireless communication such as WiFi; or long-range wireless communication such as 3G / 4G or WiMAX.

[0053] The location management unit 122 collects location information for the wearable terminal 10 and the administrator terminal 104 at regular intervals based on the outputs of the optical sensor 62 and various sensors on the wearable terminal 10 and the administrator terminal 104. Furthermore, the location management unit 122 identifies the user of the wearable terminal 10 or the administrator terminal 104 and manages the terminal ID, user ID, and location of the wearable terminal 10 or the administrator terminal 104.

[0054] The user behavior management unit 124 collects information on the user's behavior and status on the wearable terminal 10 at regular intervals based on the output of the optical sensor 62, various sensors on the wearable terminal 10, and the user behavior sensor 110 on the manufacturing equipment 106, and manages the terminal ID, user ID, and behavior / status of the wearable terminal 10. The equipment status management unit 126 collects and manages information on the status of the manufacturing equipment at regular intervals based on the output of the equipment status sensor 108 on the manufacturing equipment 106. When the status of the equipment changes, the equipment status sensor 108 may notify the equipment status management unit 126, and information on the status of the manufacturing equipment may be collected.

[0055] When the information management server 116 detects an abnormality in a device in the device status management unit 126, it notifies the administrator terminal 104 of the location information and status of the relevant manufacturing device. At the same time, it determines the status of the workers and extracts the candidate workers who can most efficiently respond to the device where the abnormality has occurred, and presents them to the administrator terminal 104.

[0056] The outline of this embodiment is to automatically generate a work checklist and present it to the user, and to automatically check the corresponding items in the checklist when the user performs the work. To this end, the information management server 116 integrates and processes information obtained from various sensors of multiple sensors 108, 110 or terminals 10, 104 connected to the network 102, and automatically estimates / recognizes the actions of each worker. Based on the results, the information management server 116 generates a work procedure (checklist) and supports automatic input (automatic entry) into the corresponding fields in the checklist. Once the automatic input (automatic entry) of all items in the work checklist is complete, the information management server 116 automatically generates a work report.

[0057] The contents of the above work checklist differ for each manufacturing device to be maintained. Furthermore, the contents of the work checklist differ depending on the location of the malfunction within the manufacturing device. Therefore, the information management server 116 collects relevant information about the manufacturing device obtained from the device status sensor 108 related to the manufacturing device that requires maintenance, and automatically estimates / recognizes the location of the malfunction in the manufacturing device. The information management server 116 then automatically identifies the wearable terminal 10 worn by the maintenance worker and displays the maintenance details on the terminal 10 in the form of a work checklist.

[0058] Figure 7 shows an example of the electrical configuration of the wearable terminal 10. The wearable terminal 10 includes a CPU 140, a system controller 142, main memory 144, a storage device 146, a microphone 148, a speaker 54, a projection processing unit 150 (which controls the light source 28 and the display unit 30), a camera 59, a wireless communication device 152, a motion sensor 154, an eye-tracking sensor 156, a gesture sensor 158, a touchpad 55, a vibrator 68, a location information receiver 159, a GPS module 155, and the like.

[0059] The CPU 140 is a processor that controls the operation of various modules within the wearable terminal 10, and executes computer programs loaded into the main memory 144 from a storage device 146 consisting of non-volatile semiconductor memory such as an SSD or flash array. These programs include an operating system (OS) and various application programs. The CPU 140 executes various application programs and communicates with the information management server 116 via the network 102 using a wireless communication device 152, thereby performing the following processing, for example. For example, the CPU 140 inputs voice using the microphone 148 and sends the voice data to the information management server 116, takes pictures using the camera 59 and sends the image data to the information management server 116, sends input data from the motion sensor 154, gaze detection sensor 156, gesture sensor 158, touchpad 55, and position information receiver 159 to the information management server 116, plays sound from a stereo earphone or speaker 54 (not shown) connected to the earphone jack 54B, vibrates the vibrator 68, and performs various other controls. Speaker 54 is assumed to be a mono speaker, but if a stereo speaker is required, a speaker may also be provided on the left eye side temple, although this is not shown in Figures 1 and 2.

[0060] The system controller 142 is a device that connects the local bus of the CPU 140 to various components. The microphone 148 is connected to the microphone jack 56 and collects the user's voice or ambient sounds. By recognizing the user's voice or analyzing ambient sounds, the user's actions can be estimated and the user can be identified. For example, by pre-storing a standard voice for each user and comparing the voice emitted by the wearer with the standard voice, the wearer can be identified. Also, by analyzing ambient sounds, the work location where the wearer is located can be identified. The speaker 54 outputs alarms, etc., to attract the user's attention. The projection processing unit 150 outputs an image signal to the display unit 30 and lights up the light source 28 to project the image from the display unit 30 onto the screen 16. This image can include not only still images but also videos. The wireless communication device 152 has, for example, a wireless LAN function and wirelessly connects the wearable terminal 10 and the access point 112.

[0061] The motion sensor 154 is an integrated sensor combining a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis geomagnetic sensor. It detects the head movements of the user using the wearable terminal 10 and determines the direction the face is facing. The worker's status may also be detected using a microphone 148, a barometer, etc. The worker's status includes not only moving, resting, but also the content of the work, the progress of the work, etc. Using the movement detected by the motion sensor 154 and the altitude determined by the barometric pressure, it is possible to determine whether the feature quantities obtained from these detection results match the feature quantities of each process obtained in advance from the worker, etc., and to determine which of multiple processes is being performed or has been completed. Alternatively, by determining whether the feature quantities of ambient sound input from the microphone 148 match the feature quantities of ambient sound characteristic of each work process obtained in advance, it is also possible to determine which of multiple processes is being performed or has been completed.

[0062] The gaze detection sensor 156 is installed inside the center of the glasses frame, facing the user's face, and captures the user's eyeballs to detect eye movements. Furthermore, the gaze detection sensor 156 may also be capable of detecting the user's iris. The gesture sensor 158 is a sensor that distinguishes gestures made by finger movements. Specifically, it is a sensor that distinguishes user gestures by analyzing finger movements on the touchpad 55 provided on the projection device 12 or hand and finger movements in images captured by the camera 59. The vibrator 68 vibrates the temples of the wearable terminal 10 by vibrating the projection device 12, thereby conveying some information to the user. The location information receiver 159 receives beacons containing location information transmitted using short-range wireless communication such as RF-ID from multiple location information transmitters 113 placed within the area of ​​LAN 102. Because it is short-range wireless communication, the positions of the transmitter and receiver (wearable terminal) can be considered to be approximately the same. The GPS module 155 detects the position (x, y, z) of the wearable device 10. By combining this detection result with the detection result of the position information receiver 159 and the detection result of the optical sensor 62 shown in Figure 3, the user's position and its changes can be detected more accurately.

[0063] The display unit 30 displays instructions and incoming calls from the administrator terminal 104 and the information management server 116, as well as the worker's work status detected by the motion sensor 154, etc. This display image is projected onto the screen 16 by the projection processing unit 150.

[0064] With 148 microphones and 54 speakers, voice communication with the outside world is possible.

[0065] The administrator terminal 104 may have the same configuration as the wearable terminal 10, or it may be a regular PC or tablet. The electrical configuration of a regular PC or tablet is equivalent to that of the wearable terminal 10, but without the projection processing unit 150, camera 59, motion sensor 154, gaze detection sensor 156, gesture sensor 158, etc. The location of the administrator terminal 104 is detected by GPS.

[0066] An example of a device status sensor 108 attached to the manufacturing device 106 will be explained with reference to Figure 8. Figure 8(a) shows the mounting position on the device, and Figure 8(b) shows the configuration of the sensor 108. Conventionally, when a malfunction occurred in the manufacturing device, an operator had to inspect and repair the malfunctioning part of the device and investigate the cause of the malfunction each time. This resulted in longer maintenance times for the manufacturing device (non-operational periods for the manufacturing device) and a decrease in product productivity. In this embodiment, the information management server 116 collects and integrates information related to the device status obtained from the device status sensor 108 connected to the network 102, and automatically estimates or recognizes the location of the malfunction in the manufacturing device. As a result, automatic diagnosis of the location of the malfunction within the manufacturing device becomes possible, significantly reducing the maintenance time for the manufacturing device (non-operational periods for the manufacturing device) and preventing a decrease in product productivity.

[0067] The device status sensor 108 consists of an acceleration sensor 108a and a wireless communication device 108b. The acceleration signal detected by the acceleration sensor 108a is transmitted to the information management server 116 via the wireless communication device 108b and the network 102. The device status sensor 108 is provided with an adhesive part or a fixing part, and can be easily attached to existing manufacturing equipment. An adhesive layer may be pre-formed on the adhesive part, or adhesive may be applied during bonding. Alternatively, the device status sensor 108 may be attached to the manufacturing equipment by screwing the fixing part to the existing manufacturing equipment.

[0068] Automatically diagnosing malfunctions in manufacturing equipment requires the automatic collection of operational data from each part of the equipment. Purchasing or replacing manufacturing equipment for this purpose would incur enormous costs. However, in this embodiment, it is only necessary to add and fix very inexpensive sensor terminals to various parts of the existing manufacturing equipment. This allows for the addition of an automated malfunction diagnosis environment at a very low cost while maintaining the existing equipment environment.

[0069] As shown in Figure 8(a), the device status sensor 108 is fixed to, for example, a part of the moving belt 136, or a part of the movable arm 134 or movable shaft 132 that grips the item. Then, if a place that should move under normal conditions becomes stationary, it can be determined that a malfunction has occurred in that movable part.

[0070] The control unit 118 within the information management server 116 has pre-stored maintenance procedure manuals, which are manuals outlining repair, maintenance, and inspection methods for each location of malfunction in various manufacturing equipment, and generates an optimal work checklist based on the above-mentioned automatic diagnostic results.

[0071] Figure 8 illustrates an acceleration detection method as an example of a device status sensor 108. However, any physical or chemical quantity, such as temperature or current flow rate, may be used to identify faulty areas. Furthermore, faulty areas within the manufacturing equipment may be automatically diagnosed by comparing images captured by a camera or ambient sounds collected by a microphone.

[0072] When a manufacturing device experiencing a malfunction is detected using the method described in Figure 8, the information management server 116 automatically selects a worker to maintain the device and displays the maintenance procedure or a related work checklist on the worker's terminal 103. The information management server 116 selects a worker who, for example, (i) is located near the malfunctioning manufacturing device, (ii) can interrupt any work they are currently doing, and (iii) can perform the maintenance work. This minimizes the loss of worker travel time.

[0073] In this embodiment, as the most efficient way to locate workers located near a malfunctioning manufacturing device, an optical sensor 106a similar to the optical sensor 62 shown in Figure 4 and a wireless communication device 106b are attached to a part of the manufacturing device 106, as shown in Figure 9. As explained with reference to Figure 5, the leaked light 50 emitted from the wearable terminal 10 contains the terminal ID information of the terminal 10. Therefore, when the information contained in the leaked light 50 detected by the optical sensor 106a is transmitted to the information management server 116 via the wireless communication device 106b and the network 102, the information management server 116 can recognize which wearable terminal 10, i.e., which worker, is located near the malfunctioning manufacturing device. Based on this information, the information management server 116 selects a worker to maintain the target manufacturing device and sends a work checklist to the wearable terminal 10 of the corresponding worker. As shown in Figure 13(a), the work checklist is displayed on the screen 16 of the terminal 10. For the sake of clarity, the diagram in Figure 13(a) is simplified, but the actual work checklist is as follows.

[0074] • Put your belongings into the cart. Close the valve. • Turn off the on / off switch. Turn off the first light switch. Turn off the third light switch. In this way, leaked light 50 emitted from the wearable terminal 10 is detected, collected, and aggregated in real time, making it possible to easily and accurately identify workers located near the manufacturing equipment 106 to be maintained. This saves workers' travel time, shortens the maintenance period, and prevents a decrease in manufacturing efficiency.

[0075] Another example of a method for recognizing workers located near the target manufacturing equipment is to use a camera 114 pre-installed near the manufacturing equipment 106. The imaging device 114a in the camera 114 captures an image of the area near the manufacturing equipment 106, and the result is transmitted to the information management server 116 via the wireless communication device 14b and the network 102. The information management server 116 may analyze the received image and automatically identify the worker captured therein.

[0076] An example of a worker performing a task according to a work checklist will be explained with reference to Figure 10. When a work checklist like the one shown in Figure 13(a) is displayed on the screen 16, the worker begins the task. If displaying the entire checklist on the screen 16 at once results in text that is too small to read, only one or a few steps may be displayed, and the work progress may be automatically recognized sequentially. When each step of the task is completed, the displayed checklist may be updated in real time accordingly. In the work site shown in Figure 10(a), the worker places the goods 162 into the cart 164, closes the valve 170 (or rotates the handle 170 to a specified angle), turns off the on / off switch 172, and turns off the first lighting switch 176 and the third lighting switch 180, according to the work checklist. In this embodiment, the worker's actions are automatically recognized / identified in real time by a user action sensor 110 attached to the manufacturing device 106, and the work completion time is automatically written to the work checklist (see Figure 13(b)). Once the final work step is completed, a work report is automatically generated in the information management server 116, and its contents are displayed on the administrator terminal 104. The work report (Figure 13(b)) is the work checklist (Figure 13(a)) with the completion time entered.

[0077] Any detection technology or a combination thereof can be used as a method for automatically recognizing / identifying worker behavior using the user behavior sensor 110. For example, worker behavior images captured using camera 114 or 59 may be analyzed to automatically recognize / identify worker behavior. However, when using image analysis captured by camera 114 for automatic recognition / identification of worker behavior, the worker's behavior may be hidden in the image depending on the situation. Alternatively, voice recognition technology may be used. By stipulating that the worker utter a specific sound each time an item on the work checklist (maintenance work procedure) displayed on the wearable terminal 10 is completed, the microphone 148 can detect the input of the specific sound to perform automatic recognition / identification of worker behavior. Alternatively, ambient sounds generated when performing a specific task may be detected by microphone 148 or a microphone built into the device status sensor 108 to automatically recognize / identify worker behavior. Furthermore, there is also a method of automatically recognizing / identifying worker behavior by identifying predetermined worker gestures. As a method for recognizing worker gestures, images of the worker's actions captured by cameras 59 and 114 may be analyzed, or the results of detecting leaked light 50 emitted from the wearable terminal 10 using multiple optical sensors 62 installed at various locations or optical sensors 106a attached to the manufacturing apparatus 106 may be compared.

[0078] A pair of light-emitting units 166a and light-receiving units 166b are installed in the opening of the cart 164. The light-blocking when the luggage 162 passes through the opening of the cart 164 is detected, and the loading and unloading of the luggage 162 is automatically detected. Figure 10(b) shows the signal characteristics detected by the light-receiving unit 166b when luggage is loaded into or unloaded from the cart 164. The vertical axis shows the amount of light detected by the light-receiving unit, and the horizontal axis shows the passage of time. While the luggage 162 passes through the opening of the cart 164, the amount of light detected by the light-receiving unit decreases. The method for detecting the loading and unloading of luggage 162 from the cart 164 is not limited to the light-based method described above; any other method can be used.

[0079] This section describes an example of detection for real-time automatic recognition / identification of actions other than loading and unloading goods, such as "closing valves," "turning on / off switches," and "turning off lighting switches." Generally, in order to perform maintenance (maintenance, inspection, and repair) on manufacturing equipment, it is necessary for workers to directly contact specific locations within the manufacturing equipment. Utilizing this characteristic, this embodiment detects worker contact with specific locations within the manufacturing equipment and reflects this in the automatic recognition / identification of worker actions. This method is very easy to detect and ensures high automatic recognition / identification accuracy. In the example shown in Figure 10(a), a contact sensor 168 is attached to the valve 170, and transparent contact sensors are also attached to the on / off switch 172 and the lighting switch board 174. The lighting switch board 174 includes the first, second, and third lighting switches 176, 178, and 180.

[0080] A contact sensor, an example of a user behavior sensor 110, includes wireless communication functionality (e.g., short-range wireless communication) and a function for detecting the contact status of an operator. Any contact-detectable element, such as a piezoelectric element, a photointerrupter, or an acceleration sensor (gyro sensor), can be used to detect the contact status. This contact sensor can be attached to existing manufacturing equipment and is very inexpensive. Therefore, by attaching this contact sensor (user behavior sensor 110) to existing manufacturing equipment, a short-range wireless communication network environment can be added very inexpensively while maintaining the existing infrastructure.

[0081] Examples of user behavior sensors 110 are shown in Figures 11 and 12. Figure 11 shows a user behavior sensor 110 (contact sensor 168) that can be attached to an existing infrastructure such as an on / off switch 172 or a valve 170, while Figure 12 shows a user behavior sensor 110 that can be attached to an existing infrastructure such as a lighting switch board 174.

[0082] As shown in Figure 11, the user behavior sensor 110 includes an adhesive layer 202 at the bottom, with a control / communication circuit 204 and a solar cell 206 sequentially formed on top of it. A transparent conductive layer 208, a transparent intermediate layer 210, a transparent conductive layer 212, and a transparent textured layer 214 are sequentially laminated on top of the solar cell 206. The control / communication circuit 204 has a wireless communication function (short-range wireless communication) and a worker contact detection function. The solar cell 206 is used as the power source for both functions. If batteries were used as the power source, it would require the hassle of replacing the batteries. Also, if an external power source connected by wires were used as the power source, the wires would interfere with the worker's contact. However, with a solar cell 206, the hassle of replacing batteries is eliminated, and the user behavior sensor 110 can be used for a long period of time without interfering with the worker's contact.

[0083] By stacking control and communication circuits 204, which perform short-range wireless communication and control, beneath the solar cell 206, the power generation efficiency of the solar cell 206 is increased, and the planar size of the user behavior sensor 110 is reduced.

[0084] For the solar cell 206 to be used, ambient light must be able to irradiate it. On the other hand, it is desirable to place the part that detects user contact on the surface of the user action sensor 110. As a way to satisfy both requirements simultaneously, the part that detects contact is made transparent and a capacitive detection method is adopted. To detect worker contact or pressure using changes in capacitance, a structure can be adopted in which a transparent and elastic transparent intermediate layer 210 (for example, a sheet formed from a transparent organic material) is sandwiched between two transparent conductive layers 208 and 212 (for example, transparent organic material sheets). An AC voltage 216 is applied between the two transparent conductive layers 208 and 212 to resonate the transparent conductive layers 208 and 212. When a worker touches the surface of the user action sensor 110, the capacitance changes, and the above AC resonance state changes. By detecting this change in the AC resonance state, worker contact can be detected. Furthermore, the user action sensor 110 is not limited to this capacitive detection method; any element that allows at least a portion of ambient light to illuminate the solar cell 206 and that can detect contact or pressure may be used.

[0085] For example, the transparent layer on the surface of the user behavior sensor 110 has fine irregularities. This serves to prevent the surface from slipping, but if braille information is recorded on these irregularities, it becomes easier for people with visual impairments to use.

[0086] Any fixing method, such as screwing, can be used to secure the user behavior sensor 110 to a part of the existing manufacturing equipment, but direct bonding or adhesive attachment can save space. This bonding or adhesive attachment method is not limited to direct bonding using adhesive, but may also involve using adhesive sheets or adhesive tapes. For the on / off switch 172 and the lighting switch board 174, an adhesive layer 202 with double-sided tape properties can be used, and for the bulb 170, an adhesive layer 202 consisting of a transparent adhesive tape layer can be used.

[0087] Figure 12 shows a user behavior sensor 110 that is mounted on a lighting switch board 174. On the lighting switch board 174, the surfaces of the first, second, and third lighting switches 176, 178, and 180 are marked with words such as "Lighting 1," "Lighting 2," and "Lighting 3," so it is desirable that these words remain visible even when the user behavior sensor 110 is mounted. For this reason, it is desirable that the area above the lighting switches 176, 178, and 180 be transparent. Furthermore, it is necessary to independently detect the contact status of multiple lighting switches 176, 178, and 180. On the other hand, the lighting switch board 174 has spare space 182 where lighting switches 176, 178, and 180 are not installed. To accommodate this situation, the user behavior sensor 110 shown in Figure 12 has a transparent sheet 208, a transparent intermediate layer 210, a transparent sheet 212, and a transparent textured layer 214 sequentially laminated on top of an adhesive layer 202. Transparent sheets 208 and 212 correspond to the transparent conductive layers 208 and 212 in Figure 11. Transparent sheet 208 includes three transparent conductive regions 208a, 208b, and 208c, and transparent sheet 210 includes three transparent conductive regions 210a, 210b, and 210c. Transparent conductive regions 208a and 210a are located at the positions of the first lighting switch 176, transparent conductive regions 208b and 210b are located at the positions of the second lighting switch 178, and transparent conductive regions 208c and 210c are located at the positions of the third lighting switch 180. An AC voltage 216 is applied between transparent sheets 208 and 212. By dividing the transparent sheets into three regions corresponding to the three lighting switches in this way, the contact status of the three lighting switches can be detected independently and individually. Braille information can also be formed on the transparent uneven layer 214 on the surface.

[0088] A control circuit 204a and a communication circuit 204b are formed on a transparent uneven layer 214 in a spare space 182 where no lighting switch is installed, and a solar cell 206 is formed on top of them. Because the solar cell 206 is positioned at the very top, the power generation efficiency is high. In addition, because the control circuit 204a, the communication circuit 204b and the solar cell 206 are positioned vertically, the planar size of the user action sensor 110 is reduced.

[0089] According to this embodiment, the system detects the status of a wearable terminal and manufacturing equipment, and based on the detection results, displays the work procedure on the wearable terminal of a worker who is able to perform maintenance, inspection, or repair work near the manufacturing equipment, thereby providing the worker with meaningful information. Furthermore, based on the detection results of the status of the wearable terminal and manufacturing equipment, the system determines the completion of each step in a series of tasks and automatically creates a work report that records the execution of the work, thereby significantly reducing the workload for the worker. Moreover, since the completion of work is detected by attaching a contact sensor to the manufacturing equipment, it is possible to estimate / recognize the worker's actions with high accuracy in a very simple and inexpensive way without modifying existing manufacturing equipment.

[0090] The above description uses the maintenance of manufacturing equipment as an example. However, this embodiment is not limited to this, and may monitor user behavior for other purposes and display corresponding work content. Furthermore, while contact sensors were provided in areas that users may come into contact with as a means of monitoring user behavior, other sensors may be used as well.

[0091] Although a glasses-type wearable terminal has been described as an embodiment, the present invention is also applicable to other head-mounted types such as goggles and helmets, as well as wristbands, pendants, etc. For example, if it is made into a helmet or goggles, the projection device 12 and camera 59 can be attached to the helmet or goggles, and it can be used by regular glasses wearers. Furthermore, if it is made into a helmet, the speaker 54 can be attached to the inside of the helmet, so clearer sound can be heard, and the microphone can be attached to the helmet and its position can be adjusted, improving the sound collection ability of the microphone.

[0092] The types of sensors used to detect the status of manufacturing equipment and wearable devices are not limited to those described above, and various sensors can be used as appropriate.

[0093] The present invention is also applicable to wearable devices other than those worn on the head. Furthermore, it is applicable to small, lightweight, portable electronic devices that are always with the user, even if they are not wearable, such as notebook computers, tablet computers, and smartphones.

[0094] The division of functions between the wearable device and the information management server is not limited to the explanation given above. Some of the functions described for the wearable device may be implemented as functions of the information management server, and some of the functions described for the information management server may be implemented as functions of the wearable device.

[0095] It should be noted that the present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0096] 10...Wearable device, 12...Projection device, 16...Screen, 54...Speaker, 55...Patch pad, 59...Camera, 62...Light sensor, 104...Administrator terminal, 106...Manufacturing equipment, 108...Equipment status sensor, 110...User behavior sensor, 114...Camera, 116...Information management server, 148...Microphone, 152...Wireless communication device.

Claims

1. A display unit that displays an image, A projection device used to project the aforementioned display image in front of the user and to form a virtual image corresponding to the aforementioned display image in front of the user, A camera that captures an image in the user's line of sight, A wearable device equipped with, The projection device includes a lens that projects divergent light obtained from the display unit in front of the user, and a user operation unit including a slide switch or a rotary switch. The camera is installed at a first position on the outer surface of the projection device. The user operation unit is installed at a second position on the outer surface of the projection device. The projection angle of the virtual image is adjustable based on the user's operation of the user control unit. The user can adjust the projection angle by blindly touching the user control unit while visually observing the virtual image, and by adjusting the projection angle, the display position of the virtual image can be adjusted to match the shape or size of the user's head. The virtual image is modified to reflect the recognized user's actions when the user's actions are recognized based on the movements of the user's hands and fingers in the image captured by the camera. The above content is a list showing the work procedures to be performed by the user, A wearable device in which, when the completion of a task is recognized as an action of the user, the list is updated, and the updated list represents the completion of the task.

2. A server that stores images, A wearable terminal comprising: a display unit connected to the server and displaying an image based on an image transmitted from the server; a projection device used to project the display image in front of the user and to form a virtual image corresponding to the display image in front of the user; and a camera that captures an image in front of the user's line of sight. A system that is equipped with, The projection device includes a lens that projects divergent light obtained from the display unit in front of the user, and a user operation unit including a slide switch or a rotary switch. The camera is installed at a first position on the outer surface of the projection device. The user operation unit is installed at a second position on the outer surface of the projection device. The projection angle of the virtual image is adjustable based on the user's operation of the user control unit. The user can adjust the projection angle by blindly touching the user control unit while visually observing the virtual image, thereby adjusting the display position of the virtual image to match the shape or size of the user's head. The virtual image is modified to reflect the recognized user's actions when the user's actions are recognized based on the movements of the user's hands and fingers in the image captured by the camera. The above content is a list showing the work procedures to be performed by the user, The system recognizes the completion of a task as an action by the user, and updates the list so that the updated list represents the completion of the task.

3. A display method for a wearable terminal comprising: a display unit that displays an image; a projection device used to project the image in front of the user and to form a virtual image corresponding to the image in front of the user; and a camera that captures an image in front of the user's line of sight, The projection device includes a lens that projects divergent light obtained from the display unit in front of the user, and a user operation unit including a slide switch or a rotary switch. The camera is installed at a first position on the outer surface of the projection device. The user operation unit is installed at a second position on the outer surface of the projection device. The projection angle of the virtual image is adjustable based on the user's operation of the user control unit. The user can adjust the projection angle by blindly touching the user control unit while visually observing the virtual image, and by adjusting the projection angle, the display position of the virtual image can be adjusted to match the shape or size of the user's head. The virtual image is modified to reflect the recognized user's actions when the user's actions are recognized based on the movements of the user's hands and fingers in the image captured by the camera. The above content is a list showing the work procedures to be performed by the user, A display method wherein, when the completion of a task is recognized as an action by the user, the list is updated, and the updated list represents the completion of the task.

Images