Wearable Devices

Abstract

An AR glass is a transmissive display device to be fitted to the head of a user. An imaging device outputs a captured image acquired by image capturing the front of the user. An inertial measurement device detects a physical amount relating to the movement of the head of the user. A display control unit causes the AR glass to display a guide image indicating the outer edge of the captured image during a display period, from a start time to an end time, defined on the basis of at least an output signal of the inertial measurement device.

Description

[Technical field]

[0001] The present invention relates to a wearable device. [Background technology]

[0002] Conventionally, XR glasses that apply XR technologies such as AR (Augmented Reality), VR (Virtual Reality) and MR (Mixed Reality) have become widespread. The XR glasses may be provided with an imaging device that captures an image in front of the user. The captured image captured by the imaging device is used, for example, to present information to assist the user wearing the XR glasses in performing a task. Alternatively, the captured image is used for recording or the like to retrospectively evaluate whether the user performed the task appropriately.

[0003] For example, the following Patent Document 1 discloses a work support system that uses a marker placed in real space to align a virtual object placed in a virtual space with an object in the real space. Specifically, the work support system includes a display unit that displays an image of the virtual space corresponding to the real space, an imaging unit that captures an image of a range including the center of the user's visual field, a position information detection unit that detects the positional relationship between the marker and the imaging unit based on the captured image of the marker, and a display control unit that controls the display of the image of the virtual space based on the detection result of the image of the marker detected while the gaze is on the marker and the detection result of mechanically detecting the user's posture. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-170249 A Summary of the Invention [Problem to be solved by the invention]

[0005] In general, the resolution and mounting position of the imaging device mounted on the XR glasses differ from product to product. Therefore, the imaging range assumed by the user may not match the actual imaging range. If the imaging range assumed by the user does not match the actual imaging range, it may not be possible to obtain the captured image required for processing by the XR glasses.

[0006] To address this issue, it is conceivable to constantly display a guide image showing the outer edge of the imaging range on the XR glasses. However, there are problems with the guide image, such as it being an obstacle to work or narrowing the display area of ​​the XR glasses.

[0007] An object of the present invention is to display the imaging range of an imaging device without blocking the user's field of view for a long period of time. [Means for solving the problem]

[0008] A wearable device according to one aspect of the present invention includes a transmissive display device worn on a user's head, an imaging device that outputs an image obtained by imaging an area in front of the user, a sensor that detects a physical quantity, and a control device that causes the display device to display a guide image indicating the outer edge of the captured image during a display period from a start timing to an end timing that is determined based on at least an output signal from the sensor. Equipped with. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to display the imaging range of an imaging device without blocking the user's field of view for a long period of time. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an overview of a wearable device 1 according to an embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a wearable device 1 according to an embodiment. [Diagram 3] 1 is an explanatory diagram showing the appearance of the AR glasses 10. FIG. [Figure 4] FIG. 2 is a block diagram showing the configuration of the AR glasses 10. [Diagram 5] 2 is a block diagram showing the configuration of a mobile device 20. FIG. [Figure 6] FIG. 2 is a front view of the device DV1. [Figure 7A] FIG. 2 is a diagram showing the relationship between a device DV1 and an XY coordinate system in real space. [Figure 7B] FIG. 2 is a diagram showing the relationship between a device DV1 and an XY coordinate system in real space. [Figure 8] 11 is a diagram showing the relationship between a captured image showing a device DV1 and an xy coordinate system. [Figure 9] 13 is a flowchart showing the operation of the processing device 206. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A. Embodiment A-1. System configuration Fig. 1 is a diagram showing an overview of a wearable device 1 according to an embodiment. Fig. 2 is a block diagram showing a configuration of the wearable device 1 according to an embodiment. The wearable device 1 includes AR glasses 10 worn on the head of a user U, a portable device 20 held by the user U, and an inertial measurement unit 30 that measures the movement of the user U's head.

[0012] In this embodiment, the user U, wearing the wearable device 1, performs wiring work between multiple devices DV stored in a rack RA. The wearable device 1 includes an imaging device 124 (see FIG. 4). The wearable device 1 stores captured images captured by the imaging device 124 in the storage device 205 (see FIG. 5) of the mobile device 20. The captured images show the user U performing work on the device DV. By recording such captured images, it is possible to verify after the fact whether the work procedure and work content of the user U were correct, and to investigate the cause if a malfunction occurs. This improves convenience for the user U and the person requesting the work.

[0013] In addition, the wearable device 1 may present the user U with a manual for wiring work, or may monitor whether any abnormalities are occurring in the device DV during work by performing image processing on captured images using AI (Artificial Intelligence).

[0014] A-2. Inertial Measurement Unit 30 The inertial measurement unit 30 calculates, for example, the acceleration of the user U's head on each of three axes representing a three-dimensional space and the angular velocity of the user U's head when each of the three axes is used as an axis of rotation, and outputs the measurement results as an output signal. The inertial measurement unit 30 is an example of a sensor that detects a physical quantity. In this embodiment, the inertial measurement unit 30 detects a physical quantity related to the movement of the head. The acceleration and angular acceleration are examples of physical quantities related to the movement of the user U's head.

[0015] More specifically, the inertial measurement unit 30 includes various sensors, a processing unit, and a communication unit. The various sensors include at least one of a gyro sensor, an acceleration sensor, a magnetic sensor, and a pressure sensor, for example. The processing unit calculates the acceleration in each of three axes representing a three-dimensional space and the angular velocity about each of the three axes as a rotation axis, using the measurement values ​​measured by the various sensors. The communication unit is capable of communicating with at least the communication unit 203 of the mobile device 20. The communication unit transmits an output signal including the acceleration and angular velocity calculated by the processing unit to the mobile device 20. The inertial measurement unit 30 transmits the output signal at a predetermined measurement interval.

[0016] In this embodiment, the inertial measurement unit 30 is attached to a hat worn on the head of the user U. Therefore, the inertial measurement unit 30 measures the acceleration and angular velocity every time the head of the user U moves. The mobile device 20, which will be described later, can calculate the moving speed V of the head of the user U using the output signal of the inertial measurement unit 30.

[0017] In this embodiment, the inertial measurement unit 30 is attached to a hat worn by the user U, but the inertial measurement unit 30 may be built into the AR glasses 10, for example. In this case, the acquisition unit 230 acquires an output signal transmitted from the communication device 123 of the AR glasses 10 via the communication device 203. The inertial measurement unit 30 is not limited to being attached to a hat worn by the user U, and may be attached anywhere that moves in conjunction with the movement of the user U's head.

[0018] A-3.AR Glasses 10 The AR glasses 10 are a transmissive wearable display worn on the head of the user U. The AR glasses 10 are an example of a display device. The AR glasses 10 display virtual objects on display panels provided on the lenses 110A and 110B for both eyes based on the control of the portable device 20. In this embodiment, the AR glasses 10 display guide images GI (GI1, GI2) indicating the outer edge of an image captured by the imaging device 124 as, for example, a virtual object. Note that, as the display device, for example, a goggle-shaped transmissive head-mounted display having the same functions as those of the AR glasses 10 may be used.

[0019] 3 is an explanatory diagram showing the external appearance of the AR glasses 10. The AR glasses 10 have temples 101 and 102, a bridge 103, frames 104 and 105, rims 106 and 107, lenses 110A and 110B, and an imaging lens LEN that can be seen from the outside.

[0020] An imaging lens LEN that constitutes an imaging device 124 shown in FIG.

[0021] The frame 104 is provided with a display panel for the left eye and an optical member for the left eye. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye displays an image based on, for example, control from the mobile device 20 described later. The optical member for the left eye is an optical member that guides light emitted from the display panel for the left eye to the lens 110A. In addition, the frame 104 is provided with a speaker 122 described later.

[0022] A display panel for the right eye and an optical member for the right eye are provided on the frame 105. The display panel for the right eye displays an image based on control from the mobile device 20, for example. The optical member for the right eye is an optical member that guides light emitted from the display panel for the right eye to the lens 110B. In addition, a speaker 122, which will be described later, is provided on the frame 105.

[0023] The rim 106 holds the lens 110A, and the rim 107 holds the lens 110B.

[0024] Each of the lenses 110A and 110B has a half mirror. The half mirror of the lens 110A transmits light representing the real space R, thereby guiding the light representing the real space R to the left eye of the user U. The half mirror of the lens 110B reflects the light guided by the optical member for the left eye to the left eye of the user U. The half mirror of the lens 110B transmits the light representing the real space R, thereby guiding the light representing the real space R to the right eye of the user U. The half mirror of the lens 110B reflects the light guided by the optical member for the right eye to the right eye of the user U.

[0025] When the user U wears the AR glasses 10, the lenses 110A and 110B are positioned in front of the left and right eyes of the user U. The user U wearing the AR glasses 10 visually recognizes a real space R represented by light transmitted through the lenses 110A and 110B superimposed on a virtual object projected by a projection device 121, which will be described later.

[0026] FIG. 4 is a block diagram showing the configuration of the AR glasses 10. The AR glasses 10 include the temples 101 and 102, the bridge 103, the frames 104 and 105, the rims 106 and 107, the lenses 110A and 110B, and the imaging lens LEN described above, as well as a projection device 121, a speaker 122, a communication device 123, an imaging device 124, a storage device 125, a processing device 126, and a bus 127. Each component shown in FIG. 4 is stored in the frames 104 and 105, for example. The projection device 121, the speaker 122, the communication device 123, the imaging device 124, the storage device 125, and the processing device 126 are connected to each other by a bus 127 for communicating information. The bus 127 may be configured using a single bus, or may be configured using different buses for each element such as a device.

[0027] The projection device 121 includes a lens 110A, a display panel for the left eye, an optical member for the left eye, a lens 110B, a display panel for the right eye, and an optical member for the right eye. The projection device 121 displays a virtual object based on the control from the mobile device 20. As described above, the virtual object in this embodiment is, for example, a guide image GI.

[0028] The speaker 122 is located in each of the frames 104 and 105. The speaker 122 may not be located in each of the frames 104 and 105, but may be located in, for example, one of the frames 104 and 105, at least one of the temples 101 and 102, or the bridge 103. The speaker 122 is controlled directly by the mobile device 20 or via a processing device 126 of the AR glasses 10. The speaker 122 outputs a work auxiliary sound, such as an alarm sound to alert the user U during work. The speaker 122 may not be included in the AR glasses 10 and may be separate from the AR glasses 10.

[0029] The communication device 123 communicates with the communication device 203 (see FIG. 4) of the mobile device 20 using wireless communication or wired communication. In this embodiment, the communication device 123 communicates with the communication device 203 of the mobile device 20 using short-range wireless communication such as Bluetooth (registered trademark).

[0030] The imaging device 124 outputs a captured image obtained by capturing an image in front of the user U. More specifically, the imaging device 124 captures an image of a subject and outputs image data D indicating the captured captured image. In this embodiment, the imaging device 124 is disposed so that the imaging direction of the imaging device 124 matches the direction of the head of the user U. Thus, the captured image shows an object or the like located in front of the user U (in the direction of view). The captured image generated by the imaging device 124 is transmitted as image data D to the mobile device 20 via the communication device 123. The imaging device 124 repeats imaging at a predetermined imaging interval, and transmits the generated image data D to the mobile device 20 each time imaging is performed.

[0031] The imaging device 124 has, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens LEN (see FIG. 3). For example, the imaging optical system may have various optical elements such as a prism, or may have a zoom lens or a focus lens. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0032] The storage device 125 is a recording medium readable by the processing device 126. The storage device 125 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM). The storage device 125 stores a program PG1.

[0033] The processing device 126 includes one or more central processing units (CPUs). The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer.

[0034] The processing device 126 reads the program PG1 from the storage device 125. The processing device 126 executes the program PG1 to function as an operation control unit 130. The operation control unit 130 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0035] The operation control unit 130 controls the operation of the AR glasses 10. For example, the operation control unit 130 transmits an image captured by the imaging device 124 to the mobile device 20. The operation control unit 130 also provides a control signal for displaying a guide image GI received by the communication device 123 from the mobile device 20 to the projection device 121. In this case, the projection device 121 displays the guide image GI indicated by the control signal on the display panel.

[0036] A-4. Mobile Devices 20 The portable device 20 stores the captured images captured by the imaging device 124 of the AR glasses 10. The portable device 20 also causes the AR glasses 10 to display a guide image GI during a display period TM, which will be described later. The portable device 20 is an example of a control device. Examples of the portable device 20 include a smartphone and a tablet.

[0037] 5 is a block diagram showing the configuration of the mobile device 20. The mobile device 20 includes a touch panel 201, a communication device 203, a storage device 205, a processing device 206, and a bus 207. The touch panel 201, the communication device 203, the storage device 205, and the processing device 206 are connected to each other by the bus 207 for communicating information. The bus 207 may be configured using a single bus, or may be configured using different buses for each device.

[0038] The touch panel 201 displays various information to the user U and detects touch operations of the user U. The touch panel 201 serves as both an input device and an output device. For example, the touch panel 201 is configured by bonding a touch sensor unit capable of detecting a touch operation between various display panels, such as a liquid crystal display panel or an organic EL display panel, and a cover glass. For example, when the finger of the user U is in contact with the touch panel 201, the touch panel 201 periodically detects the contact position of the finger of the user U on the touch panel 201 and transmits touch information indicating the detected contact position to the processing device 206.

[0039] The communication device 203 communicates with the communication device 123 (see FIG. 4) of the AR glasses 10 using wireless communication or wired communication. In this embodiment, the communication device 203 communicates with the communication device 123 using short-range wireless communication of the same type as the communication device 123 of the AR glasses 10. The communication device 203 also communicates with the communication device of the inertial measurement unit 30 (see FIGS. 1 and 2) using wireless communication or wired communication. In this embodiment, the communication device 203 communicates with the inertial measurement unit 30 using short-range wireless communication.

[0040] The storage device 205 is a recording medium readable by the processing device 206. The storage device 205 includes, for example, a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 205 stores a program PG2 and a captured image.

[0041] The image data D is data corresponding to a captured image captured by the imaging device 124 of the AR glasses 10. As described above, the captured image shows the user U performing an operation on the device DV.

[0042] The processing unit 206 includes one or more CPUs. The CPUs are one example of one or more processors. Each of the processor and the CPU is an example of a computer.

[0043] The processing device 206 reads the program PG2 from the storage device 205. The processing device 206 executes the program PG2 to function as an acquisition unit 230, a storage control unit 232, and a display control unit 234. At least one of the acquisition unit 230, the storage control unit 232, and the display control unit 234 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0044] The acquisition unit 230 acquires an output signal of the inertial measurement unit 30. The acquisition unit 230 also acquires image data D corresponding to a captured image captured by the imaging device 124 mounted on the AR glasses 10. The acquisition unit 230 acquires an output signal of the inertial measurement unit 30 and image data D indicating the captured image, which are received by the communication device 203. The acquisition unit 230 sequentially acquires the output signal and image data D while the user U is working.

[0045] The storage control unit 232 stores the image data D acquired by the acquisition unit 230 in the storage device 205. The storage control unit 232 may store the image data D after lowering the resolution thereof according to, for example, the storage capacity of the storage device 205. The storage control unit 232 may also secure the storage capacity of the storage device 205 by sequentially deleting image data D that has been stored in the storage device 205 for a predetermined period of time or more.

[0046] The display control unit 234 displays a guide image GI indicating the outer edge of the captured image on the AR glasses 10. As described above, the image capturing device 124 mounted on the AR glasses 10 has a different resolution and a different mounting position for each product, so that the image capturing range IF assumed by the user U may not match the actual image capturing range IF. In particular, when the visual field range VF of the user U changes, there is a possibility that the image capturing range IF assumed by the user U may deviate from the actual image capturing range IF. The visual field range VF of the user U may change, for example, when the user U moves in front of a new work target device DV and starts working, or when the user U working looks around and then returns his / her gaze to the work target device DV.

[0047] Fig. 6 is a diagram showing a schematic relationship between the visual field range VF of the user U in the real space R and the imaging range IF of the imaging device 124. Fig. 6 shows a state in which the user U is positioned in front of a rack RA that houses the devices DV1 to DV3, and is looking in the direction of the devices DV1 to DV3. Note that, for convenience of illustration, Fig. 6 shows the visual field range VF of the user U narrower than it actually is.

[0048] Generally, the visual field range VF of the user U has a width of about 100° to the left and right of the line of sight, totaling about 200°. In contrast, the angle of view of the imaging device 124 is about 80° even if it is an ultra-wide-angle lens that has a wider angle of view than a telephoto lens. That is, the imaging range IF of the imaging device 124 is narrower than the visual field range VF of the user U. Therefore, there may be cases where an object that the user U is looking at is not captured in the captured image.

[0049] For example, when the user U is performing a connection operation between the device DV1 and the device DV2, the user U needs to perform the operation while checking the value of the indicator IN1 provided on the device DV1. In this case, in order to evaluate after the fact whether the user U's operation was performed accurately, the indicator IN1 needs to appear in the captured image. On the other hand, when the imaging range IF is at the position shown in FIG. 6, the user U can check the value of the indicator IN1 in the visual field range VF, but the indicator IN1 is not included in the imaging range IF, and the orientation of the imaging device 124 needs to be adjusted. For example, when the user U moves the direction of his / her head upward, the imaging range IF of the imaging device 124 moves upward. Due to this movement, the imaging range IF includes the indicator IN1.

[0050] The display control unit 234 causes the AR glasses 10 to display a guide image GI that enables the user U to grasp the imaging range IF, in order to enable adjustment of the position of the imaging range IF within the visual field range VF.

[0051] FIG. 7A is an explanatory diagram showing an example of a display mode of a guide image GI. In FIG. 7A and FIG. 7B described later, the visual field range VF is visually recognized by the user U through the lenses 110A and 110B of the AR glasses 10. In the example of FIG. 7A, the guide image GI1 is a frame-shaped image showing the outer edge of the imaging range IF. The display control unit 234 uses the projection device 121 to display the guide image GI1 on the lenses 110A and 110B. Thus, the user U can recognize which range of his / her visual field range VF is included in the imaging range IF.

[0052] When the positional relationship between the imaging device 124 and the lenses 110A and 110B is fixed, and the imaging device 124 does not pan, tilt, zoom, or the like, and the imaging range IF is fixed, the size of the guide image GI1 and the display position of the guide image GI1 on the lenses 110A and 110B are constant. Also, when the imaging device 124 has a function of panning, tilting, zooming, or the like, and the size and direction of the imaging range IF are variable, the display control unit 234 calculates the position of the outer edge of the imaging range IF (the outer edge of the captured image) in the visual field range VF of the user U, and displays the guide image GI1.

[0053] Fig. 7B is an explanatory diagram showing another example of the display mode of the guide image GI. In the example of Fig. 7B, the guide image GI2 is an image in which the captured image captured by the imaging device 124 is displayed in real time. The display control unit 234 uses the projection device 121 to cause the lenses 110A and 110B to reduce the captured image of the imaging device 124 and display it as the guide image GI2. Thus, the user U can grasp how the visual field range VF appears as a captured image.

[0054] 6, the imaging range IF of the imaging device 124 is entirely included in the visual field range VF of the user U, but there are cases where a part of the imaging range IF of the imaging device 124 is not included in the visual field range VF of the user U. For example, depending on the design of the AR glasses 10, the imaging device 124 may be provided at a position corresponding to the forehead of the user U, and in such a case, the imaging range IF of the imaging device 124 may include an area above the visual field range VF of the user U. Even in such a case, by displaying the guide image GI, the user U can recognize the deviation between his / her own visual field range VF and the imaging range IF.

[0055] Here, although the guide image GI as shown in FIGS. 7A and 7B is convenient when the visual field range VF of the user U changes, after aligning the imaging range IF in the visual field range VF, it may interfere with the work. Therefore, the display control unit 234 limits the display of the guide image GI to only a certain period immediately before the movement of the user U stops. That is, the display control unit 234 causes the guide image GI indicating the outer edge of the captured image to be displayed on the AR glass 10 during the display period TM from the start timing determined based on at least the output signal of the inertial measurement device 30 to the end timing.

[0056] FIG. 8 is a graph schematically showing the display period TM of the guide image GI. In the graph of FIG. 8, the vertical axis represents the moving speed V of the head of the user U, and the horizontal axis represents the time T. The moving speed V is calculated by integrating the acceleration measured by the inertial measurement device 30 over time. Two threshold values, speed V1 and speed V2 (<V1), are set for the moving speed V. The speed V1 is an example of the first threshold value, and the speed V2 is an example of the second threshold value. The state where the moving speed V exceeds the speed V1 is the first state, the state where the moving speed V is greater than the speed V2 and less than or equal to the first threshold value is the second state, and the state where the moving speed V is less than or equal to the speed V2 is the third state.

[0057] In the graph of FIG. 8, at time T0, the moving speed V of the head of the user U is substantially zero. Thereafter, the movement of the head of the user U starts, and at time T1, the moving speed V exceeds the speed V2. After time T1, the moving speed V also increases, and at time T2, it exceeds the speed V1. After exceeding the speed V1 at time T2, the moving speed V reaches a peak. After the peak, the moving speed V decelerates and becomes less than or equal to the speed V1 at time T3. Although there are also time periods when the moving speed V accelerates again after time T3, the moving speed V decelerates without exceeding the speed V1 and becomes less than or equal to the speed V2 at time T4. After time T4, the moving speed V becomes substantially zero and approaches a stationary state. That is, from time T0 to T1 is the third state, from time T1 to T2 is the second state, from time T2 to T3 is the first state, from time T3 to T4 is the second state, and after time T4 is the third state.

[0058] The display control unit 234 causes the AR glasses 10 to display the guide image GI from time T3 to time T4. That is, the display control unit 234 sets the display period TM of the guide image GI to be from time T3 to time T4. The reason why the display period TM is set to be from time T3 to time T4 is that displaying the guide image GI during this period is considered to be most useful.

[0059] For example, when the movement of the user U is accelerating or the moving speed V is high (exceeding the speed V1), as in the case of time T1 to time T3, the visual field range VF changes significantly, so that displaying the guide image GI is not very useful. On the other hand, as in the case of time T3, when the moving speed V exceeds the speed V1 once and becomes equal to or lower than the speed V1, the user U is likely to stop moving after that. Also, when the moving speed V decreases, the change in the visual field range VF of the user U becomes small, and it becomes easier to align the image capture range IF with reference to the guide image GI. Therefore, the display control unit 234 starts displaying the guide image GI on the AR glasses 10. Also, when the moving speed V becomes equal to or lower than the speed V2, the movement of the user U is almost finished, and there is a possibility that the user U will resume the work. Therefore, the display control unit 234 ends displaying the guide image GI on the AR glasses 10. In this way, by displaying the guide image GI before the user U completely stops, the user U can align the image capture range IF to a desired position during a series of movements during the movement.

[0060] In other words, the display control unit 234 calculates the movement velocity V of the head of the user U based on the acceleration indicated by the output signal of the inertial measurement device 30. The display control unit 234 defines a state in which the movement velocity V exceeds a velocity V1 as a first state, a state in which the movement velocity V is equal to or less than a velocity V1 and exceeds a velocity V2 as a second state, and a state in which the movement velocity V is equal to or less than a velocity V2 as a third state. The display control unit 234 defines the timing of transition from the first state to the second state as the start timing of the display period TM, and defines the timing of transition from the second state to the third state as the end timing of the display period TM.

[0061] A-5. Operation of the Processing Unit 206 9 is a flowchart showing the operation of the processing device 206. The processing device 206 functions as the acquisition unit 230 and acquires an output signal of the inertial measurement device 30 (step S101). The output signal of the inertial measurement device 30 indicates the acceleration of the head of the user U. The processing device 206 functions as the display control unit 234 and calculates the moving velocity V of the head of the user U using the output signal of the inertial measurement device 30 (step S102). Note that the acquisition of the output signal of the inertial measurement device 30 in step S101 and the calculation of the moving velocity V in step S102 are also executed sequentially in the subsequent steps.

[0062] The processing device 206 functions as the display control unit 234 and waits until the moving speed V exceeds the speed V1, i.e., until the first state is reached (step S103: NO). When the moving speed V exceeds the speed V1 (step S103: YES), the processing device 206 functions as the display control unit 234 and waits until the moving speed V becomes equal to or less than the speed V1, i.e., until the first state is transitioned to the second state (step S104: NO).

[0063] When the moving speed V becomes equal to or less than the speed V1 (step S104: YES), the processing device 206 functions as the display control unit 234 and starts displaying the guide image GI on the AR glasses 10 (step S105). That is, the timing when the first state transitions to the second state is the start timing of the display period TM. The processing device 206 functions as the display control unit 234 and waits until the moving speed V becomes equal to or less than the speed V2 or the moving speed V exceeds the speed V1 again (step S106: NO). During this time, the display of the guide image GI on the AR glasses 10 continues.

[0064] When the moving speed V becomes equal to or less than the speed V2, or when the moving speed V exceeds the speed V1 again (step S106: YES), the processing device 206 functions as the display control unit 234 and ends the display of the guide image GI on the AR glasses 10 (step S107). That is, the timing when the second state transitions to the third state, or the timing when the second state transitions to the first state, is the end timing of the display period TM. After that, the processing device 206 returns the process to step S101, and repeats the subsequent processes.

[0065] A-6. Summary of embodiments As described above, according to the embodiment, the wearable device 1 causes the AR glasses 10 to display a guide image GI indicating the outer edge of a captured image during a display period TM from a start timing to an end timing determined based on an output signal of the inertial measurement device 30. The guide image GI is displayed only during the display period TM. This makes it possible to prevent the visual field of the user U from being blocked for a long period of time by the display of the imaging range IF of the imaging device 124. In addition, the user U can grasp the position of the imaging range IF in the visual field range VF by using the guide image GI. This improves the usefulness of the captured image.

[0066] Furthermore, the wearable device 1 sets the timing of transition from the first state to the second state as the start timing of the display period TM, and sets the timing of transition from the second state to the third state as the end timing of the display period TM. Therefore, due to a decrease in the moving speed V, the change in the visual field range VF of the user U becomes small, and the guide image GI is displayed at a timing when it is easy to align the imaging range IF with reference to the guide image GI. Therefore, the usefulness of the guide image GI is improved.

[0067] Furthermore, the wearable device 1 acquires a physical quantity related to the movement of the head of the user U using the inertial measurement device 30. Therefore, the wearable device 1 can detect the physical quantity related to the movement of the head of the user U with high accuracy.

[0068] B: Variation The following are examples of modified aspects of the above-described embodiment. Two or more aspects selected from the following modified aspects may be combined as appropriate within the scope of not being mutually contradictory.

[0069] B1: First modified example For example, when the user U is skilled in a task, the time from when the user U moves to when the user U stops moving is short, and the display period TM of the guide image GI may be very short. On the other hand, if the display period TM of the guide image GI is too short, the display of the guide image GI ends before the user U checks the imaging range IF, and the usefulness of displaying the guide image GI decreases. Therefore, when the display period TM is short, the display control unit 234 may continue to display the guide image GI for a certain period of time even after the moving speed V of the user U becomes equal to or less than V2.

[0070] That is, when the time from the start timing to the end timing of the display period TM is equal to or shorter than the first time, the display control unit 234 may continue to display the guide image GI for a second time that is longer than the first time. The first time and the second time represent lengths of time.

[0071] According to the first modified example, when the display period TM is equal to or shorter than the first hour, the display of the guide image GI is continued until the second hour. This ensures time for the user U to check the guide image GI, improving convenience.

[0072] B2: Second variant For example, depending on the movement of the user U, the state in which the moving speed V is equal to or less than the speed V1 and exceeds the speed V2 may continue for a long time. In this case, the guide image GI is displayed continuously in the visual field range VF of the user U for a long time, which may annoy the user U. Therefore, the display control unit 234 may end the display of the guide image GI regardless of the moving speed V of the user U when the display period TM continues for a long time.

[0073] That is, when the display period TM continues for a third time or more, the display control unit 234 may end the display of the guide image GI regardless of the output signal of the inertial measurement device 30.

[0074] According to the second modified example, when the display period TM continues for a third time or more, the display of the guide image GI is terminated regardless of the output signal (the moving speed V of the user U) of the inertial measurement device 30. Thus, the display of unnecessary guide images GI is stopped, and visibility in the visual field of the user U is ensured.

[0075] B3: Third variant In the above-described embodiment, the AR glasses 10 and the portable device 20 are separate entities. This is not limiting, and for example, the AR glasses 10 may have the functions of the portable device 20. That is, the acquisition unit 230, the storage control unit 232, and the display control unit 234 may be executed by the processing device 126 of the AR glasses 10. Also, the storage device 125 of the AR glasses 10 may store the image data D.

[0076] According to the third modified example, for example, the guide image GI can be displayed and the captured image can be recorded without using the mobile device 20.

[0077] B4: Fourth variant In the above-described embodiment, the display control of the guide image GI is performed by the mobile device 20. However, the present invention is not limited to this, and for example, a processing server connected to the mobile device 20 via a network may control the display of the guide image GI. In this case, the mobile device 20 transmits the output signal of the inertial measurement device 30 acquired by the acquisition unit 230 to the processing server. The processing server determines the display period TM of the guide image GI based on the output signal of the inertial measurement device 30, and transmits a control signal for displaying the guide image GI on the AR glasses 10 to the mobile device 20.

[0078] According to the fourth modified example, even if the mobile device 20 does not have a program for realizing the display control unit 234 or does not have the processing power to execute a program for realizing the display control unit 234, monitoring of the monitored object while the user U is working is performed.

[0079] B5: Fifth variant In the above-described embodiment, the imaging device 124 is mounted on the AR glasses 10. However, the present invention is not limited to this, and for example, the AR glasses 10 and the imaging device 124 may be separate devices.

[0080] B6: 6th variant In the above-described embodiment, the inertial measurement device 30 is used to acquire information regarding the movement of the head of the user U. However, the present invention is not limited to this, and a geomagnetic sensor, for example, may be used as a sensor for detecting a physical quantity. The geomagnetic sensor detects the geomagnetism surrounding the earth. The geomagnetic sensor detects the magnetic force values ​​in the three axial directions of X, Y, and Z. The mobile device 20 can estimate the displacement (displacement direction and displacement amount) of the head of the user U based on the change in the magnetic force values ​​in the three axial directions. Furthermore, the displacement amount of the head of the user U per unit time corresponds to the moving speed V of the user U. Therefore, even if a geomagnetic sensor is used, the same processing as in the above-described embodiment can be realized.

[0081] C:Other (1) Each function illustrated in FIG. 4 or FIG. 5 is realized by any combination of hardware and software. There is no particular limitation on the method of realizing each function. Each function may be realized by using one device that is physically or logically combined, or may be realized by using a device that is configured by directly or indirectly connecting (for example, by using wires, wirelessly, etc.) two or more devices that are physically or logically separated. Each function may be realized by combining the one device or the multiple devices with software.

[0082] (2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

[0083] (3) In each of the embodiment and the first to sixth modifications, the storage device 125 and the storage device 205 may be configured by at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Also, the program may be transmitted from a network via an electric communication line.

[0084] (4) Each of the embodiment and the first to sixth modified examples may be implemented using any of the following standards: LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (x is, for example, an integer or a decimal point), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE The present invention may be applied to at least one of systems using 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), and other appropriate systems, and next-generation systems that are expanded, modified, created, or defined based on these systems. The present invention may also be applied to a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A and 5G, etc.).

[0085] (5) The order of the process steps, sequences, or flow charts illustrated in the embodiment and each of the first to sixth modifications may be changed as long as there is no inconsistency. For example, the methods described herein present elements of various steps in an exemplary order, and are not limited to the particular order presented.

[0086] (6) In each of the embodiment and the first to sixth modified examples, the input and output information, etc. may be stored in a specific location (for example, a memory) or may be managed using a management table. The input and output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

[0087] (7) In each of the embodiment and the first to sixth variants, the determination may be made based on a value represented by one bit (0 or 1), based on a Boolean value (true or false), or based on a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0088] (8) The programs exemplified in each of the embodiments and the first to sixth modified examples should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, or functions, regardless of whether they are called software, firmware, middleware, microcode, or hardware description language, or by other names. Software, instructions, or the like, may also be transmitted and received via a transmission medium. For example, when software is transmitted from a website, a server, or other remote source using at least one of wired technology (such as coaxial cable, optical fiber cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared, microwave, etc.), at least one of these wired technology and wireless technology is included within the definition of a transmission medium.

[0089] (9) The information and the like described in each of the embodiments and the first to sixth modifications may be represented using any of a variety of different technologies. For example, data, information, and the like that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms described in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0090] (10) In each of the embodiment and the first to sixth modifications, the terms "system" and "network" are used interchangeably.

[0091] (11) In each of the embodiment and the first to sixth modifications, the portable device 20 or 20B may be a mobile station, which may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable terminology.

[0092] (12) A mobile station may be called a transmitting device, a receiving device, a communication device, or the like. A mobile station may be a device mounted on a mobile body, or the mobile body itself, or the like. A mobile body means an object that can move. The moving speed of a mobile body is arbitrary. A mobile body can be stopped. Examples of mobile bodies include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcars, rickshaws, ships and other watercraft, airplanes, rockets, artificial satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. A mobile body may be a mobile body that travels autonomously based on an operating command. A mobile body may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile body (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

[0093] (13) In each of the embodiment and the first to sixth modifications, the term "determining" may encompass a wide variety of actions. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining something as being "determined", and the like. Also, "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like as being "judgment" or "determined". In addition, "decision" may include the deeming of having "decided" something such as resolving, selecting, choosing, establishing, or comparing. In other words, "decision" may include the deeming of having "decided" to take some kind of action. In addition, "decision" may be interpreted as "assuming," "expecting," "considering," etc.

[0094] (14) In each of the embodiments and the first to sixth modifications, the term "connected" or any variation thereof means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0095] (15) In each of the embodiment and the first to sixth modified examples, the term "based on" does not mean "based only on" unless otherwise specified. In other words, the term "based on" means both "based only on" and "based at least on."

[0096] (16) As used herein, any reference to an element using designations such as "first" and "second" does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0097] (17) When the words "include," "including," and variations thereof in each of the embodiments and the first to sixth modifications are used in the present specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in the present specification or claims is not intended to be an exclusive logical OR.

[0098] (18) Throughout this application, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.

[0099] (19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is for illustrative purposes only and does not have any limiting meaning on the present invention. In addition, multiple aspects selected from the aspects exemplified in this specification may be combined. [Explanation of symbols]

[0100] 1...wearable device, 10...AR glasses, 20...mobile device, 30...inertial measurement unit, 201...touch panel, 203...communication device, 205...storage device, 206...processing device, 207...bus, 230...acquisition unit, 232...memory control unit, 234...display control unit, DV (DV1 to DV3)...device, GI (GI1, GI2)...guide image, IF...imaging range, R...real space, U...user, VF...field of view range.

Claims

1. A transparent display device that is worn on the user's head; an imaging device that outputs an image obtained by imaging an area in front of the user; A sensor for detecting a physical quantity; a control device that causes the display device to display a guide image indicating an outer edge of the captured image during a display period from a start timing to an end timing determined based on at least an output signal of the sensor, the physical quantity being related to movement of the head; the control device calculates a moving speed of the head based on the physical quantity indicated by the output signal, determines a state in which the moving speed exceeds a first threshold as a first state, determines a state in which the moving speed is greater than a second threshold smaller than the first threshold and is equal to or less than the first threshold as a second state, determines a state in which the moving speed is equal to or less than the second threshold as a third state, determines a timing at which the first state transitions to the second state as the start timing, and determines a timing at which the second state transitions to the third state as the end timing. Wearable devices.

2. When a time from the start timing to the end timing is equal to or shorter than a first time, the control device continues displaying the guide image for a second time longer than the first time. The wearable device of claim 1 .

3. the control device terminates the display of the guide image regardless of the output signal when the display period continues for a third time or more. The wearable device of claim 1 .

4. The sensor is an inertial measurement unit or a geomagnetic sensor. The wearable device of claim 1 .

Images