Control apparatus and control method

Abstract

To allow a user to easily operate a transmission type wearable device and an electronic device by a common operation device.SOLUTION: An acquisition unit 111 acquires action information and image information. The action information is information on the action of a user U mounted with MR glasses 10, which shows the intention of the user U to utilize a terminal device 20. The image information is information on an area in front of the user U. When the image of the terminal device 20 is included in images shown by the image information and it is recognized that the user U has an intention of utilizing the terminal device 20 based on the action information, a switching unit 113 switches an operational object operated by an operation device 30 from the MR glasses 10 to the terminal device 20.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a control device and a control method. [Background technology]

[0002] A user wearing a see-through wearable device such as AR glasses on their head may operate an electronic device such as a PC through the see-through wearable device. For example, Patent Document 1 discloses a display system that changes the display mode of a wearable display device depending on the relative positional relationship between the wearable display device and an information processing device based on the attitude of the wearable display device and the attitude of the information processing device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-56371 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional display systems have the problem that the user needs to use both an operation device for operating the transparent wearable device and an operation device for operating the electronic device, making operation cumbersome.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to make it possible to easily operate a see-through wearable device and an electronic device using a common operating device. [Means for solving the problem]

[0006] A preferred embodiment of the control device of the present invention includes an acquisition unit that acquires behavioral information regarding the behavior of a user wearing a transparent wearable device that indicates the user's intention to use a first device, and image information regarding an image of the area in front of the user, and a switching unit that switches the operation target operated by an operation device that can control the transparent wearable device and the first device, respectively, from the transparent wearable device to the first device when the image indicated by the image information includes an image of the first device and it is determined based on the behavioral information that the user has the intention to use the first device.

[0007] A preferred embodiment of the control method of the present invention acquires behavioral information regarding the behavior of a user wearing a transparent wearable device that indicates the user's intention to use a first device, and image information regarding an image of the area in front of the user, and if the image indicated by the image information includes an image of the first device and it is determined based on the behavioral information that the user has the intention to use the first device, switches the operation target operated by an operation device that can control the transparent wearable device and the first device, respectively, from the transparent wearable device to the first device. [Effects of the Invention]

[0008] According to the control device of the present invention, a see-through wearable device and an electronic device can be easily operated using a common operating device. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of a control system including a control device according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a usage situation of the control system of FIG. 1. [Figure 3] FIG. 2 is a perspective view showing the appearance of the MR glasses of FIG. 1. [Figure 4] 10 is an example of an image displayed on the display surface of the MR glasses. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of the MR glasses of FIG. 1. [Figure 6] 2 is a block diagram showing an example of the configuration of the terminal device of FIG. 1. [Figure 7] FIG. 2 is a perspective view showing the appearance of the operating device of FIG. 1. [Figure 8] 2 is a block diagram showing an example of the configuration of the operation device shown in FIG. 1; [Figure 9] FIG. 10 is a diagram showing the longitudinal direction of the operating device at the time of initial setting. [Figure 10] 10 is a diagram showing an example of rotation of the operating device around the direction LV as the rotation axis. FIG. [Figure 11] 10A and 10B are diagrams showing an example of rotation of the operating device around the direction LH as the rotation axis. [Figure 12] FIG. 2 is a plan view showing the positional relationship between the terminal device, MR glasses, and the left and right eyeballs. [Figure 13] 10A and 10B are diagrams illustrating a method for operating a terminal device using an operation device. [Figure 14] 6 is a flowchart showing an example of a switching operation of the processing device of FIG. 5. [Figure 15] 10 is a flowchart showing an example of a switching operation of the processing device according to the first modification. [Figure 16] FIG. 10 is a diagram showing the overall configuration of a control system including a control device according to a second modification. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1. First embodiment Hereinafter, the configuration of a control device according to a first embodiment of the present invention will be described with reference to FIGS.

[0011] 1.1. Configuration of the First Embodiment 1.1.1. Overall structure Fig. 1 is a diagram showing the overall configuration of a control system 1 including a control device according to a first embodiment. Fig. 2 is a diagram showing a usage situation of the control system 1 of Fig. 1. The control system 1 is a system that displays an image obtained by superimposing an auxiliary image visible to a user U as a virtual object on an external world image using AR (Augmented Reality) technology, and assists the user U in operations performed while looking at the displayed image.

[0012] The control system 1 includes MR (Mixed Reality) glasses 10, a terminal device 20, and an operation device 30. A user U holds the operation device 30 and wears the MR glasses 10 on his / her head.

[0013] The MR glasses 10 are a wearable display device worn on the head of a user U. The MR glasses 10 display virtual objects on display panels provided in lenses corresponding to both eyes of the user U. Each lens and display panel of the MR glasses 10 is see-through. Therefore, the user U wearing the MR glasses 10 can view the real space as an external world image through each lens and display panel of the MR glasses 10. The external world image may be a virtual image obtained by capturing an image of the user U's surroundings. A method using a real external world image is called an optical see-through method. A method using a virtual external world image is called a video see-through method. In the following explanation, the MR glasses 10 will be described as employing the optical see-through method. The MR glasses 10 are an example of a see-through wearable device.

[0014] The terminal device 20 includes a personal computer, a tablet terminal, a smartphone, a smart watch, etc. The terminal device 20 is an example of a first device.

[0015] The operation device 30 is held by the user U. The operation device 30 is a device that accepts operations performed by the user U by changing the attitude of the main body of the operation device 30. The operation device 30 can control the MR glasses 10 and the terminal device 20, respectively.

[0016] In the control system 1, the MR glasses 10 and the operation device 30 are connected to each other so as to be able to communicate with each other, and the terminal device 20 and the operation device 30 are connected to each other so as to be able to communicate with each other.

[0017] 1.1.2. Composition of MR glasses 3 is a perspective view showing the appearance of the MR glasses 10 of FIG. 1. The MR glasses 10 include temples 94L and 94R, a bridge 96, projection optical systems 98L and 98R, and a sound output device 18. In the following description, when distinguishing between similar elements, a suffix such as "L" for temple 94L and "R" for temple 94R is used. When not distinguishing between similar elements, only a common number without a suffix is ​​used, such as temple 94.

[0018] The temples 94 are rod-shaped components supported by the pinna. The bridge 96 is disposed between the projection optical system 98L and the projection optical system 98R. The projection optical system 98 includes a display device 17, a light guide path 981, and a half mirror 982. The bridge 96 is provided with an imaging device 15. The imaging device 15 captures an image of the external world in front of the user U.

[0019] The display device 17 is disposed within the temple 94. The display device 17 displays an image. The display device 17 has various display panels, such as a liquid crystal panel and an organic EL (Electro Luminescence) panel. When the display device 17 displays an image, light representing the image is emitted from the display device 17. The light emitted from the display device 17 is guided to the half mirror 982 by a light guide path 981. The half mirror 982 reflects the light guided by the light guide path 981. The light reflected by the half mirror 982 is projected onto the retina of the user U. The user U recognizes the image using this light. The half mirror 982 has a surface facing the user. Hereinafter, this surface will be referred to as the "display surface SC." When the display device 17 does not display an image, the user U can see the outside world through the half mirror 982.

[0020] The sound output device 18 is disposed on the side of the temple 94. The sound output device 18 outputs sound.

[0021] FIG. 4 is an example of an image displayed on the display surface SC of the MR glasses 10. The display surface SC displays an image in which a pointer P1 and a character input image 80 are superimposed on an image of the real outside world. The image displayed on the display surface SC is hereinafter referred to as a "display image." In the display image shown in FIG. 4, the pointer P1 is superimposed on the character input image 80. The pointer P1 and the character input image 80 are virtual objects in the AR space. The display surface SC is a flat or curved surface. The real outside world image includes the terminal device 20. A web browser window is open on the display unit DP of the terminal device 20 with a landscape photo wallpaper.

[0022] The user U can see an image in which the character input image 80 and the pointer P1 are displayed in real space. The position of the character input image 80 recognized by the user U is determined by the shape of the light guide path 981 and the shape of the half mirror 982. In the following, for simplicity of explanation, it is assumed that the display surface SC is flat. The X-axis and Y-axis on the display surface SC are orthogonal to each other, as shown in FIG. 4. The X-axis is parallel to the horizontal direction of the display surface SC, and the Y-axis is parallel to the height direction of the display surface SC.

[0023] The character input image 80 has an input character area 81 and a plurality of buttons 82 for character input. In Fig. 4, some of the plurality of buttons 82 are assigned reference numerals to avoid cluttering the drawing.

[0024] The input character area 81 is an area that displays a character string input by the user U. The button 82 is an image that indicates that a predetermined process corresponding to the button 82 is executed by tapping the button 82. The predetermined process corresponding to the button 82 is, for example, a process of writing "1" in the input character area 81, a process of deleting one character from the character string displayed in the input character area 81, etc. To allow the user U to understand whether to execute the predetermined process, the button 82 has an image or symbol that suggests the content of the predetermined process corresponding to the button 82.

[0025] FIG. 4 shows a situation in which a user attempting to input "123" has input up to "12." FIG. 4 further shows that the pointer P1 is located on the button 82 indicating "3." The position of the pointer P1 is determined according to the attitude of the operation device 30. When the user U performs a tap operation on the operation device 30 while the pointer P1 is located on the button 82 indicating "3," the MR glasses 10 generate image information indicating an image in which "3" is written in the input character area 81 as a predetermined process in accordance with the button 82 indicating "3." The MR glasses 10 display a display image indicated by the generated image information on the display surface SC.

[0026] With the above configuration, by wearing the MR glasses 10 on the head, the user U can observe images displayed on the display panels for the left and right eyes superimposed on the outside world. Furthermore, the MR glasses 10 display, of the binocular images with parallax, the image for the left eye on the display panel for the left eye and the image for the right eye on the display panel for the right eye. This allows the MR glasses 10 to make the user U perceive the displayed images as if they have depth and a three-dimensional effect.

[0027] Fig. 5 is a block diagram showing an example of the configuration of the MR glasses 10 in Fig. 1. The MR glasses 10 include a processing device 11, a storage device 12, a line-of-sight acquisition device 13, a movement detection device 14, an imaging device 15, a communication device 16, a display device 17, and a sound output device 18. The elements of the MR glasses 10 are connected to each other by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, unit, etc.

[0028] The processing device 11 is a processor that controls the entire MR glasses 10, and is configured using, for example, one or more chips. The processing device 11 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 11 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 11 executes various processes in parallel or sequentially.

[0029] The storage device 12 is a recording medium that can be read and written by the processing device 21. The storage device 12 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM).

[0030] The storage device 12 stores a plurality of programs including a control program PR1 to be executed by the processing device 11. The storage device 12 also functions as a work area for the processing device 11.

[0031] The gaze acquisition device 13 acquires the direction in which the user is looking, i.e., the direction of the user's gaze, by eye tracking, which tracks the movement of the user U's left and right eyeballs, and outputs gaze information indicating the direction of the user U's gaze to the processing device 11 based on the acquired results.

[0032] More specifically, the line-of-sight acquisition device 13 includes a pair of light sources and a pair of cameras. Each of the pair of light sources corresponds to the left eye and the right eye of the user U, and each of the pair of cameras corresponds to the left eye and the right eye of the user U. The light source corresponding to the left eye of the user U irradiates infrared light onto the left eye of the user U, and the camera corresponding to the left eye of the user U captures a corneal reflection image of the left eye of the user U as an image formed by reflection of the irradiated infrared light. Similarly, the light source corresponding to the right eye of the user U irradiates infrared light onto the right eye of the user U, and the camera corresponding to the right eye of the user U captures a corneal reflection image of the right eye of the user U.

[0033] The gaze acquisition device 13 detects the positions of the inner corners of the eyes and the irises of the user U from the acquired corneal reflection images of the left eye and the right eye, and acquires gaze information of the user U based on the detected positions of the inner corners of the eyes and the irises of the user U. The gaze information output to the processing device 11 is output to the processing device 11. Note that the method of acquiring the gaze by the gaze acquisition device 13 is not limited to the above method, and any method may be used.

[0034] The motion detection device 14 detects the motion of the MR glasses 10 and outputs the motion data to the processing device 11. The motion data includes acceleration data indicating the acceleration in each of the X-axis, Y-axis, and Z-axis directions, and angular acceleration data indicating the angular acceleration about each of the X-axis, Y-axis, and Z-axis as the center of rotation. The motion detection device 14 includes an acceleration sensor that detects acceleration, an inertial sensor such as a gyro sensor that detects angular acceleration, and a geomagnetic sensor that detects the direction in which the MR glasses 10 are facing.

[0035] The acceleration sensor detects acceleration in each of the mutually orthogonal X-axis, Y-axis, and Z-axis directions. The gyro sensor detects angular acceleration around each of the X-axis, Y-axis, and Z-axis as the center of rotation. The geomagnetic sensor detects geomagnetism in each of the X-axis, Y-axis, and Z-axis directions, and detects the direction in which the MR glasses 10 are facing. The motion detection device 14 generates posture information indicating the posture of the MR glasses 10 based on the output information from the gyro sensor. The motion detection device 14 also outputs motion information related to the movement of the MR glasses 10 to the processing device 11.

[0036] The imaging device 15 outputs imaging information obtained by capturing an image of the external world. The imaging device 15 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 11.

[0037] The communication device 16 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 16 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 16 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 16 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, USB, etc. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0038] The display device 17 is a device that displays images. The display device 17 displays various images under the control of the processing device 11. The display device 17 includes a display panel for the left eye and a display panel for the right eye.

[0039] The sound output device 18 is a device that outputs sound. The sound output device 18 displays various sounds under the control of the processing device 11. The sound output device 18 includes a speaker for the left ear and a speaker for the right ear.

[0040] The processing device 11 functions as an acquisition unit 111, a determination unit 112, a switching unit 113, and a display control unit 114, for example, by reading and executing a control program PR1 from the storage device 12.

[0041] The acquisition unit 111 acquires behavior information and image information.

[0042] The behavior information is information about the behavior of the user U wearing the MR glasses 10, which indicates the user U's intention to use the terminal device 20. The behavior information includes, for example, gaze information and operation information. The gaze information is information about the gaze of the user U. The operation information is information about the operation by the user U.

[0043] The image information is information about an image of a forward region. The forward region is a region in front of the user U. Here, the forward region is a region corresponding to the range of the field of view of the user U. The image of the forward region is captured by the imaging device 15.

[0044] The acquisition unit 111 also acquires gaze information input from the gaze acquisition device 13 , movement information input from the movement detection device 14 , and imaging information input from the imaging device 15 .

[0045] The determination unit 112 analyzes the image information and determines whether or not an image of the terminal device 20 is included in the image indicated by the image information. A known image recognition technique is used to analyze the image information. More specifically, an image of the terminal device 20 is captured in advance and stored as a registered image in the storage device 12. The determination unit 112 refers to the registered image and determines whether or not an image of the terminal device 20 is included in the image indicated by the image information.

[0046] When the image indicated by the image information includes an image of the terminal device 20, the determination unit 112 further determines whether the user U intends to use the terminal device 20 based on the behavior information of the user U. The behavioral patterns of the user U are broadly divided into the following two types.

[0047] A first mode of user U's behavior is a mode in which it is recognized that user U has an intention to use terminal device 20, and in particular, a mode in which it is clear that user U has an intention to use terminal device 20. An example of the first mode is when user U operates a switch or button to select use of terminal device 20. Alternatively, an example of the first mode is when user U utters an intention to use terminal device 20.

[0048] Furthermore, a mode in which the user U conveys his / her intention to use the terminal device 20 using a specific gesture or signal is an example of the first mode. Possible specific gestures include shaking the head quickly or blinking a predetermined number of times. Possible specific signals include shaking the operation device 30 in a specific pattern or tapping the touch sensor 34 in a specific pattern.

[0049] The second behavior of the user U is a behavior in which the user U is recognized as having an intention to use the terminal device 20. For example, if the user U's gaze is directed toward the terminal device 20 for a certain period of time or more, the user U is recognized as having an intention to use the terminal device 20. In other words, if the gaze position, which is the destination of the user U's gaze, overlaps with the terminal device 20 for a certain period of time or more, the user U is recognized as having an intention to use the terminal device 20.

[0050] From the above, behavioral information indicating the behavior of user U is roughly divided into intention expression information and intention correlation information. Intention expression information is information that clearly indicates user U's intention to use terminal device 20, i.e., information that indicates a first aspect. Intention correlation information is information that indicates that user U is recognized as having an intention to use terminal device 20, i.e., information that indicates a second aspect. Intention correlation information can also be said to be information regarding user U's behavior that correlates with the degree of user U's intention to use terminal device 20.

[0051] If the user U's gaze is directed toward the terminal device 20 for a certain period of time, it is determined that the user U intends to use the terminal device 20. Here, the certain period refers to a period of, for example, several seconds. If the certain period is one second or less, even if the user U glances at the terminal device 20, this will be detected, and there is a risk that the operation target will be unintentionally switched. Furthermore, if the certain period is longer than several seconds, it will take time for the user U to switch the operation target, which may cause the user U to feel annoyed. The operation target is a device that is to be operated by the operation device 30.

[0052] Therefore, the determination unit 112 determines whether or not an image of the terminal device 20 is included in the image indicated by the image information and whether or not the line of sight of the user U is directed toward the terminal device 20 for a certain period of time.

[0053] When it is determined that the image indicated by the image information includes an image of the terminal device 20 and that the line of sight of the user U is directed toward the terminal device 20 for a certain period of time, the switching unit 113 switches the operation target of the operation device 30 from the MR glasses 10 to the terminal device 20. In other words, the switching unit 113 instructs the operation device 30 to switch the operation target from the MR glasses 10 to the terminal device 20.

[0054] The display control unit 114 controls the display on the display device 17. In this embodiment, as shown in Fig. 4, the display device 17 is used to display a pointer P1 and a character input image 80 as virtual objects on the display surface SC.

[0055] 1.1.3. Terminal Device Configuration Fig. 6 is a block diagram showing an example of the configuration of the terminal device 20 in Fig. 1. As shown in Fig. 6, the terminal device 20 includes a processing device 21, a storage device 22, a communication device 23, a display device 24, and an input device 25. The elements included in the terminal device 20 are connected to each other by one or more buses for communicating information.

[0056] The processing device 21 is a processor that controls the entire terminal device 20, and is configured using, for example, one or more chips. The processing device 21 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 21 may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 21 executes various processes in parallel or sequentially.

[0057] The storage device 22 is a recording medium that can be read and written by the processing device 21. The storage device 22 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0058] The storage device 22 stores a plurality of programs including a control program PR2 to be executed by the processing device 21. The storage device 22 also functions as a work area for the processing device 21. The control program PR2 is a program that controls the entire processing device 21.

[0059] The communication device 23 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 23 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0060] The display device 24 is a device that displays images and text information. The display device 24 displays various images under the control of the processing device 21. For example, various display panels such as a liquid crystal panel and an organic EL panel are suitably used as the display device 24.

[0061] The input device 25 receives operations from the user U. For example, the input device 25 includes a keyboard, a touchpad, a touch panel, and a pointing device such as a mouse. Here, if the input device 25 includes a touch panel, it may also serve as the display device 24.

[0062] 1.1.4. Configuring the operating device Fig. 7 is a perspective view showing the appearance of the operating device 30 in Fig. 1. Fig. 8 is a block diagram showing an example of the configuration of the operating device 30 in Fig. 1. The operating device 30 includes a processing device 31, a storage device 32, an inertial sensor 33, a touch sensor 34, and a communication device 35. The elements included in the operating device 30 are connected to each other by one or more buses for communicating information.

[0063] The processing device 31 is a processor that controls the entire operation device 30, and is configured using, for example, one or more chips. The processing device 31 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 31 may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 31 executes various processes in parallel or sequentially.

[0064] The storage device 32 is a recording medium that can be read and written by the processing device 31. The storage device 32 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0065] The storage device 32 stores a plurality of programs including a control program PR3 to be executed by the processing device 31. The storage device 32 also functions as a work area for the processing device 31. The control program PR3 is a program that controls the entire processing device 31.

[0066] The inertial sensor 33 measures accelerations in the directions of three axes acting on the operating device 30 in three-dimensional space, and angular velocities acting on the operating device 30 about each of the three axes as a rotation axis. The processing device 31 generates posture information PI based on the measured accelerations in the directions of the three axes and the angular velocities about each of the three axes as a rotation axis. The posture information PI includes an angle θ1 and an angle θ2. The angle θ1 indicates the angle from the position at the time of initial setting of the operating device 30 when the operating device 30 rotates about the direction LV as a rotation axis. The angle θ2 indicates the angle from the position at the time of initial setting of the operating device 30 when the operating device 30 rotates about the direction LH as a rotation axis.

[0067] The touch sensor 34 detects a tap operation, a swipe operation, etc., performed by the user U. When the touch sensor 34 detects a tap operation, the processing device 31 generates tap information TI. The tap information TI is information indicating that a tap operation performed by the user U has been accepted. When the touch sensor 34 detects a swipe operation, the processing device 31 generates swipe information SI. The swipe information SI is information indicating that a swipe operation performed by the user U has been accepted.

[0068] The communication device 35 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 35 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 35 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 35 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0069] 9 to 11, a method for operating the operation device 30 will be described in detail below. Note that the case where the operation target of the operation device 30 is the MR glasses 10 will be described here.

[0070] FIG. 9 is a diagram showing the longitudinal direction of the operating device 30 at the time of initial setting. When the user U performs a long press operation on the touch sensor 34 for several seconds or more, the operating device 30 transitions to an initial setting mode in which the longitudinal direction of the operating device 30 is stored. In the initial setting mode, the operating device 30 identifies the longitudinal direction LIN of the operating device 30 in three-dimensional space. The operating device 30 stores information indicating the identified longitudinal direction LIN. The longitudinal direction LIN of the operating device 30 in three-dimensional space can be identified based on the acceleration applied to the operating device 30. The operating device 30 may include a button, a switch, or the like for transitioning the operation mode to the initial setting mode.

[0071] The operating device 30 also specifies a direction LV and a direction LH. The direction LV is a direction perpendicular to the longitudinal direction LIN. The direction LV is also a height direction of the operating device 30. The direction LH is a direction perpendicular to the longitudinal direction LIN. The direction LH is also a width direction of the operating device 30. After the initial setting mode is completed, the position of the pointer P1 in the X-axis direction of the display surface SC changes as the operating device 30 rotates around the direction LV as the rotation axis. After the initial setting mode is completed, the position of the pointer P1 in the Y-axis direction of the display surface SC changes as the operating device 30 rotates around the direction LH as the rotation axis.

[0072] 10 is a diagram showing an example of rotation of the operating device 30 with the direction LV as the rotation axis. The position of the pointer P1 in the X-axis direction changes according to the angle θ1 by which the operating device 30 is rotated from the position of the operating device 30 at the time of initial setting with the direction LV as the rotation axis. It can also be said that the angle θ1 is the angle between the longitudinal direction LIN of the operating device 30 at the time of initial setting and the current longitudinal direction L1 of the operating device 30 when the operating device 30 is viewed from the direction LV.

[0073] 11 is a diagram showing an example of rotation of the operating device 30 around the direction LH as the rotation axis. The position of the pointer P1 in the Y-axis direction changes according to the angle θ2 by which the operating device 30 is rotated around the direction LH from the position of the operating device 30 at the time of initial setting. It can also be said that the angle θ2 is the angle between the longitudinal direction LIN of the operating device 30 at the time of initial setting and the current longitudinal direction L1 of the operating device 30 when the operating device 30 is viewed from the direction LH. In this way, when the operation target is the MR glasses 10, operation by the operating device 30 is to identify the coordinates of the pointer P1 in the AR space.

[0074] 1.2. Operation of the control device according to the first embodiment 1.2.1. Relationship between user's gaze direction and terminal device position Hereinafter, a method for detecting that the gaze of the user U is directed toward the terminal device 20 will be described with reference to Fig. 12. Fig. 12 is a plan view showing the positional relationship between the terminal device 20, the MR glasses 10, and the left and right eyeballs. In Fig. 12, a flat display is assumed as the terminal device 20. Note that, for simplicity of explanation, Fig. 12 assumes that the main surface of the terminal device 20 and the surface of the MR glasses 10 are parallel to each other.

[0075] 12, the user U is gazing at point S1 on the terminal device 20 through the MR glasses 10. The left eyeball EYL of the user U is directed toward point S1 on the terminal device 20. The line of sight LSL of the left eyeball EYL passes through the half mirror 982L toward point S1. The right eyeball EYR of the user U is directed toward point S1 on the terminal device 20. The line of sight LSR of the right eyeball EYR passes through the half mirror 982R toward point S1.

[0076] The imaging device 15 is disposed between the half mirror 982L and the half mirror 982R. The distance between the imaging device 15 and the left eyeball EYL is dL, and the distance between the imaging device 15 and the right eyeball EYR is dR. In this case, the distance between the left eyeball EYL and the right eyeball EYR is expressed as "dL + dR". The direction of the line of sight LSL of the left eyeball EYL is expressed by the angle αL between the line of sight LSL and the normal NLL of the half mirror 982L. ​​The direction of the line of sight LSR of the right eyeball EYR is expressed by the angle αR between the line of sight LSR and the normal NLR of the half mirror 982R.

[0077] If the distance between the left eyeball EYL and the right eyeball EYR and the terminal device 20 is D, the distance D is expressed by the following equation (1).

[0078] D=(dL+dR) / (tanαL+tanαR) ···(1)

[0079] The determination unit 112 generates a parallax image for the left eye from the image acquired from the imaging device 15, taking into consideration the distance D and the distance dL. The determination unit 112 generates a parallax image for the right eye from the image acquired from the imaging device 15, taking into consideration the distance D and the distance dR. Here, the parallax image for the left eye is an image corresponding to an image of the external world in real space viewed by the user U with his left eye. The parallax image for the right eye is an image corresponding to an image of the external world in real space viewed by the user U with his right eye.

[0080] The generated parallax image for the left eye is associated with an image to be displayed on the display surface SCL for the left eye, and the generated parallax image for the right eye is associated with an image to be displayed on the display surface SCR for the right eye.

[0081] The determination unit 112 simulates an intersection I1 between the display surface SCL and the line of sight LSL on the parallax image for the left eye. When the intersection I1 on the display surface SCL and the terminal device 20 overlap on the line of sight LSL of the user U, the point of gaze, which is the intersection of the line of sight LSL and the terminal device 20, also overlaps with the terminal device 20 on the parallax image for the left eye. Similarly, the determination unit 112 simulates an intersection I2 between the display surface SCR and the line of sight LSR on the parallax image for the right eye. When the intersection I2 on the display surface SCR and the terminal device 20 overlap on the line of sight LSR of the user U, the point of gaze also overlaps with the terminal device 20 on the parallax image for the right eye. This allows the determination unit 112 to detect that the line of sight LSL and LSR of the user U are directed toward the terminal device 20.

[0082] 1.2.2. Method of operating a terminal device using an operating device

[0083] FIG. 13 is a diagram illustrating a method for operating the terminal device 20 using the operation device 30. Here, the operation target of the operation device 30 is the terminal device 20. In FIG. 13, the horizontal direction of the display of the terminal device 20 is the X-axis direction, and the height direction of the display is the Y-axis direction. When the operation device 30 receives an instruction from the MR glasses 10 to switch the operation target from the MR glasses 10 to the terminal device 20, the operation device 30 switches the operation target from the MR glasses 10 to the terminal device 20. An application program suitable for operation by the operation device 30 is pre-installed in the terminal device 20.

[0084] 9 to 11, the operating device 30 changes the position of the pointer P2 in the X-axis direction and the position of the pointer P2 in the Y-axis direction according to the angles θ1 and θ2 by which the operating device 30 has rotated from the position of the operating device 30 at the time of initial setting. In this way, when the operation target is the terminal device 20, the operation by the operating device 30 is to specify the coordinates of the pointer P2 on the display.

[0085] 1.2.3. Operation of the Processing Unit Fig. 14 is a flowchart showing an example of the switching operation of the processing device 11 in Fig. 5. The switching operation of the processing device 11 will be described below with reference to Fig. 14. The routine in Fig. 14 is started, for example, when the processing device 11 is started, and is executed every time a certain time period has elapsed.

[0086] In step S11, the processing device 11 functions as the acquisition unit 111 to acquire the behavior information and image information of the user U.

[0087] In step S12, the processing device 11 functions as the determination unit 112 to determine whether or not an image of the terminal device 20 is included in the image indicated by the image information.

[0088] If the image indicated by the image information does not include an image of the terminal device 20, that is, if the determination result in step S12 is negative, the processing device 11 temporarily ends this routine.

[0089] On the other hand, if the image indicated by the image information includes an image of the terminal device 20, that is, if the judgment result in step S12 is positive, the processing device 11 functions as a judgment unit 112 and judges in step S13 whether or not it is recognized that the user U intends to use the terminal device 20.

[0090] If it is not recognized that the user U intends to use the terminal device 20, that is, if the determination result in step S13 is negative, the processing device 11 temporarily ends this routine.

[0091] On the other hand, if it is determined that the user U intends to use the terminal device, that is, if the judgment result in step S13 is positive, the processing device 11 functions as a switching unit 113, switches the operation object from the MR glasses 10 to the terminal device 20, and temporarily terminates this routine.

[0092] 1.3. Advantages of the First Embodiment According to the above description, the control device according to the first embodiment includes an acquisition unit 111 and a switching unit 113. The acquisition unit 111 acquires behavioral information and image information. The behavioral information is information about the behavior of the user U wearing the MR glasses 10, which indicates the user U's intention to use the terminal device 20. The image information is information about the area in front of the user U. When an image indicated by the image information includes an image of the terminal device 20 and it is determined based on the behavioral information that the user U intends to use the terminal device 20, the switching unit 113 switches the operation target operated by the operation device 30 from the MR glasses 10 to the terminal device 20. The operation device 30 is capable of controlling both the MR glasses 10 and the terminal device 20.

[0093] According to this aspect, when the terminal device 20 is present in front of the user U and it is determined that the user U intends to use the terminal device 20 based on the behavioral information, the control device switches the operation target from the MR glasses 10 to the terminal device 20. Therefore, the user U's intention is more accurately reflected in the switching of the operation target from the MR glasses 10 to the terminal device 20. As a result, the user U can easily operate the MR glasses 10 and the terminal device 20 using the common operation device 30. In other words, the user U can use the operation device 30 to specify the coordinates of the pointer P1 in the AR space and the coordinates of the pointer P2 on the display in real space.

[0094] The behavior information is information relating to the user U's line of sight.

[0095] According to this aspect, when the user U wants to use the terminal device 20, the user U can switch the operation target to the terminal device 20 by directing his / her gaze toward the terminal device 20.

[0096] Furthermore, a case where it is determined that the user U intends to use the terminal device 20 based on the behavior information is a case where the user U's line of sight is directed toward the terminal device 20 for a certain period of time.

[0097] According to this aspect, it is possible to prevent the operation target from being unintentionally switched.

[0098] The operation device 30 is held by the user U. The operation device 30 accepts operations performed by the user U by changing the orientation of the main body.

[0099] According to this embodiment, the user U can intuitively operate the pointer P1 or P2 simply by changing the orientation of the operation device 30.

[0100] In addition, the control method of the first embodiment acquires behavioral information and image information, and if the image indicated by the image information includes an image of the terminal device 20 and it is determined based on the behavioral information that the user U intends to use the terminal device 20, the operation target operated by the operation device 30 is switched from the MR glasses 10 to the terminal device 20.

[0101] According to this aspect, when the terminal device 20 is present in front of the user U and it is determined that the user U intends to use the terminal device 20 based on the behavioral information, the operation target is switched from the MR glasses 10 to the terminal device 20. The intention of the user U is more accurately reflected in the switching of the operation target from the MR glasses 10 to the terminal device 20. As a result, the user U can easily operate the MR glasses 10 and the terminal device 20 using the common operation device 30. That is, the user U can use the operation device 30 to specify the coordinates of the pointer P1 in the AR space and the coordinates of the pointer P2 on the display in real space.

[0102] 2. Variations The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples may be combined. Furthermore, the above-described exemplary embodiments and the following modified embodiments may be combined in any combination as long as they are not mutually inconsistent.

[0103] 2.1. Variation 1 In the first embodiment, gaze information is used as the behavioral information. However, the behavioral information may be, for example, information about a position pointed to by the user U with the pointer P1 on the display surface SC of the operation device 30. Hereinafter, information about a position pointed to by the user U with the pointer P1 on the display surface SC of the operation device 30 will be referred to as "pointer information." For example, if the pointer P1 and the terminal device 20 overlap in the user U's gaze for a certain period of time, it is determined that the user U intends to use the terminal device 20. Therefore, the determination unit 112 may simulate the position of the pointer P1 instead of simulating the user U's gaze on the parallax image.

[0104] The determination unit 112 simulates the position of the pointer P1L on the display surface SCL on the parallax image for the left eye. When the pointer P1L on the display surface SCL and the terminal device 20 overlap on the line of sight LSL of the user U, the pointer P1R also overlaps on the terminal device 20 on the parallax image for the left eye. Similarly, the determination unit 112 simulates the position of the pointer P1R on the display surface SCR on the parallax image for the right eye. When the pointer P1R on the display surface SCR and the terminal device 20 overlap on the line of sight LSR of the user U, the pointer P1R also overlaps on the terminal device 20 on the parallax image for the right eye. This allows the determination unit 112 to detect that the pointer P1L and the terminal device 20 overlap on the line of sight LSL of the user U and that the pointer P1R and the terminal device 20 overlap on the line of sight LSR of the user U.

[0105] Fig. 15 is a flowchart showing an example of the switching operation of the processing device 11 according to Modification 1. The switching operation of the processing device 11 will be described below with reference to Fig. 15. The routine of Fig. 15 is started, for example, when the processing device 11 is started, and is executed every time a certain time period has elapsed.

[0106] In step S11, the processing device 11 functions as the acquisition unit 111 to acquire behavioral information and image information.

[0107] In step S12, the processing device 11 functions as the determination unit 112 to determine whether or not an image of the terminal device 20 is included in the image indicated by the image information.

[0108] If the image indicated by the image information does not include an image of the terminal device 20, that is, if the determination result in step S12 is negative, the processing device 11 temporarily ends this routine.

[0109] On the other hand, if the image indicated by the image information includes an image of the terminal device 20, that is, if the determination result in step S12 is positive, the processing device 11 functions as the determination unit 112 and determines in step S21 whether or not the pointer P1 and the terminal device 20 overlap in the line of sight of the user U for a certain period of time. The certain period of time is set to, for example, about 1 to 3 seconds.

[0110] If the pointer P1 and the terminal device 20 do not overlap in the line of sight of the user U for a certain period of time, that is, if the determination result in step S21 is negative, the processing device 11 temporarily ends this routine.

[0111] On the other hand, if the pointer P1 and the terminal device 20 overlap in the user U's line of sight for a certain period of time, i.e., if the judgment result in step S21 is positive, the processing device 11 functions as a switching unit 113, switches the operation object from the MR glasses 10 to the terminal device 20, and temporarily terminates this routine.

[0112] As described above, the behavior information is pointer information related to the position pointed to by the user U with the pointer P1 on the display surface SC of the MR glasses 10.

[0113] According to this embodiment, when the user U wishes to use the terminal device 20, the user U can switch the operation target to the terminal device 20 simply by placing the pointer P1 over the image of the terminal device 20.

[0114] Furthermore, a case where it is determined that the user U intends to use the terminal device 20 based on the behavior information is a case where the pointer P1 and the terminal device 20 overlap in the user U's line of sight for a certain period of time.

[0115] According to this aspect, it is possible to prevent the operation target from being unintentionally switched.

[0116] 2.2. Variation 2 In the first embodiment, the image information is analyzed by referring to a registered image captured in advance. However, the analysis of the image information is not limited to this method. For example, the image information may be analyzed using a learning model that has learned the relationship between images of multiple terminal devices and the types of each of the multiple terminal devices.

[0117] FIG. 16 is a diagram showing the overall configuration of a control system 1A including a control device according to Modification 2. The MR glasses 10 are connected to a learning server 40 via a communication network NET so that they can communicate with each other. The learning server 40 stores a trained learning model. The acquisition unit 111 analyzes image information using the learning model of the learning server 40.

[0118] More specifically, when the acquired image information is input to the learning model, if the image indicated by the image information includes an image of the terminal device 20, the learning model identifies the type of the terminal device 20 and the position of the terminal device 20 on the image. The learning server 40 transmits terminal information related to the identified type and position to the MR glasses 10. If the image indicated by the image information does not include an image of the terminal device 20, the learning model outputs a message indicating that the terminal device 20 is not included in the image.

[0119] According to this aspect, since the learning model identifies various types of terminal devices 20, there is no need to take an image of the terminal device 20 in advance and prepare it as a registered image. This increases convenience for the user U. Note that, when a communication network NET is used, communication between the MR glasses 10 and the operation device 30 or communication between the terminal device 20 and the operation device 30 may be performed via the communication network NET.

[0120] 2.3. Variation 3 The operation device 30 is not limited to a device that operates the pointer P1 or P2 by changing the orientation of the main body. The operation device 30 may also be a device that specifies the destination position of the pointer P1 or P2. For example, the operation device that specifies the pointer position displays a virtual ray of light that is output from the main body in the direction pointed by the user U. The user U clicks a button on the operation device while aiming the virtual ray projected on the display surface SC of the MR glasses 10 at the position to which the pointer P1 is to be moved. As a result, the position where the virtual ray of light hits is specified as the destination of the pointer P1, and the pointer moves to that position.

[0121] When the operation target is switched from the MR glasses 10 to the terminal device 20, the user U clicks a button on the operation device while directing the virtual light beam projected on the display surface SC of the MR glasses 10 to the image on the display device 24 of the terminal device 20. This causes the pointer P2 to move to the corresponding position on the display device 24.

[0122] The operation device is not limited to the devices disclosed above, but may be a smartphone, a smart watch, or an input device such as a mouse or a trackball.

[0123] 2.4. Variation 4 The terminal device 20 is not limited to a personal computer, a tablet terminal, a smartphone, a smartwatch, etc. It may also include home appliances such as a television, an air conditioner, a microwave oven, etc. that have communication functions. For example, when the operation target is switched from the MR glasses 10 to the air conditioner, information about the air conditioner's operation menu may be transmitted from the air conditioner to the MR glasses 10 via the operation device 30, and the air conditioner operation menu may be displayed on the display surface SC. In this embodiment, the desired operation can be selected from the air conditioner operation menu displayed on the display surface SC using the operation device 30.

[0124] 2.5. Variation 5 In the first embodiment, the optical see-through system is adopted for the MR glasses 10, but a video see-through system may also be adopted.

[0125] 2.6. Variation 6 In the first embodiment, the control device is included in the processing device 11 of the MR glasses 10, but the control device may be included in the operation device 30.

[0126] 3.Other (1) In the above-described embodiment, storage device 12, storage device 22, and storage device 32 are exemplified by ROM and RAM, but may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0127] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0128] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0129] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0130] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0131] (6) Each function illustrated in Figures 1 to 16 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0132] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0133] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0134] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0135] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0136] (10) In the above-described embodiments, the MR glasses 10 and the learning server 40 may be a mobile station (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term. In the present disclosure, the terms "mobile station," "user terminal," "user equipment (UE)," "terminal," etc. may be used interchangeably.

[0137] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0138] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0139] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0140] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0141] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0142] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0143] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0144] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0145] 10...MR glasses, 20...terminal device, 30...operation device, 111...acquisition unit, 113...switching unit, LSL, LSR...line of sight, P1...pointer, SC, SCL, SCR...display surface, U...user.

Claims

1. an acquisition unit that acquires behavior information regarding a behavior of a user wearing a see-through wearable device that indicates an intention of the user to use a first device, and image information regarding an image of an area in front of the user; a switching unit that switches an operation target operated by an operation device capable of controlling the transparent wearable device and the first device from the transparent wearable device to the first device when an image of the first device is included in the image indicated by the image information and when it is determined that the user has an intention to use the first device based on the behavior information; and Equipped with Control device.

2. The behavioral information is information about the user's line of sight. The control device according to claim 1 .

3. A case where it is determined based on the behavior information that the user has an intention to use the first device is a case where the user's line of sight is directed toward the first device for a certain period of time. The control device according to claim 2 .

4. The behavioral information is pointer information regarding a position pointed by the user with a pointer on a display surface of the see-through wearable device. The control device according to claim 1 .

5. A case where it is determined based on the behavior information that the user has an intention to use the first device is a case where the pointer and the first device overlap in the user's line of sight for a certain period of time. The control device according to claim 4.

6. the operation device is held by the user and receives an operation performed by the user by changing the posture of a main body of the operation device; The control device according to claim 1 .

7. acquiring behavioral information regarding a behavior of a user wearing a see-through wearable device that indicates an intention of the user to use a first device, and image information regarding an image of an area in front of the user; When an image of the first device is included in the image indicated by the image information and it is determined that the user intends to use the first device based on the behavioral information, an operation target operated by an operation device capable of controlling the transparent wearable device and the first device is switched from the transparent wearable device to the first device. Control method.

Images