Information processing device

Abstract

To provide a technique capable of performing operations while reducing false recognitions and malfunctions of other information processing devices by user's operations for one information processing device.SOLUTION: An information processing device comprises a communication unit which communicates with an external device, a visual line detection unit which detects a visual line direction of a user of the information processing device, and a control unit. The control unit is configured to: generate a first operation command for controlling an operation of the information processing device; control the operation of the information processing device according to the generated first operation command; determine whether the visual line direction detected by the visual line detection unit is directed toward the external device; and control the information processing device according to the received second operation command when receiving a second operation command from the external device via the communication unit after both the determination that the visual line direction is directed toward the external device and the receipt of a detection notification indicative of the detection of a cooperation trigger operation transmitted from the external device have occurred within a predetermined period.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a technique for operating one information processing device by a user's action on another information processing device. [Background technology]

[0002] One example of an information processing device is a smart watch worn on the wrist. A smart watch is worn on the wrist and has various functions such as timekeeping and activity measurement. There is also technology that acquires myoelectric information, detects the user's movements (gestures), and operates the smart watch based on the detected gestures (see, for example, Patent Document 1).

[0003] Using this technology, there is a technology that links a head-mounted display (HMD) with a smartwatch and allows the HMD to be operated using gestures detected by the smartwatch. An HMD is a type of information processing device that primarily has a display function and is worn on the user's head to display and visually recognize information from real and virtual spaces. This eliminates the need to make gestures in front of the user, as is the case when gestures are detected using an HMD camera, reducing the user's embarrassment and the sense of incongruity felt by others.

[0004] In this case, it is necessary to switch between a mode in which gestures detected by the smartwatch are used as operational instructions for the smartwatch and a mode in which they are used as operational instructions for the HMD. However, hands are constantly moving, and there is a limit to how accurate electromyographic information can be. For this reason, attempting to switch modes using electromyographic information alone results in many false recognitions and malfunctions. On the other hand, using actions such as touching the smartwatch screen or pushing the crown to switch modes reduces false recognitions and malfunctions. However, this requires using the other hand, and the advantage of being able to operate the device with one hand using electromyographic information is not fully utilized.

[0005] As an example of switching the operation mode to indicate which of multiple information processing devices such as a smart watch or an HMD an operation instruction is directed to, Patent Document 2 describes "an information processing device worn on a first part of a user, comprising: a sensor that outputs a first signal in response to a movement of the user; a mode setting unit that can set the information processing device to a first mode in which the first signal is used to calculate information related to the movement; and a second mode in which the first signal is used as an input signal to a display device worn on a second part of the user; and a transmission unit that, in the second mode, transmits either a second signal generated based on the first signal or the first signal to the display device." In other words, Patent Document 2 describes a technology for switching the operation mode to indicate whether the information processing device or the display device is operated based on the operation of the information processing device. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 10,671,176 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-85554 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 2, the switching of the operation mode, i.e., whether to operate the information processing device or the display device, is determined solely by the operation of the information processing device, so there is a limit to the accuracy of determining the switching of the operation mode. It is difficult to reliably switch the operation mode with higher accuracy, and there is a high possibility of misrecognition and malfunction.

[0008] The present invention has been made to solve the above-mentioned problems, and aims to provide a technology that allows a user to operate another information processing device by performing an action on one information processing device while reducing erroneous recognition and malfunction. [Means for solving the problem]

[0009] The present invention is an information processing device comprising a communication unit that communicates with an external device, a gaze detection unit that detects the gaze direction of a user of the information processing device, and a control unit, wherein the control unit generates a first operation command that controls the operation of the information processing device, controls the operation of the information processing device in accordance with the generated first operation command, determines whether the gaze direction detected by the gaze detection unit is in the direction of the external device, and, if the determination that the gaze direction is in the direction of the external device and the reception of a detection notification indicating the detection of a linked trigger operation sent from the external device occur within a predetermined period, when a second operation command is received from the external device via the communication unit, controls the operation of the information processing device in accordance with the received second operation command.

[0010] The present invention also provides an information processing device comprising: a communication unit that communicates with an external device; an action detection unit that detects actions of a user of the information processing device; a memory unit that stores trigger actions; and a control unit, wherein the control unit determines whether the user's action detected by the action detection unit matches the trigger action stored in the memory unit; and, if it is determined that the user's action does not match the trigger action, generates a first operation command that controls the operation of the information processing device in accordance with the user's action, and controls the operation of the information processing device in accordance with the generated first operation command; if it is determined that the user's action matches the trigger action, determines whether the determination of the match and the receipt of a detection notification indicating the detection of a linked trigger action sent from the external device occur within a predetermined period; and, if it is determined that the determination of the match and the receipt of the detection notification occur within the predetermined period, receives a second operation command sent from the external device via the communication unit, and controls the operation of the information processing device in accordance with the received second operation command.

[0011] Furthermore, in the above invention, the device may further include a display unit, and the control unit may display an image on the display unit indicating whether the operation of the information processing device is being controlled in accordance with the second operation command. Also, in the above invention, the device may further include a display unit, and when the control unit is controlling the operation of the information processing device in accordance with the second operation command, the control unit may display an image corresponding to the second operation command on the display unit. Furthermore, in the above invention, the information processing device may be a device worn on the user's head. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a technology that allows a user to operate one information processing device while reducing erroneous recognition and malfunction of another information processing device. Furthermore, problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is an explanatory diagram illustrating an overview of an information processing system according to an embodiment of the present invention; [Figure 2] 1 is a diagram illustrating an example of an outline of an operation of an information processing system according to an embodiment of the present invention; [Figure 3] 1A and 1B are a hardware configuration diagram and a functional block diagram, respectively, of a first information processing device (watch) according to an embodiment of the present invention. [Figure 4] 1A and 1B are explanatory diagrams illustrating an example of a watch stand-alone operation database and a cooperative operation database, respectively, according to an embodiment of the present invention. [Figure 5] 1A and 1B are a hardware configuration diagram and a functional block diagram, respectively, of a second information processing device (HMD) according to an embodiment of the present invention. [Figure 6] FIG. 2 is an explanatory diagram illustrating an example of an HMD standalone operation database according to an embodiment of the present invention. [Figure 7]10 is a flowchart of a linking process according to an embodiment of the present invention. [Figure 8] 1A and 1B are explanatory diagrams illustrating a display example on a display device of an HMD according to an embodiment of the present invention. [Figure 9] FIG. 10 is an explanatory diagram illustrating an overview of an information processing system according to a modified example of the present invention. [Figure 10] FIG. 10 is an explanatory diagram for explaining two-party cooperation processing and three-party cooperation processing according to a modified example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] In an embodiment of the present invention, a combination of trigger operations in multiple information processing devices enables highly accurate switching of operation modes. By enabling highly accurate switching of operation modes, we contribute to achieving the 9th Sustainable Development Goal (SDG) advocated by the United Nations: Build resilient infrastructure, promote inclusive and sustainable industrialization, and foster innovation.

[0015] An example of an embodiment of the present invention will be described below with reference to the drawings. The information processing system of this embodiment includes an information processing device (first information processing device) and an external device (second information processing device) that can communicate with the information processing device and is pre-communicatively linked. When a predetermined trigger action is detected at approximately the same time in the first information processing device and the second information processing device, both devices transition to a linked operation mode. In the linked operation mode, the second information processing device (operated device) is operated using the action detected by the first information processing device (operating device).

[0016] Hereinafter, an example will be described in which the first information processing device (operating device) is a smart watch and the second information processing device (operated device) is a head mounted display.

[0017] A smartwatch is an information processing device worn on the wrist in the same manner as a wristwatch. Hereinafter, in this embodiment, it will be referred to as a watch. A head-mounted display is an information processing device worn on the head of a user to display and visually recognize information from real space or virtual space. Hereinafter, in this embodiment, it will be referred to as an HMD.

[0018] In addition to its timekeeping function, the watch also has functions such as answering calls, receiving messages, and communicating with other devices, and also functions as an activity monitor that measures the user's activity level (amount of exercise).The watch can also acquire electromyographic information using an electromyographic sensor that detects bioelectric potentials.Furthermore, the watch can detect the hand movements (gestures) of the user wearing the watch using the electromyographic information acquired by the electromyographic sensor when the wrist moves.

[0019] In addition to displaying real-world and virtual-world information, HMDs can also detect the user's movements, posture, and line of sight using various sensors.

[0020] First, an overview of an information processing system 900 according to this embodiment will be described. Fig. 1 is a diagram for explaining the overview of the information processing system 900 according to this embodiment.

[0021] The information processing system 900 of this embodiment includes a watch 100 and an HMD 200. The watch 100 is worn on the wrist of a user 800. The HMD 200 is worn on the head of the user 800. The watch 100 and the HMD 200 are communicatively linked in advance.

[0022] The watch 100 includes an electromyographic sensor 131. The HMD 200 also includes a left eye gaze sensor 235 and a right eye gaze sensor 236 that detect the gaze of the left and right eyes of the user 800, an acceleration sensor 232, a gyro sensor 233, and a geomagnetic sensor 234 that detect the posture and movement of the user 800, and a camera 210 that captures images of the outside world, etc.

[0023] The watch 100 of this embodiment has two operation modes: a watch-only operation mode and an HMD-linked operation mode.

[0024] In the watch-standalone operation mode, operation commands for operating the watch 100 are generated in the watch 100, and the watch 100 is operated according to the generated operation commands. That is, in the watch-standalone operation mode, the watch 100 detects instructions by movements or inputs from the user 800, and operates the watch 100 according to the detection results.

[0025] In addition, in the HMD cooperative operation mode, the watch 100 generates operation commands for operating the HMD 200 and transmits them to the HMD 200. That is, in the HMD cooperative operation mode, the watch 100 detects instructions by movements or inputs of the user 800, generates operation commands for the HMD 200 in accordance with the detection results, and transmits them to the HMD 200.

[0026] The HMD 200 of this embodiment also has an HMD-standalone operation mode and a watch-linked operation mode as operation modes. In the HMD-standalone operation mode, the HMD 200 detects instructions by movements or inputs of the user 800 and operates the HMD 200 according to the detection results. In the watch-linked operation mode, the HMD 200 receives operation commands from the watch 100 and operates the HMD 200 according to the operation commands.

[0027] In the initial state (immediately after startup), the watch 100 and the HMD 200 operate in a watch-only operation mode and an HMD-only operation mode, respectively. When the watch 100 detects a predetermined trigger action and, at the same time or within a predetermined time, the HMD 200 also detects a predetermined trigger action, the watch 100 and the HMD 200 transition to an HMD-linked operation mode and a watch-linked operation mode, respectively. In the HMD-linked operation mode and the watch-linked operation mode, the watch 100 transmits an operation command corresponding to the user action detected by the watch 100 to the HMD 200 as operation information for operating the HMD 200, as shown in FIG. 2 .

[0028] In the following, when there is no need to distinguish between the watch standalone operation mode and the HMD standalone operation mode, they will be referred to as the standalone operation mode. Also, when there is no need to distinguish between the HMD linked operation mode and the watch linked operation mode, they will be referred to as the linked operation mode.

[0029] Furthermore, the actions and input instructions by the user 800 detected by the watch 100 and the HMD 200 are collectively referred to as "actions." Of these, an action that triggers switching the operation mode from standalone operation mode to standalone operation mode is referred to as a collaboration start trigger action, and an action that triggers switching from standalone operation mode to standalone operation mode is referred to as a collaboration end trigger action. Note that when there is no need to distinguish between the collaboration start trigger action and the collaboration end trigger action, they are simply referred to as trigger actions.

[0030] Note that the cooperation start trigger action for the watch 100 and the cooperation start trigger action for the HMD 200 may be the same or different. In the following, this embodiment will be described taking as an example a case where the two are different. In the following, this embodiment will be described taking as an example a case where it is necessary to distinguish between the two, a cooperation start trigger action for the watch 100 will be referred to as a watch-side cooperation start trigger action, and a cooperation start trigger action for the HMD 200 will be referred to as an HMD-side cooperation start trigger action (external cooperation start trigger action). Similarly, a cooperation end trigger action for the watch 100 and a cooperation end trigger action for the HMD 200 may be the same or different. In the following, this embodiment will be described taking as an example a case where the two are different (referred to as a watch-side cooperation end action and an HMD-side cooperation end action, respectively).

[0031] 1 shows an example of a collaboration start trigger action on the watch 100 side, in which the user 800 lifts and twists the wrist wearing the watch 100. This is a typical action that the user 800 takes when they want to look at the watch 100. The watch 100 uses the electromyographic sensor 131 to detect as a collaboration start trigger action a series of gesture actions in which the wrist wearing the watch 100 moves from a state 802 in which the wrist is down, to a state 803 in which the arm is lifted and the wrist is raised, and then to a state 804 in which the wrist is twisted.

[0032] 1 also shows an example of a collaboration start trigger action on the HMD 200 side, in which the user 800 momentarily closes their eyes. The HMD 200 detects, using the left eye gaze sensor 235 and right eye gaze sensor 236, a momentary change in the user 800's eyes from an open state 811 to an eye closed state 812, and detects this as a collaboration start trigger action. Note that collaboration start trigger actions are not limited to these. For example, as described above, the same collaboration start trigger action on the watch 100 side, that is, lifting and twisting the wrist wearing the watch 100, may also be used as a collaboration start trigger action on the HMD side.

[0033] Furthermore, the trigger action for ending cooperation on the watch 100 side is, for example, a movement of the hand on the side that is not wearing the watch 100. Specifically, this is an action such as touching the screen of the watch 100 or pressing the crown. This is because if cancellation (instruction to end cooperation) is performed by moving the arm wearing the watch 100, there is a high possibility that it will be mistaken for an action for operating the HMD 200.

[0034] On the other hand, the trigger action for ending cooperation on the HMD 200 side is, for example, a movement of the hand on the side not wearing the watch 100. Alternatively, a movement of a part of the body other than the hand is used. Specifically, this can be gazing at the screen of the watch 100, touching one of the sensors 130 of the HMD 200 with the hand not wearing the watch 100, or making a gesture that is not normally made with the hand not wearing the watch 100 (such as covering one eye). This is also to avoid false detection by the watch 100.

[0035] Alternatively, the collaboration end trigger action may be when the user 800, who had been looking at something else, begins to look at the watch 100 again. In this case, the HMD 200 detects by line of sight that the user 800 has started looking at the watch 100 again. When the user 800 looks at the watch 100, it is assumed that the user 800 intends to return to watch standalone operation mode in order to make the watch 100 perform some operation. By using such an action as the collaboration end trigger action, processing that easily captures this user intention can be realized. Of course, the collaboration end trigger action is not limited to this.

[0036] Next, the watch 100 and HMD 200 that realize the above-described functions of this embodiment will be described.

[0037] [watch] [Hardware configuration] The hardware configuration of the watch 100 of this embodiment is shown in Figure 3(a). Note that in this figure, parts with the same reference numerals as those shown in Figures 1 and 2 have the same operations as those already explained in those figures, so detailed explanations of those parts will be omitted.

[0038] As shown in this figure, the watch 100 of this embodiment includes a processor 101, a memory 102, a camera 111, a sensor 130, a display device 121, a second input interface (I / F) 123, an audio input device 124, an audio output device 125, a vibration generating device 126, and a communication device 127.

[0039] The sensors 130 include an electromyographic sensor 131, an acceleration sensor 132, a gyro sensor 133, and a geomagnetic sensor 134. The display device 121 also includes a first input I / F 122.

[0040] These components are connected to each other via a bus 103 .

[0041] The electromyographic sensor 131 is disposed on the housing of the watch 100, the back of the wristband, or the like, and detects the bioelectric potential flowing near the wrist at the contact point on the wrist surface. The electromyographic sensor 131 detects the bioelectric potential flowing when the wrist makes a specific movement, and acquires the electromyographic information to capture the wrist gesture. Specifically, the electromyographic sensor 131 is a sensor that picks up and processes nerve pulse signals (biological signals) flowing through the radial nerve, median nerve, and ulnar nerve, motor nerves that move the muscles of the hand, passing through the wrist. The two sensors arranged horizontally pick up nerve pulse signals flowing through the motor nerves when the hand moves. For example, when a thumb gesture is made, biosignals for moving the thumb flow from the brain to the radial nerve and, depending on the direction of movement, the median nerve, and the electromyographic sensor 131 picks up these signals to detect the gesture of the thumb moving.

[0042] The acceleration sensor 132 is a sensor that detects acceleration, which is a change in speed per unit time, and captures movement, vibration, shock, etc. When the only acceleration applied is gravity, the acceleration sensor 132 calculates the tilt angle using the gravity vector and its projection on the axis of the acceleration sensor 132, and measures and detects how much the object is tilted with respect to the ground.

[0043] The gyro sensor 133 is a sensor that detects angular velocity in the direction of rotation, and captures the vertical, horizontal, and diagonal posture states to measure and detect how much movement has occurred in each direction.

[0044] Therefore, the acceleration sensor 132 and the gyro sensor 133 can be used to detect the attitude of the watch 100, such as its tilt and direction.

[0045] The geomagnetic sensor 134 is a sensor that detects the magnetic force of the Earth and detects the direction in which the watch 100 is facing. A three-axis type is used that detects geomagnetic fields in the up and down directions in addition to the front-to-back and left-to-right directions, and by capturing changes in the geomagnetic field in response to the movement of the watch 100, the movement of the watch 100 can also be detected.

[0046] The acceleration sensor 132, gyro sensor 133, and geomagnetic sensor 134 can detect the posture and movement of the user 800 wearing the watch 100. For example, they may be used in place of the electromyographic sensor 131 for movement detection in the processing described below. They may also be used in conjunction with movement detection by the electromyographic sensor 131. Using them in combination enables even more accurate detection.

[0047] The camera 111 is used to capture images of the surroundings and recognize the actions of the user 800. For example, the action of looking at the watch 100, which was explained as a trigger action in FIG. 1, can be recognized and identified from the images captured by the camera 111.

[0048] The processor 101 is composed of a CPU (Central Processing Unit) and other components, and executes programs such as an operating system (OS) and operation control applications stored in memory 102 to realize the various functions of the watch 100. The functions that are realized will be described later.

[0049] The memory 102 is configured with a nonvolatile storage device or the like, and functions as a storage unit that stores various programs and data handled by the processor 101, etc. The stored data includes data used by the watch 100 for processing, data generated during processing, etc. Specifically, the data includes motion data detected by the watch 100, various predetermined trigger motion data, start trigger information to be transmitted to the HMD 200 when a trigger motion is detected, operation commands, operation commands for the HMD 200, etc.

[0050] The display device 121 displays the time, an application image, notification information for the user 800, etc. The notification information for the user 800 includes information indicating that a trigger action has been detected, information indicating the start or end of finger tapping measurement on the display device 121 and measurement of changes in the state of the head and neck region, information indicating that a switching request has been received, information indicating the start or end of smartwatch operation, etc.

[0051] The first input I / F 122 accepts input information from the user 800 to the watch 100. The first input I / F 122 is provided on the display surface of the display device 121, and detects touch operations using a finger, a touch pen, or the like, and accepts the input.

[0052] The second input I / F 123 is a crown, switch, or the like provided on the side or other portion of the watch 100. The user 800 can input information to the watch 100 by operating these crowns or switches.

[0053] The voice input device 124 collects the user's 800 own speech using a microphone and converts it into voice data. Instruction information based on the speech from the user 800 can be captured in the watch 100, and an operation corresponding to the instruction information can be easily performed. For example, the user 800 may vocalize a voice indicating an input operation, and the voice may be collected by the voice input device 124 to capture the input information.

[0054] The audio output device 125 outputs audio from a speaker based on the audio data, and can notify the user 800 of notification information by audio.

[0055] The vibration generating device 126 generates vibrations under the control of the processor 101. For example, the processor 101 converts notification information for the user 800 into vibrations, and the vibration generating device 126 transmits the vibrations to the wrist on which the watch 100 is worn, thereby notifying the notification information to the user 800. This can improve usability.

[0056] The communication device 127 is a communication interface that performs wireless communication with the HMD 200 by short-range wireless communication or the like. The communication device 127 includes a communication processing circuit, an antenna, etc. that correspond to various predetermined communication interfaces, and transmits and receives various information, control signals, etc. It may also include a telephone communication network.

[0057] [Function Configuration] Next, we will explain the functions realized by the watch 100 of this embodiment. Figure 3(b) is a functional block diagram of the watch 100.

[0058] The watch 100 of this embodiment includes a motion detection unit 151, a motion discrimination unit 152, a switching control unit 153, a watch motion control unit 154, an HMD operation command generation unit 155, a watch stand-alone operation database (DB) 160, and an HMD cooperative operation DB 170.

[0059] The motion detection unit 151 detects motions of the user 800 using a sensor 130 such as an electromyographic sensor 131. As described above, the detected motions of the user 800 also include operation instructions input via the first input I / F 122, the second input I / F 123, and the voice input device 124. The detected motions (hereinafter referred to as detected motions) are output to the motion discrimination unit 152 and the watch operation control unit 154 or the HMD operation command generation unit 155. As will be described later, when the operation mode is the watch standalone operation mode, the detected motions are output to the motion discrimination unit 152 and the watch operation control unit 154, and when the operation mode is the HMD-linked operation mode, the detected motions are output to the motion discrimination unit 152 and the HMD operation command generation unit 155.

[0060] The operation determination unit 152 determines whether the detected operation matches a predetermined trigger operation (a cooperation start trigger operation or a cooperation end trigger operation). The trigger operation is stored in the memory 102 in advance.

[0061] If the action determination unit 152 determines that the detected action matches the trigger action, it outputs a watch-side trigger action detection instruction to the switching control unit 153. On the other hand, if they do not match, the action determination unit 152 outputs a mismatch signal to the action detection unit 151. When the action detection unit 151 receives the mismatch signal, it outputs the detected action to the watch action control unit 154 or the HMD operation command generation unit 155, as described above, depending on the operation mode.

[0062] The switching control unit 153 switches the operation mode. In this embodiment, when the operation detected by the operation detection unit 151 is determined by the operation determination unit 152 to be a watch-side cooperation start trigger operation during the watch standalone operation mode, the switching control unit 153 switches the operation mode to the cooperation operation mode if a detection notification is received from the HMD 200 within a predetermined period that the HMD 200 has also detected an HMD-side cooperation start trigger operation.

[0063] The detection of the cooperation start trigger action may be performed first by the watch 100 or the HMD 200. In the following, this embodiment will be described taking as an example a case where the cooperation start trigger action is detected by the watch 100 and then also detected by the HMD 200. That is, a case will be described taking as an example a case where the cooperation start trigger action is detected first by the watch 100, and if the cooperation start trigger action is detected by the HMD 200 within a predetermined time, an operation command for the HMD 200 is generated in accordance with the action detected by the watch 100, and the HMD 200 is operated by the watch 100.

[0064] In this case, when the switching control unit 153 detects a cooperation start trigger action, it transmits start trigger information indicating that the cooperation start trigger action has been detected to the HMD 200, and when it receives a switching request from the HMD 200 within a predetermined period after the transmission, it switches the operation mode to the HMD cooperation operation mode. Whether or not the cooperation start trigger action has been detected is determined based on whether or not a trigger action detection instruction has been received from the action determination unit 152. Furthermore, the switching control unit 153 outputs the operation mode after switching to the action detection unit 151.

[0065] Furthermore, when a cooperation end trigger operation is detected in the HMD cooperative operation mode, the switching control unit 153 switches from the HMD cooperative operation mode to the watch standalone operation mode. At this time, the switching control unit 153 transmits end trigger information indicating that the cooperation end trigger operation has been detected to the HMD 200. Furthermore, when the switching control unit 153 receives end trigger information from the HMD 200 in the HMD cooperative operation mode, the switching control unit 153 switches from the HMD cooperative operation mode to the watch standalone operation mode.

[0066] In the watch stand-alone operation mode, the watch operation control unit 154 controls the operation of the watch 100 in response to the detection operation. In this embodiment, the watch operation control unit 154 references a watch stand-alone operation DB 160, which stores the operation details of the watch 100 in association with the detection operation, generates a command (watch operation command; own device operation command) for operating the watch 100 corresponding to the operation details, and outputs it to the related function unit.

[0067] The HMD operation command generation unit (external operation command generation unit) 155 generates a command (HMD operation command; external operation command) for operating the HMD 200 in response to the detection operation, and outputs the command to the HMD 200. In this embodiment, the command is transmitted to the HMD 200 via the communication device 127. The HMD operation command generation unit 155 refers to an HMD cooperative operation DB 170 in which operation details of the HMD 200 are stored in association with the detection operation, and generates an HMD operation command corresponding to the operation details.

[0068] Here, we will explain examples of the watch stand-alone operation DB 160 and the HMD cooperative operation DB 170. Fig. 4(a) is a diagram for explaining an example of the watch stand-alone operation DB 160 of this embodiment, and Fig. 4(b) is a diagram for explaining an example of the HMD cooperative operation DB 170.

[0069] 4(a), the watch standalone operation DB 160 registers watch operation details 162 in association with detected actions 161. For example, when the electromyographic sensor 131 detects the action of "raising a hand to stop" on the watch 100, the watch 100 performs an operation of "rejecting an incoming call." Note that instead of the watch operation details 162, the watch operation command itself that realizes the operation may be registered.

[0070] 4(b), the HMD cooperative operation DB 170 registers HMD operation content 172 in association with detected actions 171. For example, when the electromyographic sensor 131 detects an action of "raising a hand to stop" on the watch 100, an operation of "displaying a menu" is performed on the HMD 200. Note that instead of the HMD operation content 172, the HMD operation command itself that realizes the operation may be registered.

[0071] [HMD] [Hardware configuration] Next, the configuration of the HMD 200 of this embodiment will be described. Fig. 5(a) is a hardware configuration diagram of the HMD 200 of this embodiment. In this figure, parts shown in Fig. 1 and Fig. 2 and assigned the same reference numerals have the same operations as those already explained in these figures, so detailed explanations of those parts will be omitted.

[0072] As shown in this figure, the HMD 200 of this embodiment includes a processor 201, a memory 202, a camera 210, a sensor 230, a display device 221, an input I / F 223, an audio input device 224, an audio output device 225, a vibration generating device 226, and a communication device 227.

[0073] The camera 210 includes an outer camera 211 and an inner camera 212. The sensor 230 includes an acceleration sensor 232, a gyro sensor 233, a geomagnetic sensor 234, a left eye gaze sensor 235, a right eye gaze sensor 236, and a distance measurement sensor 237.

[0074] These components are connected to each other via a bus 203 .

[0075] The outer camera 211 and the inner camera 212 capture images of the front of the HMD 200 and the user 800, respectively.

[0076] The outer camera 211 can also capture images of the wrist of the user 800 and the watch 100. This allows the outer camera 211 to capture images of gestures made by the user 800 and detect trigger operations of the watch 100.

[0077] The in-camera 212 captures the eyes and surrounding area of ​​the user 800 as subjects, and measures and detects changes in state such as the movement of the eyeballs and pupils. For example, a trigger action such as closing the eyes for a moment can be detected. A known method for measuring and detecting eyeball movement is to capture an image of the eyes with a visible light camera, use the inner corner of the eye as a reference point and the iris as a moving point, and detect the gaze based on the position of the iris relative to the inner corner of the eye. This corresponds to the case where the in-camera 212 captures an image of the eyes of the user 800, and measures and detects the movement of the eyeballs and pupils from the captured image.

[0078] The left eye gaze sensor 235 and the right eye gaze sensor 236 are sensors that detect the movement and direction of the left eye and the right eye, respectively, and capture eye movement and gaze. Note that the process of detecting eye movement can utilize well-known technology that is commonly used as eye tracking processing. For example, a known method using corneal reflex is to shine an infrared light emitting diode (LED) on the face, capture an image with an infrared camera, use the position on the cornea of ​​the reflected light from the infrared LED irradiation (corneal reflex) as a reference point, and detect eye movement and gaze based on the position of the pupil relative to the position of the corneal reflex.

[0079] The acceleration sensor 232 is a sensor that detects acceleration, which is a change in speed per unit time, and captures movement, vibration, shock, etc. When the only acceleration applied is gravity, the acceleration sensor 232 can calculate the tilt angle using the gravity vector and its projection on the axis of the acceleration sensor 232, and measure and detect how much the HMD 200 is tilted with respect to the ground. The acceleration sensor 232 provided in the HMD 200 can detect the tilt of the HMD 200.

[0080] The gyro sensor 233 is a sensor that detects angular velocity in the rotation direction, and captures the vertical, horizontal, and diagonal posture states to measure and detect how much movement has occurred in each direction.

[0081] Therefore, the acceleration sensor 232 and the gyro sensor 233 can be used to detect the attitude of the HMD 200, such as the tilt and direction.

[0082] The geomagnetic sensor 234 is a sensor that detects the magnetic force of the earth and detects the direction in which the HMD 200 is facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions is used, and by capturing changes in geomagnetism in response to the movement of the HMD 200, the movement of the HMD 200 can also be detected.

[0083] The acceleration sensor 232, the gyro sensor 233, and the geomagnetic sensor 234 can detect and determine the posture of the user 800 wearing the HMD 200. For example, it can detect a state in which the user 800 turns their gaze toward the watch 100, even though they are facing away from the watch 100.

[0084] The ranging sensor 237 is a sensor that can measure the distance and angle to an object and capture the shape of the object in three dimensions. The ranging sensor 237 may be a LiDAR (Light Detection and Ranging), TOF (Time Of Flight) sensor, millimeter-wave radar, or the like. LiDAR irradiates an object with laser light such as infrared light, measures the scattered light that bounces back, and analyzes and detects the distance to a distant object and the state of that object. The TOF sensor measures the distance by measuring the reflection time of pulsed light irradiated onto the subject for each pixel. The millimeter-wave sensor emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the object from which the waves are reflected and the state of the object.

[0085] The distance measurement sensor 237 measures the distance and angle to the index finger, thumb, etc., and based on this measurement information, can determine, for example, whether the hand is clenched or open. Therefore, the distance measurement sensor 237 can detect actions such as triggering the watch 100.

[0086] The processor 201 is configured with a CPU or the like, and executes programs such as an OS and an operation control application stored in the memory 202, thereby realizing the functions of the HMD 200. The functions to be realized will be described later.

[0087] The memory 202 is configured with a nonvolatile storage device or the like, and functions as a storage unit that stores various programs and data handled by the processor 201, etc. The stored data includes data used by the HMD 200 for processing, data generated by processing, etc. Specifically, the data includes motion data detected by the HMD 200, various predetermined trigger motion data, operation commands for the HMD, and switching requests to be sent to the watch 100.

[0088] The display device 221 displays various types of information.

[0089] When the HMD 200 is an optical see-through type, the display device 221 includes a projection unit and a transparent half mirror. For example, virtual objects such as an operation guide screen image for measuring finger tap distance, an image that relaxes the user 800, and notification information for the user 800 are projected onto the projection unit. The projected virtual objects are displayed as images in front of the user's eyes on the half mirror. This allows the user 800 to visually recognize both real objects in the field of view in front of the user's eyes and the imaged virtual objects together, as if they were floating.

[0090] When the HMD 200 is a video see-through type, the display device 221 includes a display such as a liquid crystal panel. This display displays the real object in front of the user's eyes photographed by the outer camera 211 together with the virtual object, etc. This allows the user 800 to visually recognize the real object in the field of view image in front of the user's eyes superimposed on the virtual object, etc.

[0091] The input I / F 223 is an input means such as key buttons, touch keys, etc., and accepts setting input of information desired to be input by the user 800. The input I / F 223 is provided in a position and form within the HMD 200 that makes it easy for the user 800 to perform input operations. The input I / F 223 may be separated from the main body of the HMD 200 and connected via a wired or wireless connection. An input operation screen may also be displayed on the display screen of the display device 221. In this case, input operation information may be captured based on the position on the input operation screen to which the line of sight is directed, or a pointer may be displayed on the input operation screen and the pointer may be operated by the input I / F 223 to capture the input operation information.

[0092] The audio input device 224 uses a microphone to collect external sounds and speech from the user 800 and converts the collected sounds into audio data. Instruction information from the speech of the user 800 can be captured in the HMD 200, and an operation corresponding to the instruction information can be easily performed. For example, the user 800 may vocalize a voice indicating an input operation, and the voice may be collected by the audio input device 224 to capture the input operation information.

[0093] The audio output device 225 outputs audio from a speaker based on the audio data, and can notify the user 800 of notification information by audio.

[0094] The vibration generating device 226 generates vibrations under the control of the processor 201. For example, the processor 201 converts notification instruction information for the user 800 transmitted by the HMD 200 into vibrations, and transmits the vibrations to the head of the user 800 wearing the HMD 200 via the vibration generating device 226 to notify the user of the notification information. This can improve usability.

[0095] The communication device 227 is a communication interface that performs wireless communication with the watch 100 via short-range wireless communication. The communication device 227 includes a communication processing circuit, an antenna, etc. that correspond to various predetermined communication interfaces, and transmits and receives various information, control signals, etc. It may also include a telephone communication network.

[0096] [Function Configuration] Next, a description will be given of functions realized by the HMD 200 of this embodiment.

[0097] The HMD 200 of this embodiment includes a motion detection unit 251, a motion determination unit 252, a switching control unit 253, an HMD motion control unit 254, a cooperation control unit 255, and an HMD standalone operation DB 260.

[0098] The movement detection unit 251 detects the movement of the user 800 using the outer camera 211, the inner camera 212, and the sensor 130. The sensor 130 may include, for example, a left eye gaze sensor 235, a right eye gaze sensor 236, an acceleration sensor 232, a gyro sensor 233, a geomagnetic sensor 234, and a distance measurement sensor 237.

[0099] The detected operation (hereinafter referred to as the detected operation) is output to the operation determination unit 252 and the HMD operation control unit 254 or the cooperation control unit 255. As will be described later, when the operation mode is the HMD stand-alone operation mode, the detected operation is output to the HMD operation control unit 254, and when the operation mode is the watch cooperation operation mode, the detected operation is output to the cooperation control unit 255.

[0100] The operation determination unit 252 determines whether the detected operation matches a predetermined trigger operation (a cooperation start trigger operation or a cooperation end trigger operation). The trigger operation is stored in the memory 202 in advance.

[0101] If the action determination unit 252 determines that the detected action matches the trigger action, it outputs a trigger action detection instruction to the switching control unit 253. On the other hand, if they do not match, the action determination unit 252 outputs a mismatch signal to the action detection unit 251. Note that, upon receiving the mismatch signal, if the operation mode is the HMD standalone operation mode, the action detection unit 251 outputs the detected action to the HMD operation control unit 254. On the other hand, if the operation mode is the watch-linked operation mode, the detected action is ignored.

[0102] The switching control unit 253 switches the operation mode. When, in the standalone operation mode, the operation detected by the operation detection unit 251 is determined by the operation determination unit 252 to be an HMD-side cooperation start trigger operation, and the switching control unit 253 receives a detection notification from the watch 100 within a predetermined period that the watch 100 has also detected a watch-side cooperation start trigger operation, the switching control unit 253 switches the operation mode to the cooperation operation mode.

[0103] As described above, the detection of the cooperation start trigger action may be performed first by the watch 100 or the HMD 200. In the following, this embodiment will be described taking as an example a case where the watch 100 detects the cooperation start trigger action, and then the HMD 200 also detects it.

[0104] In this case, the switching control unit 253 determines whether start trigger information has been received from the watch 100 at predetermined time intervals in the HMD standalone operation mode. Then, if a cooperation start trigger action is detected within a predetermined period after receiving the start trigger information from the watch 100, the switching control unit 253 transmits an operation mode switching request to the watch 100 and switches the operation mode of the HMD 200 to watch cooperation operation mode. The operation mode switching request is an instruction to the watch 100, which is the transmission destination, to switch the operation mode. Whether a cooperation start trigger action has been detected is determined by whether a trigger action detection instruction has been received from the action determination unit 252. The switching control unit 253 also outputs the switched operation mode to the action detection unit 251.

[0105] Furthermore, when a cooperation end trigger operation is detected in the watch cooperative operation mode, the switching control unit 253 switches from the watch cooperative operation mode to the HMD standalone operation mode. At this time, the switching control unit 253 transmits end trigger information indicating that a cooperation end trigger operation has been detected to the watch 100. Furthermore, when the switching control unit 253 receives end trigger information from the watch 100 in the watch cooperative operation mode, the switching control unit 253 switches from the watch cooperative operation mode to the HMD standalone operation mode.

[0106] In the HMD standalone operation mode, the HMD operation control unit 254 controls the operation of the HMD 200 in accordance with the detection operation. In this embodiment, the HMD operation control unit 254 refers to an HMD standalone operation DB 260 in which operation contents of the HMD 200 are stored in association with the detection operation, generates a command (HMD operation command) for operating the HMD 200 corresponding to the operation contents, and outputs it to the related function unit.

[0107] In the watch cooperative operation mode, when the cooperative control unit 255 receives an HMD operation command from the watch 100 via the communication device 227, it outputs the HMD operation command to the related function unit. This realizes control of the operation of the HMD 200 according to the operation instruction from the watch 100.

[0108] Here, a description will be given of an example of the HMD stand-alone operation DB 260. Fig. 6 is a diagram for explaining an example of the HMD stand-alone operation DB 260 of this embodiment.

[0109] 6, the HMD standalone operation DB 260 registers HMD operation details 262 in association with detected actions 261. For example, when the HMD 200 detects an action of "tapping the wrist with the opposite hand" by analyzing an image acquired by the outer camera 211, an operation of "displaying a menu" is performed in the HMD 200. Note that instead of the HMD operation details 262, an HMD operation command itself that realizes the operation may be registered.

[0110] [Linkage process (operation mode switching process)] Next, the flow of processing when the watch 100 and HMD 200 having the above configurations are linked together will be described. Here, the explanation will focus on the operation mode switching process. It is assumed that the watch 100 and HMD 200 are already linked together for communication.

[0111] 7 is a processing flow of the cooperative processing (operation mode switching processing) of the information processing system 900 of this embodiment. Note that in this diagram, an example will be described in which the watch 100 and the HMD 200 are both started up in the standalone operation mode as an initial state.

[0112] Although not specified in the processing flow, this processing ends when an action indicating termination is detected in each action detection step. The areas surrounded by dashed lines are processing areas in standalone operation mode, and the areas surrounded by dashed lines are processing areas in cooperative operation mode.

[0113] After the watch 100 is started, the motion detector 151 detects the motion of the user 800 at predetermined time intervals (step S1101).

[0114] Then, each time the action detection unit 151 detects an action, the action determination unit 152 determines whether the detected action matches a predetermined trigger action (step S1102). Here, since the current operation mode of the watch 100 is the watch-standalone operation mode, it determines whether the detected action is a watch-side cooperation start trigger action that transitions to the cooperation operation mode.

[0115] In step S1102, if the trigger operation to start cooperation on the watch side is not detected, the watch stand-alone operation mode continues as it is, and the watch operation control unit 154 operates the watch 100 according to the detected operation (step S1103), and the process returns to step S1101. As a result, if a trigger operation is not detected, the stand-alone operation mode continues as it is.

[0116] On the other hand, if it is determined in step S1102 that the trigger action is a watch-side cooperation start trigger action, the action determination unit 152 outputs a trigger action detection instruction to the switching control unit 153. In response to this, the switching control unit 153 executes an operation mode switching process. Here, trigger information (start trigger information) indicating that the trigger action has been performed is generated and transmitted to the HMD 200 (step S1201).

[0117] On the HMD 200 side, after startup, the motion detection unit 251 detects the motion of the user 800 at predetermined time intervals, just like the watch 100 (step S2101). During this time, the switching control unit 253 determines whether or not trigger information (start trigger information) has been received from the watch 100 (step S2102). If start trigger information has not been received, the HMD operation control unit 254 controls the HMD 200 according to the detected motion (step S2201), and returns to step S2101.

[0118] On the other hand, when the switching control unit 253 determines that it has received the start trigger information, the action determination unit 252 determines whether or not the detected action detected by the action detection unit 251 is a predetermined trigger action (step S2201). Here, since the current operation mode of the HMD 200 is the HMD standalone operation mode, it determines whether or not it is a trigger action for starting a cooperative operation mode (HMD-side cooperative start trigger action). Note that if it is not an HMD-side cooperative start trigger action, the process returns to step S2101 and continues.

[0119] If it is an HMD-side collaboration start trigger operation, the switching control unit 253 generates an operation mode switching request and transmits it to the watch 100 that sent the start trigger information, because the HMD 200 side also detected an HMD-side collaboration start trigger operation at the time the start trigger information was received from the watch 100 (step S2202).

[0120] On the watch 100 side, the switching control unit 153 determines whether or not a predetermined reply is received from the HMD 200 within a predetermined period of time after the generation and transmission of the start trigger information (step S1202). Here, it determines whether or not an operation mode switching request is received within the predetermined period of time (step S1202). If it is received, the switching control unit 153 generates a result indicating reception (so-called ACK), replies to the HMD 200 that sent it (step S1203), and switches the operation mode (step S1204). Here, the watch transitions from standalone operation mode to HMD-linked operation mode. Note that if no response is received in step S1202, the process returns to step S1101.

[0121] Here, in step S1202, even if an operation mode switching request is not received within a predetermined period, a result (ACK) indicating that the request has not been received may be generated and returned to the HMD 200 that is the sender.

[0122] On the HMD 200 side, after transmitting the operation mode switching request, the switching control unit 253 receives a result within a predetermined period (step S2203), and then switches the operation mode (step S2204). Here, the operation mode transitions from the HMD standalone operation mode to the watch linked operation mode. On the other hand, if no result is received in response to the switching request, the process returns to step S2101 and continues.

[0123] In this way, the operation mode is switched only when a cooperation start trigger action is detected within a predetermined period on both the watch 100 side and the HMD 200 side.

[0124] On the watch 100 side, after transitioning to the HMD cooperative operation mode, when the motion detection unit 151 detects a motion, the motion determination unit 152 determines whether the detected motion matches a predetermined trigger motion each time the motion detection unit 151 detects a motion (step S1301). Here, because the current operation mode of the watch 100 is the HMD cooperative operation mode, it determines whether the detected motion is a trigger motion for ending the cooperative operation mode (watch-side cooperative end trigger motion).

[0125] If there is no cooperation end trigger action, the HMD operation command generation unit 155 generates an HMD operation command in response to the detected action and transmits it to the HMD 200 (step S1303).

[0126] On the other hand, if it is a trigger action, the action determination unit 152 determines whether the action is an operation end action for the watch 100 (step S1304). If it is an operation end instruction, the processing of the watch 100 ends. At this time, the HMD 200 is notified that the operation of the watch 100 itself has ended. Then, upon receiving the notification, the HMD 200 transitions to stand-alone operation mode.

[0127] On the other hand, if the instruction is not to end the operation, the switching control unit 153 performs an operation mode return process, which will be described later (step S1401).

[0128] On the other hand, in the HMD 200 that has transitioned to the watch cooperative operation mode, when the cooperative control unit 255 receives an HMD operation command from the watch 100 (step S2301), it controls (operates) the operation of the HMD 200 in accordance with the HMD operation command (step S2302).

[0129] During this time, the action detection unit 251 continuously detects actions (step S2303). Then, every time the action detection unit 251 detects an action, the action determination unit 252 determines whether the detected action is a trigger action or not (step S2304). Here, it determines whether the detected action is a cooperation end trigger action or not. If it is not a trigger action, the process returns to step S2301 and continues.

[0130] On the other hand, if it is a trigger action, the action determination unit 252 first determines whether the action is an instruction to end operation of the HMD 200 (step S2305). If it is an instruction to end operation, the process ends. At this time, the watch 100 is notified that the operation of the HMD 200 itself has ended. Then, upon receiving the notification, the watch 100 transitions to standalone operation mode.

[0131] On the other hand, if the instruction is not to end the operation, the switching control unit 253 executes the operation mode return process (step S2401).

[0132] Here, the operation mode return processing executed in steps S1401 and S2401 will be described. In this embodiment, when a cooperation end trigger action is detected in either the watch 100 or the HMD 200, the cooperation operation mode is returned to the stand-alone operation mode.

[0133] For this reason, for example, when the watch 100 determines that the detected action is a cooperation end trigger action, the switching control unit 153 communicates end trigger information indicating that the cooperation end trigger action has been detected to the HMD 200, and switches the operation mode of the watch 100 to watch-standalone operation mode. Furthermore, in the HMD 200 that has received the end trigger information from the watch 100, the switching control unit 253 switches the operation mode of the HMD 200 to the HMD-standalone operation mode.

[0134] On the other hand, for example, if the HMD 200 determines that the detected action is a cooperation end trigger action, the switching control unit 253 communicates end trigger information indicating that the cooperation end trigger action has been detected to the watch 100, and switches the operation mode of the HMD 200 to the HMD standalone operation mode. Furthermore, in the watch 100 that has received the end trigger information from the HMD 200, the switching control unit 153 switches the operation mode of the watch 100 to the watch standalone operation mode.

[0135] After the operation mode return process is complete, the watch 100 returns to step S1101 and continues operating in the watch stand-alone operation mode. Similarly, after the operation mode return process is complete, the HMD 200 returns to step S2101 and continues operating in the HMD stand-alone operation mode.

[0136] In this embodiment, if the HMD 200 detects a trigger action to start cooperation within a predetermined period (T1) after the watch 100 receives trigger information to start cooperation, the watch 100 transitions to the cooperation operation mode. Therefore, the HMD 200 may be configured to hold the trigger information received from the watch 100 for the period T1, and in step S2201, determine whether or not a trigger action to start cooperation has occurred during that period.

[0137] As described above, in the information processing system 900 of this embodiment, only if the HMD 200 detects a cooperation start trigger action within a predetermined period after the watch 100 detects a cooperation start trigger action, do the two systems transition to cooperation operation mode. For example, if only one system detects a cooperation start trigger action, or if both systems detect the cooperation start trigger action but not within the predetermined period, the two systems do not transition to cooperation operation mode. On the other hand, if either system detects a cooperation end trigger action during cooperation operation mode, the two systems transition to standalone operation mode.

[0138] In this way, in contrast to cases where the operation mode is switched solely by a trigger operation on either the watch 100 side or the HMD 200 side, according to this embodiment, the trigger operations of both sides are captured in a simple manner to transition to the linked operation mode, making it easy to use, reliable, and highly accurate, and eliminating or significantly reducing the possibility of misrecognition and malfunction when selecting and switching the operation mode.

[0139] [Display example] Here, a description will be given of a display example on the HMD 200. In this embodiment, the HMD operation control unit 254 controls the display on the display device 221 (display unit) in the HMD standalone operation mode, and the cooperation control unit 255 controls the display on the display device 221 (display unit) in the watch cooperation operation mode.

[0140] As shown in FIGS. 8(a) and 8(b), the display device 221 of this embodiment includes a menu display area 270 and an operation mode display area 271.

[0141] Menus are displayed as virtual objects in the menu display area 270. The user 800 selects a desired operation from the menu and operates the HMD 200. The selection is made, for example, by line of sight. In this case, the HMD 200 uses sensors such as the left eye line of sight sensor 235 and the right eye line of sight sensor 236 to detect the intersection position between the line of sight and the display device 221, and determines which menu has been selected.

[0142] The operation mode is displayed in the operation mode display area 271. Fig. 8(a) is a display example when the HMD 200 is in the HMD standalone operation mode. Fig. 8(b) is a display example when the HMD 200 is in the watch linked operation mode.

[0143] As shown in FIG. 8(b), in the watch linked operation mode, an operation image display area 272 may also be provided. The operation image display area 272 displays an operation image showing what kind of movement the user 800 is making. The operation image is, for example, a simulated image of a hand, which shows what kind of movement the user 800 is making as detected by the watch 100. The operation image is an image of an movement corresponding to an HMD operation command sent from the watch 100.

[0144] For example, when an HMD operation command corresponding to "click on selected item" in the HMD operation content 172 of the HMD cooperative operation DB 170 shown in FIG. 4(b) is received from the watch 100, an image showing the corresponding action of "lowering the hand quickly" is generated. In this case, the cooperation control unit 255 generates a motion image corresponding to the received HMD operation command. The HMD 200 stores the motion image associated with the HMD operation command in the memory 202.

[0145] Note that the movement information of the user 800 detected by the watch 100 may be transmitted to the HMD 200 together with an HMD operation command, and the cooperation control unit 255 may use this information to generate a movement image (simulation image).

[0146] By displaying the operation mode in the operation mode display area 271, the user 800 can clearly and easily know how the HMD 200 is being operated. Note that the operation mode display may not be a character display, but may be an image display such as an icon.

[0147] As described above, by displaying the operation image on the HMD 200, the user 800 can intuitively understand how the movement is detected by the watch 100. This can be expected to encourage the user to move their hands in a way that improves the detection accuracy of the watch 100.

[0148] As described above, the watch 100, which is the first information processing device of the information processing system 900 of this embodiment, is capable of mutual communication with the HMD 200, which is an external information processing device, and is equipped with a memory 202 that stores predetermined cooperation start trigger actions, a motion detection unit 151 that detects actions of the user 800 wearing the watch 100, an action determination unit 152 that determines whether the action detected by the motion detection unit 151 is a cooperation start trigger action, and a switching control unit 153 that switches the operation mode of the watch 100. The operation modes include a cooperation operation mode in which the watch 100 generates operation commands to operate the HMD 200 and transmits them to the HMD 200, and a standalone operation mode in which the watch 100 generates operation commands to operate the watch 100 and operates the watch 100 in accordance with the generated operation commands. In addition, when in the standalone operation mode, if the operation discrimination unit 152 determines that the operation is a watch-side collaboration start trigger operation and the HMD 200 receives a detection notification indicating that the HMD 200 has detected an HMD-side collaboration start trigger operation within a predetermined period, the switching control unit 153 switches the operation mode to the collaboration operation mode.

[0149] The HMD 200, which is a second information processing device, is capable of mutual communication with the watch 100, which is an external information processing device, and includes a memory 202 that stores a predetermined cooperation start trigger action, a motion detection unit 251 that detects a motion of the user 800 wearing the HMD 200, an action determination unit 252 that determines whether the motion detected by the motion detection unit 251 is a cooperation start trigger action, and a switching control unit 253 that switches the operation mode of the HMD 200. The operation modes include a cooperation operation mode in which an operation command to operate the HMD 200 is received from the watch 100 and the HMD 200 is operated in accordance with the operation command, and a standalone operation mode in which an operation command to operate the HMD 200 is generated in the HMD 200 and the HMD 200 is operated in accordance with the generated operation command. In addition, when in stand-alone operation mode, if the operation discrimination unit 252 determines that the operation is an HMD-side collaboration start trigger operation and the watch 100 receives a detection notification indicating that the watch 100 has detected a watch-side collaboration start trigger operation within a predetermined period, the switching control unit 253 switches the operation mode to the collaborative operation mode.

[0150] As described above, according to this embodiment, in an information processing system 900 having multiple information processing devices such as the watch 100 and the HMD 200, for example, when the watch 100 is capable of executing an operation mode for operating the watch 100 itself and an operation mode for operating the HMD 200, switching between the operation modes can be performed easily and with high accuracy. Specifically, when trigger actions are detected in both the watch 100 and the HMD 200, the information processing device operated in response to the user action detected by the watch 100 switches from the standalone operation mode of the watch 100 to the collaborative operation mode of the HMD 200. Therefore, compared to switching the operation mode solely by trigger actions on the watch 100 or the HMD 200, switching the operation mode is performed by capturing both trigger actions in a simple manner, making it possible to switch the operation mode easily, reliably, and with high accuracy. This eliminates or significantly reduces the possibility of erroneous recognition or malfunction when switching the operation mode.

[0151] Furthermore, in the above embodiment, myoelectric information acquired by the myoelectric sensor 131 is used to detect trigger operations in the watch 100. This allows both the watch 100 and the HMD 200 to be freely operated as information processing devices, even in situations where only one hand is available and, for example, the watch 100 cannot be touched.

[0152] That is, according to this embodiment, in an information processing system having multiple information processing devices that can switch between operation targets, when switching or selecting an operation mode, i.e., which of the multiple information processing devices to operate, the switching or selection of the operation mode can be performed easily and conveniently with high accuracy and precision. This is particularly effective for information processing devices such as the watch 100 and the HMD 200, which are difficult to operate and whose operations are difficult to detect.

[0153] <Variation 1> In the above embodiment, the watch 100 generates an HMD operation command based on the detection action in the cooperative operation mode and transmits it to the HMD 200, but this is not limiting. For example, information indicating the detection action may be transmitted to the HMD 200, and the HMD 200 may generate an operation command corresponding to that action.

[0154] <Variation 2> Furthermore, the predetermined cooperation start trigger operation is not limited to the one described in the above embodiment. Figure 9 is a diagram for explaining another example of the cooperation start trigger operation.

[0155] 9 shows a case where a clenched hand motion is predetermined as the cooperation start trigger motion on the watch 100 side. The watch 100 uses the electromyography sensor 131 to detect a gesture motion that changes from an open hand state 821 to a closed, clenched hand state 822.

[0156] On the other hand, the example shows a case where a collaboration start trigger action on the HMD 200 side is predetermined as an action of the user 800 wearing the HMD 200 directing their gaze toward the watch 100. The HMD 200 uses the left eye gaze sensor 235 and the right eye gaze sensor 236 to capture a state 831 in which the user 800 directs their gaze toward the watch 100 from a state 832 in which the user 800 is facing in a direction different from the watch 100, and detects this as a collaboration start trigger action.

[0157] Note that the state 831 in which the user 800 is directing his / her gaze in the direction of the watch 100 may be captured using the acceleration sensor 232, gyro sensor 233, and geomagnetic sensor 234 that detect the posture and movement of the user 800.

[0158] When the watch 100 and the HMD 200 detect these cooperation start trigger actions at approximately the same time, the watch 100 transmits to the HMD 200 an HMD operation command generated from the action detected by the watch 100 as operation information for operating the HMD 200.

[0159] In the above embodiment, an example was shown in which the electromyographic sensor 131 was used to detect hand and finger gestures as a way of detecting motions, including trigger motions, on the watch 100, but this is not limiting. Motions on the watch 100 may be detected using other sensors 130 or means, rather than the electromyographic sensor 131. For example, the posture and motion of the watch 100 may be detected using sensors 130 such as the acceleration sensor 132, gyro sensor 133, and geomagnetic sensor 134, and motions such as "lifting a hand and twisting" or "lifting a hand and stopping" may be acquired.

[0160] In addition, in the above embodiment, examples of detecting trigger operations in the HMD 200 include detecting changes in the state of the user's 800's eyes using a gaze sensor (left eye gaze sensor 235 and / or right eye gaze sensor 236) and detecting changes in the state of the user's 800's head using an acceleration sensor 232, a gyro sensor 233, and a geomagnetic sensor 234, but detection may also be performed using other sensors 230 or means.

[0161] In other words, the actions detected by the first information processing device and the second information processing device can be any action, not limited to the actions described above, as long as they can be detected by some kind of sensor or means, and it is clear that the same actions and effects as those described above can be obtained.

[0162] <Variation 3> In the above embodiment, the cooperation start trigger action and the cooperation end trigger action have been described as different actions, but they may be the same. In this case, the action determination unit 152 and the action determination unit 252 determine which trigger action is involved in combination with the operation mode of the own device. That is, if the trigger action is detected when the own device is in standalone operation mode, it is determined to be a cooperation start trigger action, and if the trigger action is detected when the own device is in cooperative operation mode, it is determined to be a cooperation end trigger action.

[0163] <Variation 4> In addition, in the above embodiment, an example was given in which the watch 100 (operating device) first detects a trigger action, transmits it to the HMD 200 (operated device), and receives a switching request from the HMD 200, but the HMD 200 may first detect a trigger action and transmit it to the watch 100.

[0164] In this case, after receiving start trigger information indicating that an HMD-side cooperation start trigger action has been detected from the HMD 200, if the action detected by the action detection unit 151 within a predetermined period is determined to be the cooperation start trigger action, the switching control unit 153 transmits an operation mode switching request to the HMD 200. At this time, the operation mode of the watch 100 is switched to the HMD cooperation operation mode.

[0165] Furthermore, on the HMD 200 side, when the switching control unit 253 detects an HMD-side cooperation start trigger operation, it transmits start trigger information to the watch 100. Then, when an operation mode switching request is received within a predetermined time from the transmission of the start trigger information, the switching control unit 253 switches the operation mode of the HMD 200 to the HMD cooperation operation mode.

[0166] Furthermore, in the cooperative operation mode, the HMD 200 is operated by the movements detected on the watch 100 side, but the opposite is also possible. In other words, the watch 100 may be operated by the movements detected on the HMD 200 side.

[0167] As described above, the information processing system 900 of the present embodiment is an information processing system including a first information processing device and a second information processing device that are communicatively linked with each other, and when a second cooperation start trigger action is detected in the second information processing device within a predetermined time after a first cooperation start trigger action is detected in the first information processing device, the first information processing device and the second information processing device are operated in a cooperation operation mode. This cooperation operation mode is either a first cooperation operation mode in which the second information processing device is operated in accordance with the action of a user of the first information processing device detected by the first information processing device, or a second cooperation operation mode in which the first information processing device is operated in accordance with the action of a user of the second information processing device detected by the second information processing device.

[0168] <Variation 5> In the above embodiment and modified example, the operation mode switching request is generated and transmitted by a device other than the device that first detected the cooperation start trigger operation, but this is not limiting. The operation mode switching request may be generated by the device that first detected the cooperation start trigger operation.

[0169] This modified example will be described using an example in which, for example, the watch 100 first detects a cooperation start trigger action. When a cooperation start trigger action is detected in step S1102 above, the switching control unit 153 generates start trigger information and transmits it to the HMD 200. On the HMD 200 side, after receiving the start trigger information in step S2102, if cooperation start trigger information is detected within the processing period in step S2201, the switching control unit 253 generates start trigger information (second start trigger information) and transmits it to the watch 100. That is, instead of generating an operation mode switching request in the HMD 200 and transmitting it to the watch 100 as a reply in the above embodiment, in this modified example, the HMD 200 also transmits start trigger information to the watch 100 as a reply.

[0170] If the watch 100 receives start trigger information from the HMD 200 within a predetermined period after first detecting a cooperation start trigger action or after generating and transmitting start trigger information, the switching control unit 153 generates an operation mode switching request and notifies the HMD 200. The operation mode of the watch 100 is also switched. After replying and receiving the operation mode switching request, the HMD 200 switches the operation mode.

[0171] When the HMD 200 first detects a collaboration start trigger action, the start trigger information is sent and received in the reverse order to the above, and the switching control unit 253 of the HMD 200 finally sends an operation mode switching request to the watch 100, and both devices switch operation modes.

[0172] <Variation 6> In the above embodiment, when a trigger action to terminate collaboration is detected in either device, collaboration is terminated and the operation mode is returned to the standalone operation mode. However, this is not limited to this. Collaboration may also be terminated only if a trigger action to terminate collaboration is detected in both devices within a predetermined period of time.

[0173] <Variation 7> In the above embodiment, the HMD 200, which is the second information processing device, transitions to the HMD standalone operation mode upon startup. However, this is not limiting. The HMD 200 can be in any operation mode immediately after startup as long as it can receive start trigger information from the watch 100.

[0174] <Variation 8> In the above embodiment, the watch 100 is described as a specific example of the first information processing device, and the HMD 200 is described as a specific example of the second information processing device, but the present invention is not limited to this. The first information processing device and the second information processing device may be any device that has a function capable of detecting operations on the device.

[0175] In the information processing system 900 of this embodiment, it goes without saying that the same functions and effects can be obtained even if the first information processing device and the second information processing device are information processing devices other than a smartwatch or an HMD. For example, the first information processing device may be a smartphone that is held in the hand, and the second information processing device may be a personal computer. In this case, for example, the smartphone may use a change in posture or movement as a trigger action, and the personal computer may detect the posture, line of sight, etc. of the user 800 viewing the smartphone as a trigger action, and operate the personal computer in accordance with the change in posture or movement of the smartphone.

[0176] <Variation 9> 10, the information processing system 900 may further include a third information processing device. In FIG. 10, the third information processing device is illustrated as a smartphone 300. The watch 100, the HMD 200, and the smartphone 300 are communicatively linked in advance.

[0177] In this case, the cooperation process may be two-way cooperation. Specifically, for example, when the watch 100 and the HMD 200 detect a predetermined first cooperation start trigger action at approximately the same time, cooperation between the watch 100 and the smartphone 300 is started, and when the watch 100 and the HMD 200 detect a predetermined second cooperation start trigger action at approximately the same time, cooperation between the HMD 200 and the smartphone 300 is started.

[0178] During linkage, for example, the smartphone 300 transmits operation commands to the watch 100 and the HMD 200, respectively. Alternatively, conversely, operation commands may be transmitted from the watch 100 or the HMD 200 to the smartphone 300. This allows the user 800 to use functions of the smartphone 300 by operating the watch 100 or the HMD 200, for example.

[0179] For example, the smartphone 300 can be operated while being placed in a pocket, a bag, or the like.

[0180] Alternatively, three-way cooperation may be used. In this case, for example, when the watch 100 and the HMD 200 detect a predetermined cooperation start trigger action at approximately the same timing, three-way cooperation between the watch 100, the HMD 200, and the smartphone 300 is started.

[0181] In three-way cooperation, for example, as in the above embodiment or modified example, cooperative processing is performed between the watch 100 and the HMD 200, all processing details performed by each information processing device are sent to the smartphone 300, and all processing histories are stored in the smartphone 300. Furthermore, three-way cooperation may be cooperation in which the screen of the smartphone 300 is transferred and displayed on the HMD 200, and the smartphone 300 is operated on the watch 100.

[0182] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0183] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0184] Some or all of the functions of the present invention described above may be implemented in hardware, for example, by designing them as integrated circuits. Furthermore, as described in the embodiments, they may be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program that implements each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Furthermore, some or all of the functions may be implemented by a server. Note that the server may be any type of server, as long as it can execute functions in cooperation with other components via communications. For example, the server may be a local server, a cloud server, an edge server, an online service, or the like. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0185] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0186] 100: Watch, 101: Processor, 102: Memory, 103: Bus, 111: Camera, 121: Display device, 122: First input I / F, 123: Second input I / F, 124: Audio input device, 125: Audio output device, 126: Vibration generating device, 127: Communication device, 130: Sensor, 131: EMG sensor, 132: Acceleration sensor, 133: Gyro sensor, 134: Geomagnetic sensor, 151: Motion detection unit, 152: Motion discrimination unit, 153: Switching control unit, 154: Watch operation control unit, 155: HMD operation command generation unit, 160: Watch standalone operation DB, 161: Detected operation, 162: Watch operation content, 170: HMD linked operation DB, 171: Detected operation, 172: HMD operation content, 200: HMD, 201: processor, 202: memory, 203: bus, 210: camera, 211: outer camera, 212: inner camera, 221: display device, 223: input I / F, 224: audio input device, 225: audio output device, 226: vibration generating device, 227: communication device, 230: sensor, 232: acceleration sensor, 233: gyro sensor, 234: geomagnetic sensor, 235: left eye gaze sensor, 236: right eye gaze sensor, 237: distance measurement sensor, 251: action detection unit, 252: action discrimination unit, 253: switching control unit, 254: HMD action control unit, 255: cooperation control unit, 260: HMD standalone operation DB, 261: detected action, 262: HMD operation content, 270: menu display area, 271: operation mode display area, 272: operation image display area, 300: Smartphone, 800: User, 802: Status, 803: Status, 804: Status, 811: Status, 812: Status, 821: Status, 822: Status, 831: Status, 832: Status, 900: Information Processing Systems

Claims

1. An information processing device, a communication unit for communicating with an external device; a gaze detection unit that detects a gaze direction of a user of the information processing device; a control unit, The control unit generating a first operation command for controlling an operation of the information processing device; controlling an operation of the information processing device in accordance with the generated first operation command; determining whether the line of sight direction detected by the line of sight detection unit is a direction toward the external device; when a determination that the line of sight direction is in the direction of the external device and a detection notification indicating the detection of a linkage trigger operation transmitted from the external device are received within a predetermined period, and when a second operation command is received from the external device via the communication unit, the information processing device controls an operation of the information processing device in accordance with the received second operation command; Information processing device.

2. 2. The information processing device according to claim 1, the control unit, when the determination and the reception of the detection notification occur within the predetermined period, transmits a command transmission request to the external device via the communication unit. Information processing device.

3. 2. The information processing device according to claim 1, if the determination and the reception of the detection notification do not occur within the predetermined period, even if the control unit receives the second operation command, it does not perform control in accordance with the received second operation command. Information processing device.

4. An information processing device, a communication unit for communicating with an external device; a motion detection unit that detects a motion of a user of the information processing device; a storage unit that stores the trigger operation; a control unit, The control unit determining whether the user's motion detected by the motion detection unit matches the trigger motion stored in the storage unit; when it is determined that the user's action does not match the trigger action, generating a first operation command for controlling an operation of the information processing device in accordance with the user's action, and controlling the operation of the information processing device in accordance with the generated first operation command; If it is determined that the user's action matches the trigger action, it is determined whether the determination of the match and the reception of a detection notification indicating the detection of the linked trigger action transmitted from the external device occur within a predetermined period of time; receiving a second operation command transmitted from the external device via the communication unit when it is determined that the determination of the match and the reception of the detection notification have occurred within the predetermined period of time; and controlling an operation of the information processing device in accordance with the received second operation command. Information processing device.

5. 5. The information processing device according to claim 4, When the control unit determines that the determination of the match and the reception of the detection notification do not occur within the predetermined period, even if the control unit receives the second operation command, it does not perform control in accordance with the received second operation command. Information processing device.

6. 5. The information processing device according to claim 4, the movement detection unit is a gaze detection unit that detects a gaze direction of a user, the trigger action stored in the storage unit is an action of the user directing their gaze in the direction of the external device; Information processing device.

7. 5. The information processing device according to claim 4, the storage unit stores a second trigger action; When the control unit determines that the user's action matches the second trigger action, even if the control unit receives the second operation command, the control unit does not perform control in accordance with the received second operation command. Information processing device.

8. 8. The information processing device according to claim 1, Further comprising a display unit, the control unit displays, on the display unit, an image indicating whether the operation of the information processing device is being controlled in accordance with the second operation command. Information processing device.

9. 8. The information processing device according to claim 1, Further comprising a display unit, when the control unit controls the operation of the information processing device in accordance with the second operation command, an image corresponding to the second operation command is displayed on the display unit. Information processing device.

10. 8. The information processing device according to claim 1, The information processing device is a device worn on the user's head when used. Information processing device.

Images