Device, information processing method, and program

Abstract

To improve usability in continuous pointer operation.SOLUTION: A device for operating an operation target apparatus having a display screen, performs: acquiring motion information relating to a motion of a predetermined part of a user; controlling a pointer based on an operation mode that is set to one of a movement mode in which a signal related to the movement of the pointer displayed on the display screen can be transmitted to the operation target apparatus based on the motion information, a restriction mode in which the movement of the pointer is restricted, and a press mode in which a signal for executing a press operation at a position of the pointer can be transmitted to the operation target apparatus 3 based on the motion information; setting the operation mode to the press mode; and after setting the operation mode to the press mode, setting the operation mode to one of the movement mode and the restriction mode based on whether or not information on a first motion for transmitting a press signal and information on a second motion after the first motion satisfy a first condition related to the motion of the predetermined part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a device, an information processing method, and a program. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a technique for controlling input operations such as click operations on an input means such as a mouse using a sensor that detects the movement of a predetermined part of the user (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-179927 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technology, pointer movement is temporarily restricted after a user performs a click or other operation to prevent unintended operations. Experiments and other tests on this technology have revealed a need for not only restricting pointer movement after a user-initiated operation, but also for allowing the user to continue moving the pointer and perform input operations such as clicking. This leaves room for improvement in the usability of pointer movement and input operations based on user actions.

[0005] Therefore, the disclosed technology aims to improve the usability of pointer movement and input operations based on user actions. [Means for solving the problem]

[0006] A device according to one aspect of the disclosed technology is a device for operating an operation target device having a display screen, and includes a processor. The processor acquires operation information regarding the operation of a predetermined part from a sensor attached to a predetermined part of the user's body, controls the pointer based on an operation mode set to one of: (1) a movement mode in which a movement signal related to moving a pointer displayed on the display screen on the display screen can be transmitted to the operation target device based on the operation information; (2) a restriction mode in which movement of the pointer on the display screen is restricted; and (3) a press mode in which a press signal related to performing a press operation at the position of the pointer on the display screen can be transmitted to the operation target device based on the operation information; sets the operation mode to the press mode; and, after setting the operation mode to the press mode, sets the operation mode after the press operation to either the movement mode or the restriction mode based on whether or not operation information acquired in the press mode, including operation information related to a first operation for transmitting a press signal and operation information related to a second operation after the first operation, satisfy a first condition related to the operation of the predetermined part. [Effects of the Invention]

[0007] According to the disclosed technology, it is possible to improve usability for pointer movement and input operations based on user actions. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an outline of the operation of the information processing device according to the embodiment. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of an external information processing device in the embodiment. [Figure 4] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing device according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of operation information in the embodiment. [Figure 6]FIG. 10 is a diagram showing an example of the relationship between data, movement, output, etc. in the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of the relationship between data, movement, output, etc. in the embodiment. [Figure 8] 10 is a flowchart illustrating an example of an operation of the information processing device according to the embodiment. [Figure 9A] FIG. 10 is a diagram showing an example of a display screen of an external information processing device in the embodiment. [Figure 9B] FIG. 10 is a diagram showing an example of a display screen of an external information processing device in the embodiment. [Figure 9C] FIG. 10 is a diagram showing an example of a display screen of an external information processing device in the embodiment. [Figure 9D] FIG. 10 is a diagram showing an example of a display screen of an external information processing device in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely examples, and are not intended to exclude various modifications or applications of techniques not explicitly described below. In other words, the present invention can be implemented with various modifications within the scope of its spirit. Furthermore, in the following description of the drawings, identical or similar parts are denoted by identical or similar reference numerals. The drawings are schematic and do not necessarily correspond to actual dimensions, ratios, etc. Parts in the drawings may have different dimensional relationships or ratios.

[0010] [Example] In the embodiments, an eyewear device is cited as an example of a device for operating an operation target device having a display screen, but is not limited thereto. The information processing device according to the embodiments may be a device integrated with eyewear, or an external device that can be attached to or detached from eyewear such as glasses that a user normally wears. The eyewear-integrated device may be, for example, a device built into eyewear, or a device that cannot be removed from eyewear (or may be damaged if removed). The device may be, for example, a wearable device such as a ring-type device or a wristwatch-type device, or a general-purpose information processing device such as a personal computer, a smartphone, or a tablet terminal.

[0011] Fig. 1 is a diagram illustrating an example of an information processing system 1 according to an embodiment. The information processing system 1 illustrated in Fig. 1 includes an eyewear device 2 and an operation target device 3 that is an object of operation by the eyewear device 2. The operation target device 3 and the eyewear device 2 are connected via a network, enabling data communication therebetween.

[0012] The eyewear device 2 includes an information processing device 22 mounted on a frame 20. The information processing device 22 includes a sensor. The information processing device 22 acquires, from the sensor, motion information relating to the motion of a predetermined part (in this case, the user's head) on which the eyewear device 2 is worn. The information processing device 22 transmits a signal for operating the operation target device 3 based on the motion information. That is, the eyewear device 2 allows the operation target device 3 to be operated based on the motion of the user's head. The frame 20 is an example of a main body of the eyewear device 2 that is configured to be wearable on the user's head.

[0013] In the following description, the predetermined part is assumed to be the user's head, but is not limited to this. The predetermined part varies depending on the part on which the device according to one aspect of the disclosed technology is worn. For example, if the device is ring-shaped, the predetermined part may be a finger, a hand, an arm, or the like.

[0014] The operation target device 3 is an information processing device having a communication function. The operation target device 3 is, for example, a server device, a personal computer, a tablet terminal, or a mobile terminal such as a smartphone. The operation target device 3 receives a signal from the information processing device 22 and executes a process corresponding to the signal.

[0015] In one embodiment, the operation target device 3 is communicatively connected to the information processing device 22, and thereby recognizes the information processing device 22 as an input device (e.g., a mouse). In one example, the operation target device 3 receives from the information processing device 22 a movement signal related to moving a pointer (cursor) on its own display screen. In another example, the operation target device 3 receives from the information processing device 22 a press signal related to performing a press operation (e.g., left click, right click, double click, scroll, drag and drop, etc.) at the position of the pointer on the display screen of the operation target 3.

[0016] 2 is a state transition diagram showing an outline of an example of operation of the information processing device 22. The information processing device 22 stores an operation mode that is set to one of a movement mode, a limit mode, and a press mode. The information processing device 22 executes various processes based on the set operation mode.

[0017] The movement mode is an operation mode in which, based on the operation information, a movement signal is transmitted to the operation target device 3. The operation target device 3 moves the pointer based on the received movement signal.

[0018] The press mode is an operation mode in which, based on the operation information, a press signal is transmitted to the operation target device 3. Based on the received press signal, the operation target device 3 executes a press operation at the position of the pointer.

[0019] The limited mode is an operation mode that limits the operation of the pointer. In one example, when the operation mode is set to the limited mode, the information processing device 22 does not transmit movement signals or press signals. In another example, when the operation mode is set to the limited mode, the information processing device 22 transmits signals for performing fewer types or fewer amounts of operations compared to when the operation mode is set to a mode other than the limited mode. In another example, when the operation mode is set to the limited mode, the information processing device 22 transmits a signal related to not moving the pointer.

[0020] The information processing device 22 changes the operation mode based on the operation information. The arrows in Fig. 2 indicate the transition of the operation mode according to one embodiment.

[0021] In one embodiment, when the operation mode is set to the restricted mode and it is determined based on the operation information that the user has nodded, the information processing device 22 changes the operation mode from the restricted mode to the movement mode (S1). This changes the state in which the user is able to move the pointer from a state in which the operation of the operation target device 3 is restricted. Note that whether the user has nodded is an example of a third condition, which will be described later.

[0022] In one embodiment, when the operation mode is set to the movement mode and it is determined based on the operation information that the user's head has been stopped for a predetermined time, the information processing device 22 changes the operation mode from the movement mode to the press mode (S2). This changes the state in which the user can move the pointer to a state in which the user can perform a press operation at the pointer position. Note that whether the user's head has been stopped for a predetermined time is an example of a second condition, which will be described later.

[0023] In one embodiment, when the operation mode is set to the press mode and then a press operation based on a user action described later is performed, the information processing device 22 changes the operation mode to one of the limit mode and the movement mode based on whether a predetermined condition is satisfied (S3 / S4). At this time, if a predetermined condition (a first condition described later) related to changing the operation mode to the movement mode is satisfied, the information processing device 22 changes the operation mode from the press mode to the movement mode. If it is determined that the predetermined condition is not satisfied, the information processing device 22 changes the operation mode from the press mode to the limit mode.

[0024] In one example, the predetermined condition is that the user's head moves in a first direction and then moves in a second direction opposite to the first direction. That is, when it is determined based on the motion information that the user's head moves in the first direction and then moves in a second direction opposite to the first direction (the user's head has returned to its original position), the information processing device 22 changes the operation mode from the press mode to the move mode (S3). On the other hand, when it is determined based on the motion information that the user's head moves in the first direction and then does not move in the second direction opposite to the first direction (for example, the user's head moves in the first direction and then stops for a predetermined period of time), the information processing device 22 changes the operation mode from the press mode to the limit mode (S4).

[0025] In the disclosed technology, the movement of a predetermined part in a certain direction may be a translation or a rotation.

[0026] For example, Patent Document 1 describes a technology in which a mode is set in which pointer movement is restricted each time a pressing operation is performed at the pointer position. This technology can prevent erroneous operations by the user after a pressing operation is performed at the pointer position. Regarding operations after a pressing operation, there is a need for users to be able to operate the pointer continuously even after a pressing operation (for example, to move the pointer immediately after performing a pressing operation to perform a new input operation). Therefore, the disclosed technology satisfies this need and aims to improve usability.

[0027] According to the disclosed technology, when a user performs a pressing operation at the position of the pointer, the user can select, based on the user's own behavior, whether to change the operation mode to a movement mode to continuously operate the pointer, or to a restricted mode to interrupt the operation. This makes it possible to improve usability for continuously operating the pointer while suppressing user operation errors.

[0028] <Configuration of operation target device 3> 3 is a block diagram showing an example of the configuration of the operation target device 3 in the embodiment. The operation target device 3 includes one or more processing units (CPUs) 110, one or more communication interfaces 120, a memory 130, a user interface 150, an image sensor 160, and one or more communication buses 170 for interconnecting these components.

[0029] The communication interface 120 is capable of performing data communication with the information processing device 22 via a wired LAN cable, a USB cable, a mobile communication antenna, and a wireless LAN communication antenna.

[0030] In one embodiment, the communication interface 120 includes a power supply interface 122. The power supply interface 122 is configured to supply power to other devices connected via the communication interface 120.

[0031] The user interface 150 includes a display device and an input device (such as a keyboard and / or a mouse or some other pointing device). The user interface 150 allows the user to move and operate a pointer displayed on the display device. The imaging sensor 160 is an image sensor that receives optical signals and converts the optical signals into electrical signals to generate images. The images include at least one of still images and moving images.

[0032] Memory 130 may be, for example, a high-speed random access memory such as DRAM, SRAM, or other random access solid-state memory, or may be a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices, or may be a non-transitory computer-readable recording medium.

[0033] The memory 130 stores data used by the information processing system 1. For example, the memory 130 stores data transmitted from the eyewear device 2 (information processing device 22).

[0034] Memory 130 may also be one or more storage devices located remotely from CPU 110. In one embodiment, memory 130 stores programs, modules, and data structures, or a subset thereof, that are executed by CPU 110.

[0035] The CPU 110 executes the programs stored in the memory 130 to configure an acquisition unit 112 and a process control unit 114 .

[0036] The acquisition unit 112 acquires a movement signal, a press signal, and the like from the information processing device 22 of the eyewear device 2. The movement signal may include information regarding the movement amount of the pointer. The information regarding the movement amount of the pointer may be expressed, for example, as a relative movement amount with respect to the current position of the pointer on the display screen. In the disclosure, the movement signal is displayed in the form of "movement amount (α, β)." This corresponds to moving the pointer from its current position by α along the x-axis direction and by β along the y-axis direction. In this case, a movement signal for not moving the pointer can be displayed as movement amount (0, 0). Note that this display format is adopted for convenience of explanation and does not limit the actual signal format. The press signal may also include information regarding a press operation to be performed at the pointer position on the display screen.

[0037] The process control unit 113 executes a process according to the signal acquired by the acquisition unit 112. The process control unit 113 moves the pointer based on the movement signal. The process control unit 113 executes a pressing operation corresponding to the pressing signal based on the pressing signal.

[0038] Furthermore, the operation target device 3 may be installed with a program for performing display in accordance with a UI assistance signal, which will be described later.

[0039] <Configuration of information processing device 22> Fig. 4 is a block diagram showing an example of the configuration of the information processing device 22 in this embodiment. As shown in Fig. 4, the information processing device 22 includes a processing unit 200, a communication interface 210, a display unit 220, and a six-axis sensor 230. The information processing device 22 also includes a memory that stores processing data. This memory may be a computer-readable non-transitory recording medium.

[0040] The six-axis sensor 230 includes a three-axis acceleration sensor and a three-axis angular velocity sensor (or a three-axis angular acceleration sensor). The six-axis sensor 230 is provided in the information processing device 22 on the eyewear device 2. Here, the eyewear device 2 is worn on the user's head. Therefore, the six-axis sensor 230 generates motion information related to the motion of the user's head. An example of the motion information is measured values ​​of acceleration and angular velocity at a predetermined sampling rate. The six-axis sensor 230 outputs the generated motion information to the processing unit 200.

[0041] As described above, the information processing device 22 is mounted on the eyewear device 2 that is worn on the head. The head is a part of the human body where unintended vibrations by the user are unlikely to occur. Therefore, with this configuration, noise contained in the motion information generated by the six-axis sensor 230 can be suppressed.

[0042] The processing unit 200 includes an acquisition unit 202, a mode control unit 204, and an output unit 208. The processing unit 200 includes a processor.

[0043] The acquisition unit 202 acquires motion information from the six-axis sensor 230. In one embodiment, the motion information includes measurement values ​​by the six-axis sensor 230 at each of a plurality of time points. The acquisition unit 202 may acquire motion information including a measurement value sequentially each time the six-axis sensor 230 generates the measurement value, or may acquire motion information including a plurality of measurement values ​​generated by the six-axis sensor 230 within a predetermined period. Acquiring the motion information by the acquisition unit 202 includes acquiring the motion information directly or indirectly from the six-axis sensor 230.

[0044] The mode control unit 204 sets the operation mode to one of the movement mode, the limit mode, and the press mode. The mode control unit 204 includes a determination unit 206.

[0045] (Judgment unit 206) In one embodiment, the determination unit 206 determines whether the motion information acquired by the acquisition unit 202 satisfies a first condition, a second condition, and a third condition related to the user's head motion. In one embodiment, whether the motion information satisfies the first condition is determined based on whether the user's head motion identified based on the motion information corresponds to the first condition. Similarly, in one embodiment, whether the motion information satisfies the second condition is determined based on whether the user's head motion identified based on the motion information corresponds to the second condition. Similarly, in one embodiment, whether the motion information satisfies the third condition is determined based on whether the user's head motion identified based on the motion information corresponds to the third condition. Note that the motion corresponding to the first condition, the motion corresponding to the second condition, and the motion corresponding to the third condition may be different from one another, and each may include one or more steps (e.g., turning the head left and then returning it to the front).

[0046] In one embodiment, whether or not a user's head movement identified based on the movement information is a predetermined movement (for example, a movement corresponding to any one of the first to third conditions) can be determined based on the direction of the user's head rotation identified based on the movement information, as well as the transition and cumulative value of the amount of rotation. In one example, the determination unit 206 can determine that the user has turned toward the first direction based on the fact that the user's head rotation identified based on the movement information within a predetermined time period is along the first direction and the cumulative value of the amount of rotation exceeds a predetermined threshold (for example, 30 degrees). In another example, the determination unit 206 can determine that the user has shook their head based on the fact that the user's head movement identified based on the movement information within a predetermined time period rotates along the first direction until it exceeds a predetermined threshold, and then rotates along a second direction opposite to the first direction until it exceeds a predetermined threshold.

[0047] (Mode control unit 204) The mode control unit 204 changes the operation mode based on the determination result by the determination unit 206 .

[0048] In one embodiment, when the operation mode is set to the press mode, the operation information acquired in the press mode, including operation information related to a first operation for transmitting a press signal and operation information related to a second operation following the first operation, determines whether or not a first condition related to the operation of the user's head is satisfied. The mode control unit 204 sets the operation mode after the press operation to either the movement mode or the limit mode based on the determination result of the first condition by the determination unit 206 (see S3 / S4 in FIG. 2). In one example, when the mode control unit 204 determines that the operation information satisfies the first condition, the mode control unit 204 sets the operation mode after the press operation to the movement mode, and when the mode control unit 204 determines that the operation information does not satisfy the first condition, the mode control unit 204 sets the operation mode after the press operation to the limit mode.

[0049] The first and second actions do not have to be consecutive. For example, the "second action after the first action" may be a "second action immediately after the first action," a "second action after another action after the first action," or a "second action performed within a predetermined period after the first action."

[0050] In another embodiment, when the operation mode is set to the press mode, when the output unit 208 described later transmits a press signal based on the operation information, the judgment unit 206 judges whether the operation information (i.e., the operation information that is the source of the press signal) satisfies a first condition regarding the operation of the user's head.

[0051] In addition, in the disclosed technology, "performing a second process when executing a first process" means that the order in which the first process and the second process are performed can be arbitrary, as long as they are performed consecutively or integrally.

[0052] In one embodiment, the first condition includes a condition that the motion information regarding the first motion (motion information for transmitting a press signal) includes information indicating that the user's head has moved along a first direction, and the motion information regarding the second motion (motion information after the first motion) includes information indicating that the user's head has moved along a second direction. That is, the first condition may include a condition that the user's head further moves along the second direction after transmitting a press signal by moving the user's head in the first direction. In one embodiment, the first condition further includes a condition that the period from the user's head moving along the first direction to the user's head further moving along the second direction is equal to or less than a predetermined threshold for the continuity between the motion along the first direction and the motion along the second direction.

[0053] In one embodiment, the second direction is the opposite direction to the first direction. In one example, a movement of the user's head along the first direction followed by a further movement along the opposite second direction is, for example, the user nodding (i.e., looking down and then facing forward), the user looking to the left and then facing forward, the user looking to the right and then facing forward, the user looking up and then facing forward, etc. In contrast, a movement of the user's head along the first direction followed by no movement along the opposite second direction is, for example, the user stopping their head while facing left, etc.

[0054] As mentioned above, other actions may be included between the first action and the second action, so for example, a user turning to the left (first action), shaking their head up and down (other action), and then turning to face forward (second action) may be included in the user turning to the left and then turning to face forward.

[0055] In one embodiment, when the operation mode is set to the movement mode, the determination unit 206 determines whether the motion information acquired in the movement mode satisfies a second condition related to the user's head motion. When the determination unit 206 determines that the motion information satisfies the second condition, the mode control unit 204 sets the operation mode to the press mode (see S2 in FIG. 2). In one embodiment, the second condition includes that the motion information indicates that the user's head has been stationary for a predetermined period of time. Note that the second condition exemplified here is merely an example, and the user's head motion related to the second condition can be set arbitrarily.

[0056] In one embodiment, when the operation mode is set to the restricted mode, the determination unit 206 determines whether or not the motion information acquired in the restricted mode satisfies a third condition related to the user's head motion. When the determination unit 206 determines that the motion information satisfies the third condition, the mode control unit 204 sets the operation mode to the movement mode (see S1 in FIG. 2). In one embodiment, the third condition includes the determination unit 206 indicating that the user's head has nodded. Note that the third condition exemplified here is merely an example, and the user's head motion related to the third condition can be set arbitrarily.

[0057] The output unit 208 may output a movement signal and an operation signal based on the operation mode set by the mode control unit 204 and the operation information acquired by the acquisition unit 202. Specifically, when the operation mode is set to the movement mode, the output unit 208 outputs a movement signal based on the operation information. Furthermore, when the operation mode is set to the press mode, the output unit 208 outputs a press signal based on the operation information. Furthermore, when the operation mode is set to the restricted mode, the output unit 208 restricts the output of the movement signal and the operation signal. Outputting a movement signal may mean outputting a signal of a movement amount (α, β) (where at least one of α and β is not zero). Furthermore, restricting the output of a movement signal may mean outputting a signal of a movement amount (0, 0).

[0058] In one embodiment, the output unit 208 outputs a movement signal that moves a pointer to follow the user's head movement (or line of sight) identified based on the movement information. In one example, when it is identified based on the movement information that the user's head has rotated along the left direction, the output unit 208 transmits a movement signal that moves the pointer leftward on the display screen of the operation target device 3. In another example, when it is identified based on the movement information that the user's head has rotated along the diagonally upper right direction, the output unit 208 transmits a movement signal that moves the pointer diagonally upper right on the display screen of the operation target device 3. This configuration allows the user to move the pointer in an intuitive manner.

[0059] In one embodiment, the output unit 208 outputs, based on first motion information, a press signal for causing a first press operation corresponding to the first motion information to be performed at the position of the pointer, and outputs, based on second motion information, a press signal for causing a second press operation corresponding to the second motion information to be performed at the position of the pointer. In one example, the output unit 208 outputs a press signal for causing a left click (an example of a press operation) to be performed at the position of the pointer, based on motion information indicating a motion in which the user's head turns downward and then faces forward. In another example, the output unit 208 outputs a press signal for causing a right click (an example of a press operation) to be performed at the position of the pointer, based on motion information indicating a motion in which the user's head turns right and then faces forward. This configuration allows the user to easily perform multiple different press operations at the position of the pointer.

[0060] Examples of the correspondence between the user's head movements and pressing operations are as follows (1) to (6): In each row, the left side indicates the user's head movement, and the right side indicates the pressing operation that is executed at the pointer position when it is determined that the movement has been performed based on the movement information. (1) Look down...Left click (2) To face upwards: double click (3) Turn right…right click (4) Turn left...Start drag and drop (5) Tilt to the right...scroll down (6) Tilt left: Scroll up

[0061] In one embodiment, the output unit 208 outputs a UI (user interface) assistance signal. The UI assistance signal is a signal that displays information to assist the user on the display screen of the operation target device 3. The UI assistance signal may be information for displaying information about an operation method when the operation mode is set to the press mode on the display screen of the operation target device 3. In one example, the UI assistance signal is a signal that displays on the display screen of the operation target device 3 whether the currently set operation mode is the move mode, the press mode, or the limit mode. In another example, the UI assistance signal is information that displays on the display screen of the operation target device 3 the relationship between operation information and the press operation corresponding to the operation information. Examples of screens based on the UI assistance signal will be described later with reference to FIG. 9.

[0062] The processing unit 200 can packetize the movement signal, operation signal, etc., and output the packetized signals to the communication interface 210 via the output unit 208 .

[0063] The display unit 220 may be an LED light, a liquid crystal display, or the like. The display unit 220 displays characters, colors, symbols, and the like based on at least a part of the information output by the output unit 208. The display unit 220 is an example of an output device included in the eyewear device 2.

[0064] The communication interface 210 transmits the movement signal, operation signal, UI assistance signal, etc. output by the output unit 208 to the operation target device 3. The communication interface 210 performs data communication with the operation target device 3 via a wired LAN cable, a USB cable, a mobile communication antenna, and a wireless LAN communication antenna.

[0065] In one embodiment, the communication interface 210 includes a power supply interface 212. The power supply interface 212 is configured to receive power from another device connected via the communication interface 210.

[0066] In one embodiment, the eyewear device 2 does not include a battery for power. In this case, the information processing device 22 may receive power from the operation target device 3 via the power supply interface 212. This configuration allows the eyewear device 2 to be lightweight.

[0067] <Data example> FIG. 5 is a diagram showing an example of motion information in an embodiment. The motion information shown in FIG. 5 represents data acquired at a predetermined sampling rate (for example, 20 Hz = 0.05 seconds). D_1, D_2, ..., D_60, etc. may each be a vector formed by measurements taken by the six-axis sensor 230. In one example, this vector includes measurements of acceleration and angular velocity. The data shown in FIG. 5 is stored, for example, in a memory in the information processing device 22. Note that the motion information may further include information regarding the movement speed of the gaze based on an electro-oculography signal.

[0068] <Example 1> 6 and 7 are diagrams showing an example of the relationship between the motion information, the motion of the user's head at that time, the set operation mode, and the information transmitted from the information processing device 22 to the operation target device 3 at each of a plurality of points in time. Fig. 6 shows an example of the operation when it is determined that the first condition is satisfied when a press signal is transmitted, and the mode control unit 204 changes the operation mode from the press mode to the move mode. In contrast, Fig. 7 shows an example of the operation when it is determined that the first condition is not satisfied when a press signal is transmitted, and the mode control unit 204 changes the operation mode from the press mode to the limit mode.

[0069] First, assume that the operation mode is set to the limited mode at time T_000, which is the initial state. Therefore, from time T_000 to time T_150, which is immediately before the operation mode is changed, the output unit 208 does not transmit a movement signal or a press signal, or transmits a signal with a movement amount of (0,0) regardless of the user's head movement.

[0070] Next, it is assumed that the user nods from time T_105 to time T_150. In this case, the determination unit 206 determines that the user nods based on the motion information (D_105 to D_150) from time T_105 to time T_150. Furthermore, based on the determination that the user nods, the determination unit 206 determines that the third condition (i.e., the condition for changing the operation mode from the restricted mode to the moving mode) is satisfied. Based on the determination that the third condition is satisfied, the mode control unit 204 sets the operation mode to the moving mode. Note that the motion information (D_105 to D_150) from time T_105 to time T_150 is an example of motion information acquired in the restricted mode.

[0071] Assume that the user moves his / her head to move the pointer from time T_155 to time T_250. At this time, the operation mode is set to the movement mode, so the output unit 208 continuously transmits a movement signal based on the movement information (D_155 to D_250) acquired during this period.

[0072] It is assumed that the user stops moving his / her head from time T_255 to time T_350. As a result, the output unit 208 stops transmitting the movement signal during this period. Note that the output unit 208 may transmit a movement signal (0,0) instead of stopping the transmission of the movement signal.

[0073] Furthermore, the determination unit 206 determines that the user's head has been stationary for a predetermined period based on the motion information (D_255 to D_350) from time T_255 to time T_350. Furthermore, based on the determination that the user's head has been stationary for a predetermined period, the determination unit 206 determines that a second condition (i.e., a condition for changing the operation mode from the movement mode to the press mode) has been satisfied. Based on the determination that the second condition has been satisfied, the mode control unit 204 sets the operation mode to the press mode. Note that the motion information (D_255 to D_350) from time T_255 to time T_350 is an example of motion information acquired in the movement mode.

[0074] Next, from time T_355 to time T_400, the user faces downward. The determination unit 206 determines that the user's head has moved in a downward direction based on the motion information (D_355 to D_400) from time T_355 to time T_400. Based on the determination that the user's head has moved in a downward direction, the output unit 208 transmits a press signal for causing a press operation (e.g., left click) corresponding to the motion to be performed at the position of the pointer. Note that the motion information (D_355 to D_400) from time T_355 to time T_400 is an example of motion information related to a first motion for transmitting a press signal, among the motion information acquired in the press mode.

[0075] Immediately thereafter, from time T_405 to time T_450, the user faces forward. The determination unit 206 determines that the user's head has moved in an upward direction based on the motion information (D_405 to D_450) from time T_405 to time T_450. Note that this motion is not associated with a press operation, and the output unit 208 does not transmit a press signal. Note that the motion information (D_405 to D_450) from time T_405 to time T_450 is an example of motion information related to a second motion after a first motion, among the motion information acquired in the press mode.

[0076] Based on a series of movements of the user's head from time T_355 to time T_450, the determination unit 206 determines that the user has nodded. Furthermore, based on the determination that the user has nodded, the determination unit 206 determines that a first condition (i.e., a condition for changing the operation mode from the press mode to the movement mode) is satisfied. The mode control unit 204 sets the operation mode to the movement mode based on the determination that the first condition is satisfied when transmitting the press signal. Note that, for simplicity, in this example, a case has been described in which the user performs a second movement (face forward) immediately after a first movement (face down), but other movements may be included between the first movement and the second movement, and even in that case, the determination result may be the same.

[0077] From time T_455 onwards, the operation mode is set to the movement mode, so the user can move the pointer by moving his or her head.

[0078] Next, FIG. 7 will be described. The description of FIG. 7 will focus on the differences from FIG. 6. In FIG. 7, it is assumed that the user stops moving his / her head from time T_405 immediately after the user turns downward to time T_450. The determination unit 206 determines that the user's head did not move in an upward direction based on the motion information (D_405 to D_450) from time T_405 to time T_450. As a result, the determination unit 206 determines that the first condition is not satisfied. The mode control unit 204 sets the operation mode to the restricted mode based on the determination that the first condition is not satisfied when transmitting the press signal.

[0079] After time T_455, the operation mode is set to the restricted mode, and therefore the output unit 208 does not transmit any movement signal or press signal regardless of the movement of the user's head.

[0080] Comparing Figures 6 and 7, the actions up to the point where the user performs the action of transmitting a press signal are the same, but the subsequent movements of the user's head are different. Specifically, in the example of Figure 6, the user looks downward and then faces forward. As a result, it is determined that the first condition is met, and the operation mode is changed from press mode to movement mode. On the other hand, in the example of Figure 7, the user looks downward and then stops moving their head. As a result, it is determined that the first condition is not met, and the operation mode is changed from press mode to limit mode. In other words, from the user's perspective, when transmitting a press signal by looking downward, they can choose whether to continue moving the pointer or interrupt pointer operation by turning their head forward or keeping their head still.

[0081] <Example 2> 8 is a flowchart showing an example of the operation of the information processing device 22. It is assumed that the operation mode is set to the restricted mode in the initial state. First, in step S100, the acquisition unit 202 acquires movement information. In step S102, the determination unit 206 determines whether the head movement of the user identified based on the movement information is a nodding movement. If the determination in step S102 is NO, the acquisition unit 202 continues to acquire movement information in step S100. Whether the head movement of the user is a nodding movement is an example of a third condition.

[0082] On the other hand, if the determination in step S102 is YES, the mode control unit 204 sets the operation mode to the movement mode, and in step S200, the output unit 208 transmits a UI assistance signal to the operation target device 3. Upon receiving this UI assistance signal, the operation target device 3 displays information that the currently set operation mode is the movement mode. This information may be text information (see FIG. 9), or may be displayed by a predetermined movement such as a pointer drawing a circle.

[0083] Next, in step S202, the acquisition unit 202 acquires motion information. In step S204, the determination unit 206 determines whether the user's head has been stationary for a predetermined period of time based on the motion information. If the determination in step S204 is NO (i.e., the user's head has been moving), in step S206, the output unit 208 transmits a movement signal based on the motion information. Whether the user's head has been stationary for a predetermined period of time is an example of a second condition.

[0084] On the other hand, if the determination in step S204 is YES, the mode control unit 204 sets the operation mode to the press mode, and in step S300, the output unit 208 transmits a UI assistance signal to the operation target device 3. Upon receiving this UI assistance signal, the operation target device 3 displays information that the currently set operation mode is the press mode. Furthermore, upon receiving this UI assistance signal, the operation target device 3 displays a plurality of movements of the user's head and information related to the press operations associated with each of the plurality of movements.

[0085] Next, in step S302, the acquisition unit 202 acquires motion information. In step S304, the determination unit 206 determines whether the user's head has moved along the first direction based on the motion information. If the determination in step S304 is NO, in step S305, it is further determined whether the time limit for detecting the user's head motion has timed out. If the determination in step S305 is NO, in step S302, the acquisition unit 202 continues to acquire motion information. If the determination in step S305 is YES, the mode control unit 204 sets the operation mode to the limited mode, and in step S104, the output unit 208 transmits a UI assistance signal to the operation target device 3.

[0086] On the other hand, if the determination in step S304 is YES, in step S306, the output unit 208 transmits a press signal for causing a press operation corresponding to a movement of the user's head along the first direction to be performed at the position of the pointer. In this case, the motion information acquired in step S302 can be said to be an example of motion information related to the first motion for transmitting a press signal, among the motion information acquired in the press mode.

[0087] After that, in step S307, the acquiring unit 202 further acquires motion information. The motion information acquired here is an example of motion information related to a second motion after a first motion, among the motion information acquired in the press mode.

[0088] In step S308, the determination unit 206 determines whether the user's head has moved in a first direction and then moved in a second direction opposite to the first direction, and whether the user's head has been stationary for a predetermined period of time, based on the motion information acquired in steps S302 and S307. Note that the user's head moving in the first direction and then moving in the second direction opposite to the first direction is an example of the first condition.

[0089] If it is determined in step S308 that the user's head has moved in the second direction, the mode control unit 204 sets the operation mode to the movement mode, and in step S200, the output unit 208 transmits a UI assistance signal to the operation target device 3. Upon receiving this UI assistance signal, the operation target device 3 displays information that the currently set operation mode is the movement mode.

[0090] On the other hand, if it is determined in step S308 that the user's head has stopped for the predetermined period of time, the mode control unit 204 sets the operation mode to the restricted mode, and in step S104, the output unit 208 transmits a UI assistance signal to the operation target device 3. Upon receiving this UI assistance signal, the operation target device 3 displays information that the currently set operation mode is the restricted mode.

[0091] If it is determined in step S308 that the user's head is not moving in the second direction and is not stopped, the motion information continues to be acquired in step S307.

[0092] In this way, according to the information processing device 22, the user can operate the operation target device 3 while appropriately switching the operation mode among the limit mode, the movement mode, and the press mode. In particular, when the operation mode is set to the press mode, the user can transmit a press signal (see step S306) based on the movement of his or her head (see step S302) and select whether to set the operation mode to the limit mode (see step S308 NO) or the movement mode (see step S308 YES).

[0093] The information processing device 22 may execute some of the processes (for example, acquisition of motion information) in parallel or in parallel with other processes.

[0094] <Screen example> An example of a display screen of the operation target device 3 that has received a UI assistance signal will be described with reference to FIG.

[0095] Fig. 9A shows an example of a display screen when the operation mode is set to the movement mode. The display screen as shown in Fig. 9A may be displayed on the operation target device 3 between time T_155 and time T_250 in Figs. 6 and 7. The display screen as shown in Fig. 9A may be displayed on the operation target device 3 that has received a UI assistance signal in accordance with step S200 in Fig. 8.

[0096] The example display screen in Fig. 9A displays icons d100 to d107, a pointer d108, and an operation mode d110. The icons d100 to d107 are UI elements corresponding to, for example, folders, files, etc. The operation mode d110 is an element indicating the currently set operation mode, and is displayed as "movement mode" in Fig. 9A.

[0097] 9B shows an example of a display screen when the operation mode is set to the press mode because it is determined that the user's head has been stationary for a predetermined period of time (i.e., because it is determined that the second condition is satisfied). The display screen shown in FIG. 9B may be displayed, for example, on the operation target device 3 immediately after time T_350 in FIGS. 6 and 7. The display screen shown in FIG. 9B may also be displayed, for example, on the operation target device 3 that has received a UI assistance signal in accordance with step S300 in FIG. 8.

[0098] In addition to the elements displayed in the example display screen of Fig. 9A, the example display screen of Fig. 9B displays a front icon d200, a left-click icon d202, a drag-and-drop icon d204, a double-click icon d206, and a right-click icon d208. Also, in the example display screen of Fig. 9B, "press mode" is displayed in the operation mode d110.

[0099] The left click icon d202 is displayed below the front icon d200. This indicates that when the user looks downward, a press signal is sent to perform a left click at the position of the pointer d108.

[0100] The drag-and-drop icon d204 is displayed to the left of the front icon d200. This indicates that when the user faces left, a press signal is sent to start a drag-and-drop operation from the position of the pointer d108.

[0101] The double click icon d206 is displayed above the front icon d200. This indicates that when the user looks upward, a press signal is sent to cause a double click to be performed at the position of the pointer d108.

[0102] The right click icon d208 is displayed to the right of the front icon d200. This indicates that when the user faces right, a press signal is sent to execute a right click at the position of the pointer d108.

[0103] Fig. 9C shows an example of a display screen when the user faces downward after the operation mode is set to the press mode. The display screen as shown in Fig. 9C may be displayed on the operation target device 3 between time T_355 and time T_400 in Figs. 6 and 7. The display screen as shown in Fig. 9C may also be displayed on the operation target device 3 in response to a determination of YES in step S304 in Fig. 8.

[0104] In the example display screen of Figure 9C, the left click icon d202 is highlighted, indicating that it has been determined that the user is looking downward and that the conditions for performing a left click at the position of the pointer d108 have been met.

[0105] 9D shows an example of a display screen when the user faces downward and then faces forward, i.e., when the user moves in a first direction and then moves in a second direction opposite to the first direction. The display screen shown in FIG. 9D may be displayed on the operation target device 3 between time T_405 and time T_450 in FIG. 6, for example. The display screen shown in FIG. 9D may also be displayed on the operation target device 3 in response to a determination of YES in step S308 in FIG. 8, for example.

[0106] 9D, the front icon d200 is highlighted. This indicates that the user has faced forward after facing downward, and the condition (first condition) for changing the operation mode from the press mode to the move mode is met.

[0107] With this configuration, the user can easily understand what operations he or she can perform.

[0108] Although the present invention has been described above using examples, the technical scope of the present invention is not limited to the scope described in the above examples. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above examples. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0109] <Variation 1> In the above-described embodiment, an example has been described in which the information processing device 22 on the eyewear device 2 side performs operations from acquiring operation information to outputting a press signal, and the operation target device 3 executes a press operation based on the received press signal, but this is not limiting. For example, the information processing device 22 may transmit operation information to the operation target device 3, and the operation target device 3 may generate a press signal based on the received operation information. In this case, the operation target device 3 may have the function of the processing unit 200 described above, and may output a press signal to an OS (Operating System) or the like to control the movement and operation of a pointer on the screen.

[0110] <Variation 2> In Modification 2, the processing contents described above in the embodiment may be realized by an information processing device in which the information processing device 22 and at least a portion of the configuration of the operation target device 3 are integrally formed. For example, a head-mounted display may have the processing unit 200 of the information processing device 22 and the display 151 of the operation target device 3, and a pointer corresponding to the movement of the user's head may be superimposed on an image displayed on the display 151, allowing the above-described input operation to be performed.

[0111] <Variation 3> In Modification 3, an imaging sensor 160 is used as a sensor capable of measuring the movement of a predetermined part of the user's body. In Modification 3, the operation target device 3 is the above-mentioned information processing device, and the CPU 110 has a processing unit 200. As a specific example, the movement of a predetermined part of the user's body (such as the head) is photographed using a live view mode of the imaging sensor 160, and data related to the movement of the predetermined part recognized by image recognition is acquired by the processing unit 200. The subsequent processing is the same as the processing in the embodiment. As a result, by introducing the above-mentioned non-movement mode using a personal computer or the like equipped with a camera, it is possible to more appropriately realize the operation intended by the user, thereby improving usability.

[0112] <Variation 4> In the above embodiment, an example has been described in which the information processing device 22 transmits a UI assistance signal to the operation target device 3, and whether the operation mode is set to the limit mode, the move mode, or the press mode is displayed on the operation target device 3, but this is not limiting. The processing unit 200 of the information processing device 22 may transmit a UI assistance signal to the display unit 220. In this case, the display unit 220 may display whether the operation mode is set to the limit mode, the move mode, or the press mode by, for example, the color of light emitted by an LED light. This configuration allows the user to more easily understand the currently set operation mode.

[0113] <Variation 5> The information processing device 22 may be configured to include a speaker and to output a sound corresponding to the type of the press signal from the speaker when the press signal is transmitted to the operation target device 3. With this configuration, the user can also understand from the sound that a press operation is being performed.

[0114] <Variation 6> The information processing device 22 may be configured to be detachable from the frame 20 of the eyewear device 2. There are no limitations on the structure that allows the information processing device 22 to be detachable from the frame 20. In one example, the information processing device 22 is attached by clamping the temples of the frame 20 with clips provided on the housing of the information processing device 22 itself, or by providing mounting holes through which the temples are inserted. In another example, the information processing device 22 is attached to the frame 20 that has a magnetic material by magnetic adhesion. In another example, the information processing device 22 is attached by being adhered to the frame 20 with adhesive, tape, or the like. In another example, the information processing device 22 has a band, and the information processing device 22 is attached by wrapping the band around the frame 20.

[0115] <Variation 7> The information processing device 22 may further include a bioelectrode. The bioelectrode may be provided on the information processing device 22 so as to come into contact with the surface of the user's skin when the user wears the eyewear device 2. The bioelectrode is electrically connected to the processing unit 200, for example, via an amplifier. The acquisition unit 202 may acquire an electro-oculography signal obtained from the bioelectrode. The output unit 208 may output the electro-oculography signal obtained from the bioelectrode as needed. In one example, the bioelectrode is used to detect whether the user is wearing the eyewear device 2. In another example, the bioelectrode is used to measure the amount of activity of the user himself or herself or near the eyes. [Explanation of symbols]

[0116] 1...information processing system, 2...eyewear device, 3...operation target device, 20...frame, 22...information processing device, 110...processing device (CPU), 112...acquisition unit, 113...processing control unit, 114...processing control unit, 120...communication interface, 122...power supply interface, 130...memory, 150...user interface, 151...display, 160...imaging sensor, 170...communication bus, 200...processing unit, 202...acquisition unit, 204...mode control unit, 206...determination unit, 208...output unit, 210...communication interface, 212...power supply interface, 220...display unit, 230...axis sensor

Claims

1. A device for operating an operation target device having a display screen, a processor, the processor comprising: acquiring, from a sensor attached to a predetermined part of the user's body, motion information relating to the motion of the predetermined part; (1) a movement mode in which a movement signal relating to moving a pointer displayed on the display screen on the display screen can be transmitted to the operation target device based on the operation information; (2) a limit mode that limits movement of the pointer on the display screen; (3) a press mode in which a press signal for executing a press operation at the position of the pointer on the display screen can be transmitted to the operation target device based on a part of the operation information; controlling the pointer based on an operation mode set to any one of the above; setting the operation mode to the pressing mode; after setting the operation mode to the press mode, setting the operation mode after the press operation to one of the movement mode and the limit mode based on whether or not the operation information acquired in the press mode, including the operation information related to a first operation for transmitting the press signal and the operation information related to a second operation after the first operation, satisfy a first condition related to the operation of the predetermined part; The device that runs it.

2. The processor further comprises: The device according to claim 1 , wherein after the operation mode is set to the movement mode, the operation mode is set to the press mode based on the fact that the operation information acquired in the movement mode satisfies a second condition regarding the operation of the specified part.

3. The processor further comprises: The device according to claim 2 , wherein after the operation mode is set to the restricted mode, the operation mode is set to the movement mode based on the fact that the motion information acquired in the restricted mode satisfies a third condition regarding the motion of the specified part.

4. The processor further comprises: The device according to claim 2 , further comprising: a display unit configured to display information on the display screen relating to an operation method in the press mode when the operation mode is set to the press mode;

5. The processor further comprises: The device according to claim 1, which transmits information indicating at least one of whether the operation mode is set to the movement mode, whether the operation mode is set to the restriction mode, and whether the operation mode is set to the press mode, for output on at least one of the display screen and an output device provided in the device.

6. The first condition is The motion information regarding the first motion includes information indicating that the predetermined part has moved along a first direction, The motion information regarding the second motion includes information indicating that the predetermined part has moved along a second direction. including conditions relating to The device of claim 1 , wherein the processor sets the operation mode to the travel mode when the processor determines that the motion information satisfies the first condition.

7. The device of claim 6 , wherein the second direction is opposite to the first direction.

8. further comprising an interface for connecting to the operation target device by wire; The device of claim 1 , wherein the movement signal and the press signal are transmitted through the interface.

9. The device according to claim 8 , wherein the device receives power from the operation target apparatus via the interface.

10. A main body configured to be attachable to a head, The device of claim 1 , wherein the processor and the sensor are provided on the body.

11. A processor of a device that operates an operation target apparatus having a display screen, acquiring, from a sensor attached to a predetermined part of the user's body, motion information relating to the motion of the predetermined part; (1) a movement mode in which a movement signal relating to moving a pointer displayed on the display screen on the display screen can be transmitted to the operation target device based on the operation information; (2) a limit mode that limits movement of the pointer on the display screen; (3) a press mode in which a press signal related to executing a press operation at the position of the pointer on the display screen can be transmitted to the operation target device based on the operation information; controlling the pointer based on an operation mode set to any one of the above; setting the operation mode to the pressing mode; after setting the operation mode to the press mode, setting the operation mode after the press operation to one of the movement mode and the limit mode based on whether or not the operation information acquired in the press mode, including the operation information related to a first operation for transmitting the press signal and the operation information related to a second operation after the first operation, satisfy a first condition related to the operation of the predetermined part; An information processing method that performs the above.

12. A processor of a device for operating an operation target device having a display screen, acquiring, from a sensor attached to a predetermined part of the user's body, motion information relating to the motion of the predetermined part; (1) a movement mode in which a movement signal relating to moving a pointer displayed on the display screen on the display screen can be transmitted to the operation target device based on the operation information; (2) a limit mode that limits movement of the pointer on the display screen; (3) a press mode in which a press signal related to executing a press operation at the position of the pointer on the display screen can be transmitted to the operation target device based on the operation information; controlling the pointer based on an operation mode set to any one of the above; setting the operation mode to the pressing mode; after setting the operation mode to the press mode, setting the operation mode after the press operation to one of the movement mode and the limit mode based on whether or not the operation information acquired in the press mode, including the operation information related to a first operation for transmitting the press signal and the operation information related to a second operation after the first operation, satisfy a first condition related to the operation of the predetermined part; A program that executes.

Images