Information processing device, input device, information processing system, and information processing method

Abstract

The present technology relates to an information processing device, an input device, an information processing system, and an information processing method which enable an improvement in the operability of the information processing device. An information processing device according to the present technology comprises: a communication unit that performs wireless communication with an input device provided with a main body portion that is worn forward of the wrist of a user, an operation input sensor that detects an operation input to the main body portion, and a motion sensor that detects movement of the main body portion, the communication unit receiving input information including at least one of detection information relating to the operation input and the detection result of the movement of the main body portion, and user operation information relating to a user operation detected on the basis of the operation input and the detection result of the movement of the main body portion; and a presentation control unit that controls the presentation of presentation information, which is information to be presented to the user, on the basis of the user operation detected on the basis of the detection information, or the user operation information. The present technology is applicable to head mounted displays (HMDs), for example.

Description

Information processing device, input device, information processing system, and information processing method

[0001] The present technology relates to an information processing device, an input device, an information processing system, and an information processing method, and particularly to an information processing device, an input device, an information processing system, and an information processing method that are configured to improve operability.

[0002] 2. Description of the Related Art Conventionally, a ring-shaped input device has been proposed that is worn on a user's index finger and has a touch sensor at a position that can be operated with the thumb (see, for example, Patent Document 1).

[0003] International Publication No. 2023 / 286316

[0004] It is desirable to improve the operability of information processing devices such as HMDs (Head Mounted Displays) by using input devices that can be worn on the user's wrist, such as this ring-shaped input device.

[0005] The present technology has been made in view of such circumstances, and aims to improve the operability of an information processing device.

[0006] An information processing device according to a first aspect of the present technology includes a communication unit that wirelessly communicates with an input device including a main body unit that is worn on a user's wrist, an operation input sensor that detects operation input to the main body unit, and a motion sensor that detects movement of the main body unit, and receives input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body unit, and user operation information regarding a user operation detected based on the detection results of the operation input and the movement of the main body unit, and a presentation control unit that controls presentation information, which is information to be presented to the user, based on the user operation detected based on the detection information or the user operation information.

[0007] An information processing method according to a first aspect of the present technology includes an information processing device that wirelessly communicates with an input device including a main body that is worn on a user's wrist, an operation input sensor that detects operation input to the main body, and a motion sensor that detects movement of the main body, receives input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body, and user operation information regarding a user operation detected based on the detection results of the operation input and the movement of the main body, and controls presentation of presentation information, which is information to be presented to the user, based on the user operation detected based on the detection information or the user operation information.

[0008] An input device according to a second aspect of the present technology includes a main body unit that is worn on a user's wrist, an operation input sensor that detects an operation input to the main body unit, a motion sensor that detects movement of the main body unit, an input information generation unit that generates input information including at least one of detection information regarding a detection result of the operation input and the movement of the main body unit and user operation information regarding a user operation detected based on the detection result of the operation input and the movement of the main body unit, and a communication unit that wirelessly communicates with an information processing device that controls presentation of presentation information, which is information to be presented to the user, and transmits the input information.

[0009] An information processing method according to a second aspect of the present technology is an input device including a main body that is worn on a user's wrist, an operation input sensor that detects operation input to the main body, and a motion sensor that detects movement of the main body, which generates input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body, and user operation information regarding user operation detected based on the detection results of the operation input and the movement of the main body, and wirelessly communicates with an information processing device that controls presentation of presentation information, which is information to be presented to the user, and transmits the input information.

[0010] An information processing system according to a third aspect of the present technology includes a main body unit that is worn on a user's wrist, an operation input sensor that detects operation input to the main body unit, a motion sensor that detects movement of the main body unit, a user operation detection unit that detects user operation based on the operation input detected by the operation input sensor and the movement of the main body unit detected by the motion sensor, and a presentation control unit that controls the presentation of presentation information, which is information to be presented to the user, based on the user operation.

[0011] In a first aspect of the present technology, wireless communication is performed with an input device including a main body unit worn on a user's wrist, an operation input to the main body unit being detected, and an operation input sensor and a motion sensor that detects movement of the main body unit, and input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body unit, and user operation information regarding a user operation detected based on the detection results of the operation input and the movement of the main body unit is received, and presentation of presentation information, which is information to be presented to the user, is controlled based on the user operation detected based on the detection information or the user operation information.

[0012] In a second aspect of the present technology, an input device including a main body that is worn on the user's wrist, an operation input sensor that detects operation input to the main body, and a motion sensor that detects movement of the main body generates input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body, and user operation information regarding user operation detected based on the detection results of the operation input and the movement of the main body, and wireless communication is performed with an information processing device that controls the presentation of presentation information, which is information to be presented to the user, and the input information is transmitted.

[0013] In a third aspect of the present technology, an operation input to a main body unit worn on a user's wrist is detected, movement of the main body unit is detected, user operation is detected based on the operation input and the movement of the main body unit, and presentation of presentation information, which is information to be presented to the user, is controlled based on the user operation.

[0014] 1 is a diagram illustrating an embodiment of an information processing system to which the present technology is applied. FIG. 1 is a diagram illustrating an example of how an input device is worn. FIG. 2 is a diagram illustrating an example of how an input device is worn. FIG. 3 is a block diagram illustrating an example of the configuration of an information processing device and an input device. FIG. 4 is a perspective view illustrating an example of the configuration inside a housing of an input device. FIG. 5 is a perspective view illustrating an example of the configuration inside a housing of an input device. FIG. 6 is a diagram illustrating an example of how an area of ​​a touchpad is divided. FIG. 7 is a flowchart for explaining operation area detection processing. FIG. 8 is a diagram for explaining an example of a method for detecting an operation area. FIG. 9 is a diagram for explaining an example of a method for detecting an operation area. FIG. 10 is a diagram for explaining a first modified example for improving operability of an edge area. FIG. 11 is a diagram for explaining a second modified example for improving operability of an edge area. FIG. 12 is a flowchart for explaining information presentation control processing. FIG. 13 is a diagram illustrating an example of assignment of functions to each area of ​​a touchpad of an input device. FIG. 14 is a diagram illustrating an example of transition of display modes of an information processing device. FIG. 15 is a diagram illustrating an example of a menu display. FIG. 16 is a diagram for explaining an example of an operation method for a virtual object. FIG. 17 is a diagram for explaining an example of an operation method for a virtual object. FIG. 18 is a diagram for explaining an example of an operation method for a virtual object. FIG. 19 is a diagram for explaining an example of an operation method for a virtual object. 1 is a diagram for explaining a method of operating an audio hierarchical menu. FIG. 1 is a diagram for explaining an example of the configuration of an audio hierarchical menu. FIG. 2 is a diagram for explaining an example of an operation method using a thumb and index finger. FIG. 3 is a diagram for explaining an example of the waveforms of acceleration and angular velocity of a motion sensor during a pinch operation. FIG. 4 is a diagram for explaining an example of the waveforms of acceleration and angular velocity of a motion sensor during a release operation. FIG. 5 is a diagram for explaining an example of the waveforms of acceleration and angular velocity of a motion sensor during a tap operation. FIG. 6 is a diagram for explaining an example of the waveform of angular velocity when a finger is moved when the motion sensor is located directly above the finger. FIG. 7 is a diagram for explaining an example of the waveform of angular velocity when a finger is moved when the motion sensor is located to the left of the finger.10A and 10B are diagrams illustrating an example of a waveform of angular velocity when a finger is moved in a case where a motion sensor is disposed to the right of the finger. 10B are diagrams illustrating an example of a user operation detection process. 10C are diagrams illustrating modified examples of a touchpad. 10D are diagrams illustrating modified examples of a touchpad. 10E are diagrams illustrating modified examples of a touchpad. 10F are diagrams illustrating modified examples of a touchpad. 10G are diagrams illustrating modified examples of a touchpad. 10H are diagrams illustrating an example of a touchpad. 10H are diagrams illustrating modified examples ...

[0015] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Embodiment 2. Modification 3. Other

[0016] <<1. Embodiment>> <Configuration Example of Information Processing System 1>> FIG. 1 shows a configuration example of an information processing system 1 according to an embodiment of the present technology.

[0017] The information processing system 1 includes an information processing device 11 and an input device 12 .

[0018] The input device 12 is worn on the user's finger and is used to operate the information processing device 11. The input device 12 transmits input information, which is information based on operation input to the input device 12 and movement of the input device 12, to the information processing device 11, which is an external device.

[0019] The information processing device 11 is a device that presents information to a user. For example, the information processing device 11 presents visual information, auditory information, vibration information, and the like to a user.

[0020] The information processing device 11 is configured, for example, by an optically transmissive or non-transmissive head-mounted display (HMD) worn on the user's head. For example, the information processing device 11 can be realized by a glasses-type device having optically transmissive display units 57L and 57R in the lens unit. The display units 57L and 57R display images (hereinafter referred to as virtual objects) that are information about a virtual space superimposed on the real space, with the real space directly visible to the user's eyes. In this way, the information processing device 11 provides the user with so-called augmented reality (AR).

[0021] Hereinafter, when there is no need to distinguish between the display unit 57L and the display unit 57R, they will be referred to as the display unit 57.

[0022] The information processing device 11 that provides AR is not limited to an optically transparent device, but may also be a non-transparent device using a video see-through display. In a video see-through display, an image captured in the direction of the user's head (image of real space) is displayed in real time, and a virtual object is superimposed on the image. In addition, by providing the video see-through display with a configuration that dynamically blocks light from the real space, it is possible to switch between a transparent display (optical see-through display) and a non-transparent display.

[0023] 1, for example, virtual objects VO1 to VO3 presenting various types of information are displayed on the display unit 57 of the information processing device 11. The virtual objects may be two-dimensional content, three-dimensional objects, or UIs (user interfaces). Examples of two-dimensional content include websites, SNS (social networking service) screens, photos, videos, and electronic books.

[0024] 2 and 3 show examples of how the input device 12 is attached.

[0025] The input device 12 is formed in a shape that can be worn on a finger. For example, the input device 12 is formed assuming that it is worn on the proximal or middle joint of the index finger. For example, the input device 12 is formed in a ring shape.

[0026] The ring shape may be a closed loop shape or a partially open, non-closed loop shape (C-ring). The non-closed loop shape allows for slight variations in finger size. To accommodate different finger sizes, the main body may include a deformable portion.

[0027] Here, the main body of the input device 12 refers to a portion including a housing that is worn on a user's finger and a portion surrounded by the housing.

[0028] A touchpad 103 is provided on the side of the main body of the input device 12 at a position facing a finger other than the finger on which the input device 12 is worn when the input device 12 is worn. That is, the touchpad 103 is provided on the surface of the main body of the input device 12 worn by the user's finger at a position facing a finger other than the finger on which the input device 12 is worn when the input device 12 is worn. The touchpad 103 may be formed in the vertical and horizontal directions so as to cover an area excluding the upper end of the left side of the main body. For example, as shown in FIG. 3 , when the input device 12 is worn on the index finger, the touchpad 103 is located at a position facing the thumb, so that the user can perform a touch operation on the touchpad 103 with the thumb.

[0029] The touchpad 103 has a function for detecting touch operations by the user. A touch operation is an operation performed by touching the operation surface on the surface of the touchpad 103. Examples of touch operations include a touch operation, a swipe operation, and a click operation. A touch operation is, for example, an operation of touching the operation surface on the surface of the touchpad 103, and includes a tap operation and a hold operation (long press), for example. A tap operation is, for example, an operation of touching the operation surface of the touchpad 103 and then immediately releasing it. A hold operation is, for example, an operation of touching the operation surface of the touchpad 103 for a predetermined period of time or longer. A swipe operation is, for example, an operation of sliding a finger on the operation surface of the touchpad 103. A click operation is, for example, an operation of pressing the operation surface of the touchpad 103.

[0030] Here, the user can perform various operations on the virtual objects displayed on the display unit 57 of the information processing device 11 using the input device 12. Specifically, for example, by performing a contact operation such as a touch operation, swipe operation, or click operation with the thumb on the touchpad 103 of the input device 12, the user can perform a selection operation, a decision operation, or a scroll operation on the virtual object.

[0031] In this way, in this embodiment, the ring-shaped input device 12 worn on a finger can be operated with other fingers, allowing for more natural, low-stress, and less cumbersome operations to be performed in daily life.

[0032] For example, methods such as raising one's arm in front of oneself to directly manipulate a virtual object with one's hand can be conspicuous and cause resistance in crowded streets. Furthermore, methods such as directly tapping on a glasses-type device require bringing one's hand up to one's face, which can easily tire the arm. Furthermore, methods using a handheld remote control are cumbersome to carry around and take out.

[0033] In contrast to this, in this embodiment, by using an input device 12 that is attached to a finger and can be operated at hand even with the hand down, the above-mentioned inconvenience can be reduced and more natural and low-stress operations can be realized in daily life activities.

[0034] <Configuration Example of Information Processing Device 11 and Input Device 12> FIG. 4 is a block diagram showing an example of the configuration of the information processing device 11 and the input device 12 included in the information processing system 1. As shown in FIG.

[0035] The information processing device 11 includes, for example, a communication unit 51 , a control unit 52 , a gaze detection unit 53 , an outward-facing camera 54 , an operation input unit 55 , a microphone 56 , a display unit 57 , an audio output unit 58 , and a memory unit 59 .

[0036] The communication unit 51 is connected to an external device via wired or wireless communication to transmit and receive data. For example, the communication unit 51 can be connected to the input device 12 via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The communication unit 51 can also be connected to the Internet via a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 3G (third generation mobile communication system), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), or the like, and can transmit and receive data to and from a server on the network.

[0037] The control unit 52 functions as an arithmetic processing unit and a control device, and controls the overall operation of the information processing device 11 in accordance with various programs. The control unit 52 is realized by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 52 may also include a ROM (Read Only Memory) that stores programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0038] The control unit 52 also functions as a user operation detection unit 61 , a presentation control unit 62 , and a sound recording control unit 63 .

[0039] The user operation detection unit 61 detects user operations on the information processing device 11 based on input information from the input device 12, gaze information from the gaze detection unit 53, captured images from the outward-facing camera 54, and operation input information from the operation input unit 55, etc.

[0040] The input information received from the input device 12 includes, for example, operation input to the main body of the input device 12 and detection information regarding the movement of the main body.

[0041] For example, the user operation detection unit 61 detects a user operation using the input device 12 based on a detection result of an operation input to the input device 12. For example, the user operation detection unit 61 detects a touch operation on the input device 12.

[0042] For example, the user operation detection unit 61 detects a user operation based on the movement of the hand (hereinafter referred to as the operating hand) or fingers (e.g., gesture (motion input)) of the user wearing the input device 12, based on the movement of the main body of the input device 12. For example, the user operation detection unit 61 detects a pinch operation, a release operation, a tap operation, a twist operation, etc.

[0043] A pinch operation is, for example, an operation of bringing the pad of the thumb and the pad of the index finger of the operating hand into contact. A release operation is, for example, an operation of separating the pad of the thumb and the pad of the index finger from a contacting state. A tap operation is an operation of bringing the pad of the thumb and the pad of the index finger into contact from a separated state, and then immediately separating them. A twist operation is, for example, an operation of twisting the operating hand, that is, an operation of rotating the operating hand counterclockwise or clockwise in the direction of rotation of the wrist.

[0044] The presentation control unit 62 controls the presentation of information to be presented to the user (hereinafter referred to as presentation information) based on the user operation detected by the user operation detection unit 61 and gaze information from the gaze detection unit 53, etc.

[0045] For example, the presentation control unit 62 performs control to display a virtual object on the display unit 57 .

[0046] The presentation control unit 62 can also control the display position and orientation of a virtual object superimposed on real space based on the analysis results of an image captured in the direction of the user's head (an image that includes at least the user's field of view) acquired by the outward-facing camera 54. The presentation control unit 62 can also control the display position and orientation of a virtual object in consideration of the user's own position and orientation.

[0047] For example, analysis of the captured image (such as object recognition) may be performed by the presentation control unit 62. Furthermore, the user's own position and posture may be detected by various sensors provided in the information processing device 11. Examples of the various sensors include the outward-facing camera 54, a gyro sensor, an acceleration sensor, a geomagnetic sensor, and a position measurement unit. The position measurement unit may be a measurement unit that measures an absolute position (for example, position measurement using a GNSS (Global Navigation Satellite System)), or a measurement unit that measures a relative position (for example, position measurement using a Wi-Fi or Bluetooth signal).

[0048] The data of the virtual object to be displayed may be data stored in the storage unit 59, may be data received from an external device via the communication unit 51, or may be data generated by the presentation control unit 62. Examples of the external device include a server on the Internet and a communication terminal owned by the user (e.g., a smartphone, a tablet terminal, a wearable device, etc.).

[0049] Furthermore, the presentation control unit 62 can control the display of virtual objects based on input information received from the input device 12 via the communication unit 51. For example, the presentation control unit 62 controls selection, confirmation, scrolling, zoom, or the like for the virtual objects in accordance with the received input information.

[0050] The presentation control unit 62 may also control the operation position on the display screen of the display unit 57 in accordance with the user's gaze information detected by the gaze detection unit 53. That is, the presentation control unit 62 may perform a pointing operation by gaze input. Here, "gaze information" is, for example, information indicating the direction of the user's gaze. The presentation control unit 62 recognizes the position of the user's gaze center (point of gaze) on the display screen of the display unit 57 as the operation position based on the user's gaze direction. Then, the presentation control unit 62 recognizes the virtual object displayed at the operation position as the operation target, and executes control on the virtual object in accordance with the operation on the input device 12, etc.

[0051] In this way, the user can perform an operation (e.g., a scroll operation) on any virtual object by directing their gaze toward the virtual object to be operated and then operating the input device 12. Such an operation method that combines gaze information and input information from the input device 12 may be used, for example, as an operation method for mainly two-dimensional images (including the above-mentioned two-dimensional content).

[0052] On the other hand, when the virtual object is a three-dimensional image (including the above-mentioned three-dimensional object and UI), operation methods include a method of directly operating the virtual object and a method of remotely operating the virtual object. An example of a direct operation method is a method in which the display of the virtual object is controlled in accordance with the user's movements of touching, grabbing, and releasing the virtual object.

[0053] In this case, the presentation control unit 62 appropriately uses the positions and movements of the user's hands and fingers obtained by analyzing the captured images acquired by the outward-facing camera 54 and the input information received from the input device 12 .

[0054] For example, as will be described later, the presentation control unit 62 controls movement between levels and items in a hierarchical menu consisting of multiple levels and items, and selection of items, based on user operations. The presentation control unit 62 also controls the output of audio related to an item specified in the hierarchical menu.

[0055] The presentation control unit 62 may perform control such as power on / off, volume adjustment, and display on / off in response to an operation input from the operation input unit 55 provided in the information processing device 11 .

[0056] The presentation control unit 62 may also control audio output from the audio output unit 58. For example, the presentation control unit 62 controls audio playback when the virtual object is a moving image.

[0057] Furthermore, the presentation control unit 62 may provide vibration feedback using the vibration unit 105 provided in the input device 12 as feedback of an operation performed by the user using the input device 12 .

[0058] The recording control unit 63 controls the storage of the audio data supplied from the microphone 56 in the storage unit 59, that is, the recording of the audio data.

[0059] For example, the recording control unit 63 temporarily stores audio data collected by the microphone 56 in the storage unit 59. Then, unless otherwise specified by the user, the recording control unit 63 deletes audio data from the storage unit 59 after a predetermined period of time has elapsed. On the other hand, if specified by the user, the recording control unit 63 stores audio data for a specified period of time in the storage unit 59. This makes it possible to record audio data going back in time from the present.

[0060] Furthermore, the recording control unit 63 stores (records) the audio data supplied from the microphone 56 in the storage unit 59 from the present or from a date and time designated by the user.

[0061] The gaze detection unit 53 is a gaze information acquisition unit that has a function of detecting (acquiring) information related to the user's gaze (gaze information). The gaze detection unit 53 may be an inward-facing camera that captures an image of the user's eyes when the information processing device 11 is worn, or an electromyographic sensor that detects the electrooculography. The gaze detection unit 53 detects, for example, the gaze direction as gaze information based on information acquired by various sensors. There are no particular limitations on the method for detecting gaze information.

[0062] The outward-facing camera 54 is an imaging unit provided in the information processing device 11 to capture an image in the direction of the user's head when the information processing device 11 is worn. The angle of view of the outward-facing camera 54 includes at least a field of view (also referred to as a field of view). The outward-facing camera 54 includes a lens system including an imaging lens, an aperture, a zoom lens, a focus lens, etc.; a drive system that performs focusing and zooming operations on the lens system; and a solid-state imaging element array that photoelectrically converts imaging light obtained by the lens system to generate an imaging signal. The solid-state imaging element array may be realized, for example, by a charge-coupled device (CCD) sensor array or a complementary metal oxide semiconductor (CMOS) sensor array.

[0063] The operation input unit 55 receives an operation from the user and outputs information about the received operation to the control unit 52. The operation input unit 55 can be realized by, for example, a push button, a touch sensor, an optical sensor, or the like.

[0064] The microphone 56 collects sounds around the information processing device 11 and supplies audio data representing the collected sounds to the control unit 52 .

[0065] The display unit 57 is realized by, for example, a transmissive display. A transmissive display is a display that can deliver light from real space directly to the user's eyes. The user can directly view the real space through the transmissive display. The transmissive display may be, for example, an optical see-through display. The optical see-through display may employ known types, including a half mirror type, a light guide plate type, a direct retinal imaging type, and the like.

[0066] The audio output unit 58 has a function of outputting audio. For example, the audio output unit 58 may be configured as headphones, earphones, or a bone conduction speaker.

[0067] The memory unit 59 includes a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 52, an EEPROM (Electrically Erasable and Programmable Read Only Memory) that stores image data acquired by the outward-facing camera 54 and audio data acquired by the microphone 56, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0068] The configuration of the information processing device 11 is not limited to the example shown in Fig. 2. For example, the information processing device 11 does not need to include all of the components shown in Fig. 2, or may include other components. Furthermore, the information processing device 11 may be composed of multiple devices.

[0069] The input device 12 includes a communication unit 101, a control unit 102, a touchpad 103, a motion sensor 104, a vibration unit 105, and a storage unit 106. Note that the configuration shown in Fig. 1 is an example, and the configuration of the input device 12 is not limited to this.

[0070] The communication unit 101 is connected to an external device via wired or wireless communication to transmit and receive data. For example, the communication unit 101 can be connected to the information processing device 11 via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0071] The control unit 102 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the input device 12 in accordance with various programs. The control unit 102 is realized by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 102 may also include a ROM (Read Only Memory) that stores programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0072] The control unit 102 also functions as an input information generation unit 111 and a vibration control unit 112 .

[0073] The input information generation unit 111 generates input information including, for example, an operation input to the main body of the input device 12 and detection information regarding the detection result of the movement of the main body of the input device 12. The detection information includes, for example, data indicating the detection result of the operation input to the touchpad 103 (hereinafter referred to as touchpad data) and data indicating the detection result of the movement of the main body of the input device 12 by the motion sensor 104 (hereinafter referred to as motion data). The input information generation unit 111 controls transmission of the input information to the information processing device 11 via the communication unit 101.

[0074] The vibration control unit 112 controls the vibration of the vibration unit 105 in accordance with the vibration control signal received from the information processing device 11 via the communication unit 101. Such vibration control can be presented as feedback of an operation input by the user, for example.

[0075] The touchpad 103 is an operation input sensor that detects an operation input (the above-mentioned touch operation) by the user, and supplies touchpad data indicating the detection result to the control unit 102 .

[0076] The motion sensor 104 detects the movement (posture information) of the main body of the input device 12 (or the hand or fingers of the user wearing it), and supplies motion data indicating the detection results to the control unit 102. For example, the motion sensor 104 may include a three-axis acceleration sensor and a three-axis gyro sensor. The motion sensor 104 may also include a geomagnetic sensor. Therefore, the motion data includes sensor data indicating, for example, detected values ​​of acceleration in three axial directions (hereinafter referred to as acceleration values), detected values ​​of angular velocity around three axes (hereinafter referred to as angular velocity values), and detected values ​​of geomagnetism (hereinafter referred to as geomagnetic values).

[0077] The vibration unit 105 is configured, for example, by a vibration actuator or the like, and has a function of presenting tactile information (tactile stimulation to the skin). Note that the vibration method is not particularly limited. There may be one vibration unit 105, or multiple vibration units 105 may be provided.

[0078] The storage unit 106 includes a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 102, and a RAM (Random Access Memory) that temporarily stores parameters that change as needed.

[0079] The configuration of the input device 12 is not limited to the example shown in Fig. 4. For example, it is not necessary to include all of the components shown in Fig. 4, or other components may be included. For example, the input device 12 may include one or more other sensors. For example, the input device 12 may further include a distance sensor. The distance sensor may be an optical sensor (for example, a time-of-flight (ToF) type).

[0080] An example of the configuration inside the housing of the input device 12 will now be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a perspective view of the input device 12 as seen obliquely from above. Fig. 6 is a perspective view of the input device 12 as seen from the left.

[0081] A motion sensor 104, a wireless communication microcomputer (hereinafter referred to as wireless communication microcomputer) 131, and a battery 132 are built into the upper part of the main body of the input device 12 (the part located on the back side of the user's finger). The wireless communication microcomputer 131 and the battery 132 are stacked in the vertical direction, and the battery 132 is placed below the wireless communication microcomputer 131.

[0082] The wireless communication microcomputer 131 realizes, for example, the communication unit 101, the control unit 102, and the storage unit 106 shown in FIG.

[0083] The battery 132 supplies power to each part of the input device 12 .

[0084] The motion sensor 104 may be provided below the main body of the input device 12 (at the location near the pad of the user's finger), on the left side (on the thumb side when worn on the index finger), or on the right side (on the middle finger side when worn on the index finger).

[0085] The left side of the input device 12 has built-in vibration unit 105, touch sensor 133, and tactile switch 134. The touch sensor 133 and tactile switch 134 constitute the touchpad 103. The touch sensor 133 is disposed over almost the entire surface of the left side of the input device 12, and constitutes the operation surface of the touchpad 103. When the input device 12 is worn on the index finger, the touch sensor 133 is located in a position opposite the thumb, so that the user can perform a contact operation on the touch sensor 133 with the thumb.

[0086] The touch sensor 133 may be of a resistive type, a capacitive type, an ultrasonic type, an optical type, or the like, and any of these may be used in this embodiment. The touch sensor 133 has an array of multiple nodes, and is capable of detecting contact (touch) at each node.

[0087] In the following, a case will be described in which the touch sensor 133 is of a capacitance type and outputs data indicating the capacitance of each node (hereinafter referred to as touch sensor data).

[0088] The tactile switch 134 is disposed approximately in the center of the back surface (the surface facing the fingers) of the touch sensor 133. The tactile switch 134 detects a pressing operation (click operation) on the touchpad 103. The tactile switch 134 outputs data indicating whether the switch is on or off (hereinafter referred to as switch state data).

[0089] Therefore, the touchpad data output from the touchpad 103 includes touch sensor data from the touch sensor 133 and switch state data from the tactile switch 134 .

[0090] A pressure sensor may be disposed on the back side of the touchpad 103. For example, the pressure sensor detects the pressure applied to the touchpad 103, and when the detected value exceeds a preset threshold, a click operation on the touchpad 103 is detected.

[0091] This realizes a function equivalent to a push switch, allowing the user to freely change the push pressure (hardness) and switch feel to suit their preferences. Furthermore, by preventing a click operation from being detected when the touch sensor 133 is not detecting a user's touch, even if the detected pressure exceeds a threshold, it is possible to prevent a click operation from being erroneously detected when a handheld object other than a finger touches the touch sensor 133. Furthermore, the analog value obtained by the pressure sensor can be used for analog volume operations, such as zooming the display area according to the pressure value.

[0092] The vibration unit 105 is disposed above the tactile switch 134 on the rear surface of the touch sensor 133. Control may be performed such that vibration feedback using the vibration unit 105 is returned to the user.

[0093] The main body of the input device 12 may also be provided with a distance sensor (not shown). The distance sensor measures the distance to another finger (e.g., the thumb) or the distance to a specific joint (e.g., the middle joint) of the finger on which the sensor is attached. The distance sensor may be, for example, a Time Of Flight (ToF) optical sensor. It is also possible to arrange multiple distance sensors to expand the detection range.

[0094] Furthermore, the main body of the input device 12 may be provided with a bent portion without a substrate disposed therein, and this portion may be made plastically deformable, thereby making it possible to accommodate different finger sizes for each user.

[0095] Although an example of the structure of the input device 12 has been described above, the arrangement of each component according to this embodiment is not limited to this.

[0096] <Regarding the area of ​​the operation surface of the touchpad 103> As described above, the touch sensor 133 is provided on the operation surface, which is the surface of the touchpad 103. As shown in Fig. 7 , the operation surface of the touchpad 103 is divided into four areas: a central area 103C, a left edge area 103L, a right edge area 103R, and a bottom edge area 103B.

[0097] The bottom area 103B is an area that includes the bottom end of the touch sensor 133. The left area 103L is an area that includes the left end of the touch sensor 133 excluding the bottom area 103B. The right area 103R is an area that includes the right end of the touch sensor 133 excluding the bottom area 103B. The center area 103C is an area that includes the center of the touch sensor 133 excluding the left area 103L, right area 103R, and bottom area 103B of the touch sensor 133.

[0098] Hereinafter, the left edge area 103L, the right edge area 103R, and the bottom edge area 103B will be collectively referred to as edge areas. Hereinafter, the central area 103C, the left edge area 103L, the right edge area 103R, and the bottom edge area 103B of the touchpad 103 will sometimes be simply referred to as the central area 103C, the left edge area 103L, the right edge area 103R, and the bottom edge area 103B, respectively. For example, hereinafter, touching the central area 103C of the touchpad 103 will sometimes be simply referred to as touching the central area 103C.

[0099] <Method of Detecting Operation Area> Next, a method of detecting the operation area of ​​the touchpad 103 will be described.

[0100] Here, the operation area refers to the area to be operated among the central area 103C, the left end area 103L, the right end area 103R, and the bottom end area 103B of the operation surface (touch sensor 133) of the touchpad 103.

[0101] Here, the operation area detection process executed by the information processing device 11 will be described with reference to the flowchart of FIG.

[0102] In step S1, the information processing device 11 determines whether or not the touchpad 103 is being touched. For example, the control unit 52 receives input information from the input device 12 via the communication unit 51. The user operation detection unit 61 determines whether or not the touchpad 103 is being touched, based on touchpad data included in the input information.

[0103] This process is repeatedly executed until it is determined that the touchpad 103 is touched, and if it is determined that the touchpad 103 is touched, the process proceeds to step S2.

[0104] In step S2, the user operation detection unit 61 detects the center of the touch area. For example, the user operation detection unit 61 detects the center of the touch area using a method similar to a conventional method. For example, the user operation detection unit 61 calculates the center of gravity of the touch area, which is an area consisting of nodes whose capacitance is equal to or greater than a predetermined threshold, based on the capacitance of each node of the touch sensor 133. Alternatively, for example, the user operation detection unit 61 calculates the center of gravity of the touch area based on the shape of the touch area. The user operation detection unit 61 detects the node of the touch sensor 133 that includes the calculated center of gravity of the touch area as the center of the touch area.

[0105] For example, as shown in FIG. 9, a center P1 is detected for a touch area T1 on the operation surface of the touch pad 103.

[0106] In step S3, the user operation detection unit 61 determines whether the center of the touch area is within the central area 103C. If it is determined that the center of the touch area is within the central area 103C, the process proceeds to step S4.

[0107] For example, in the example of FIG. 9, the center P1 of the touch area T1 is within the central area 103C, so the process proceeds to step S4.

[0108] In step S4, the user operation detection unit 61 detects a reaction node. For example, the user operation detection unit 61 detects, as a reaction node, a node of the touch sensor 133 whose capacitance is equal to or greater than a predetermined threshold, based on the touchpad data acquired in the process of step S1.

[0109] In step S5, the user operation detection unit 61 determines whether the number of reaction nodes in the right-end area 103R is equal to or greater than a threshold value. If it is determined that the number of reaction nodes in the right-end area 103R is less than the threshold value, the process proceeds to step S6.

[0110] In step S6, the user operation detection unit 61 determines whether the number of reaction nodes in the leftmost area 103L is equal to or greater than a threshold value. If it is determined that the number of reaction nodes in the leftmost area 103L is less than the threshold value, the process proceeds to step S7.

[0111] In step S7, the user operation detection unit 61 determines whether the number of reaction nodes in the lower end area 103B is equal to or greater than a threshold value. If it is determined that the number of reaction nodes in the lower end area 103B is less than the threshold value, the process proceeds to step S8.

[0112] In step S8, the user operation detection unit 61 determines the central area 103C as the operation area. That is, if the number of reaction nodes in any edge area is less than the threshold, the central area 103C is determined as the operation area.

[0113] Thereafter, the process returns to step S1, and the processes from step S1 onwards are executed.

[0114] On the other hand, if it is determined in step S7 that the number of reaction nodes in the lower end area 103B is equal to or greater than the threshold value, the process proceeds to step S9.

[0115] In step S9, the user operation detection unit 61 determines the bottom area 103B as the operation area.

[0116] Thereafter, the process returns to step S1, and the processes from step S1 onwards are executed.

[0117] On the other hand, if it is determined in step S6 that the number of reaction nodes in the left end area 103L is equal to or greater than the threshold value, the process proceeds to step S10.

[0118] In step S10, the user operation detection unit 61 determines the left end area 103L as the operation area.

[0119] Thereafter, the process returns to step S1, and the processes from step S1 onwards are executed.

[0120] If it is determined in step S5 that the number of reaction nodes in the right end area 103R is equal to or greater than the threshold value, the process proceeds to step S11.

[0121] In step S10, the user operation detection unit 61 determines the right end area 103R as the operation area.

[0122] Thereafter, the process returns to step S1, and the processes from step S1 onwards are executed.

[0123] On the other hand, if it is determined in step S3 that the center of the touch area is not within the central area 103C, that is, if it is determined that the center of the touch area is within one of the edge areas, the process proceeds to step S12.

[0124] In step S12, the user operation detection unit 61 determines an area including the center of the touch area as the operation area, i.e., the edge area including the center of the touch area among the left edge area 103L, the right edge area 103R, and the bottom edge area 103B.

[0125] Thereafter, the process returns to step S1, and the processes from step S1 onwards are executed.

[0126] This improves the operability in the edge area.

[0127] For example, because the size of the touchpad 103 is small, as shown in Fig. 9, even if the user attempts to touch the right edge area 103R, the central area 103C is often touched at the same time. Also, because the central area 103C is larger than the right edge area 103R, there is a high possibility that the center P1 of the touch area T1 is included in the central area 103C. Therefore, if the operation area is detected simply based on the position of the center P1 of the touch area T1, it will be erroneously detected that the central area 103C has been operated.

[0128] On the other hand, in the present technology, even if the center P1 of the touch area T1 is within the central area 103C, it is possible to detect that the right edge area 103R has been operated.

[0129] 10 schematically shows the states of nodes in the central area 103C, left edge area 103L, and right edge area 103R of the touchpad 103. Each square box represents a node. The box with a dotted pattern on the left edge represents a node in the left edge area 103L. The box with a dotted pattern on the right edge represents a node in the right edge area 103R. The other boxes represent nodes in the central area 103C. The box with a black square in the center represents a reaction node.

[0130] In this example, the center P1 of the touch area is included in the central area 103C. Meanwhile, the number of reaction nodes in the right-end area 103R is 7. For example, if the threshold for the number of reaction nodes in the right-end area 103R is set to 5, the right-end area 103R can be detected as an operation area.

[0131] In this way, the user can easily set the edge area as the operation target, improving the operability of the edge area.

[0132] The threshold value for each edge area does not necessarily have to be the same, and may be set individually based on the size, shape, etc. Also, the threshold value for each edge area may be made variable so that the sensitivity to each edge area can be adjusted.

[0133] In addition, although the flowchart in FIG. 8 shows an example in which the operation area is determined in the order of the right edge area 103R, the left edge area 103L, and the bottom edge area 103B, this determination order can be changed.

[0134] Furthermore, for example, the number of reaction nodes in the right end area 103R, the left end area 103L, and the bottom end area 103B may be detected, and an area in which the number of reaction nodes is equal to or greater than a threshold may be determined to be an operation area.

[0135] Furthermore, for example, the threshold for each edge area may be set based on the ratio of reaction nodes within each edge area, rather than the number of reaction nodes.

[0136] For example, the operation area may be determined based on the capacitance response value (e.g., a value in 1024 stages) of each node. For example, the operation area may be determined by comparing the total capacitance value of each area. For example, the area including the node with the largest capacitance may be determined to be the operation area.

[0137] For example, the operation area may be determined based on the ratio between the areas of the total, average, or maximum capacitance of the reaction nodes in each area. In this case, even if a node with the largest overall capacitance is present in the central area 103C, if the maximum capacitance of the nodes in the right-end area 103R is equal to or greater than a certain percentage of the maximum capacitance of the nodes in the central area 103C, the right-end area 103R may be determined to be the operation area.

[0138] <Modification of Method for Improving Operability in Edge Areas> Next, a modification of the method for improving operability in edge areas will be described with reference to FIGS. 11 and 12. FIG.

[0139] For example, as shown in FIG. 11, in addition to the touch sensor 133, a touch sensor for the edge area may be provided.

[0140] In this example, a touch sensor 133 is provided on the top surface of the base member 151. This touch sensor 133 is dedicated to the central area 103C.

[0141] A touch sensor 152R for the right end area 103R is provided on the right side surface of the base member 151. A touch sensor 152L for the left end area 103L is provided on the left side surface of the base member 151. Although not shown, a touch sensor 152B for the bottom end area 103B is provided on the side surface of the base member 151 on the near side in the drawing.

[0142] The side surfaces of the base member 151 do not necessarily have to be perpendicular to the top surface, but may be formed at an angle. In this case, for example, as shown in Fig. 12B described later, the edge areas will be oriented at an angle to the central area 103C.

[0143] Hereinafter, when there is no need to distinguish between the touch sensor 152R, the touch sensor 152L, and the touch sensor 152C, they will be simply referred to as the touch sensor 152.

[0144] For example, the touch sensor 152 is prioritized over the touch sensor 133. That is, even if a touch is detected in the touch sensor 133, if a touch is detected in one of the touch sensors 152, the edge area corresponding to the touch sensor 152 where the touch is detected is set as the operation area.

[0145] The touch sensor 133 and the touch sensor 152 may be integrated and separated by nodes.

[0146] For example, as shown in FIG. 12, the central area 103C and the edge area may be distinguished from each other in terms of shape.

[0147] 12A, the central area 103C and the bottom area 103B form a continuous surface, but the bottom area 103B is a curved surface. Although not shown, the left and right areas 103L and 103R are also curved surfaces, similar to the bottom area 103B.

[0148] This allows the user to distinguish between the central area 103C and the edge area by touch, improving operability for the edge area.

[0149] 12B, the bottom area 103B is bent relative to the central area 103C, forming a discontinuous surface between the central area 103C and the bottom area 103B. Although not shown, the left and right areas 103L and 103R are also bent relative to the central area 103C, forming a discontinuous surface between the central area 103C and the left and right areas 103L and 103R.

[0150] This allows the user to distinguish between the central area 103C and the edge area by touch, improving operability for the edge area.

[0151] 12C, for example, a protrusion 171 is formed at the boundary between the central area 103C and the bottom area 103B. Although not shown, protrusions similar to the protrusion 171 are also formed at the boundary between the central area 103C and the left-end area 103L and the boundary between the central area 103C and the right-end area 103R.

[0152] This allows the user to distinguish between the central area 103C and the edge area by touch, improving operability for the edge area.

[0153] For example, by forming the protrusion 171 inside the edge area, the user can sense by touch that the finger is within the edge area when the center of the finger is touching the protrusion 171.

[0154] <Operation Method of Information Processing System 1> Next, an example of an operation method of the information processing system 1 will be described with reference to FIGS.

[0155] <Information Presentation Control Processing> First, the information presentation control processing executed by the information processing device 11 will be described with reference to the flowchart of FIG.

[0156] In step S31, the gaze detection unit 53 detects the gaze of the user. The gaze detection unit 53 supplies the control unit 52 with gaze information indicating the result of the gaze detection of the user.

[0157] In step S32 , the information processing device 11 receives input information from the input device 12 .

[0158] For example, the input information generation unit 111 of the input device 12 acquires touchpad data from the touchpad 103 and motion data from the motion sensor 104. The input information generation unit 111 generates input information including the touchpad data and the motion data. The input information generation unit 111 transmits the input information to the information processing device 11 via the communication unit 101.

[0159] In response to this, the control unit 52 of the information processing device 11 receives the input information via the communication unit 51 .

[0160] In step S33, the user operation detection unit 61 detects a user operation based on the input information.

[0161] For example, the user operation detection unit 61 detects whether or not an operation has been performed on the touchpad 103 and the operation area based on the touchpad data included in the input information by the method described above with reference to Fig. 8. Furthermore, the user operation detection unit 61 detects the type of operation performed on the operation area (for example, a tap operation, a long press operation, a swipe operation, a click operation, etc.) based on the touchpad data.

[0162] For example, the user operation detection unit 61 detects the movement of the main body of the input device 12 based on motion data included in the input information. Furthermore, the user operation detection unit 61 detects a user operation based on a gesture of the operating hand or the fingers of the operating hand based on the movement of the main body of the input device 12. For example, the user operation detection unit 61 detects the above-mentioned pinch operation, release operation, tap operation, twist operation, etc.

[0163] In step S34, the presentation control unit 62 controls the presentation of the presentation information based on the user operation and line-of-sight information.

[0164] A specific example of the process in step S34 will be described later with reference to FIGS.

[0165] In step S35, the control unit 52, as necessary, feeds back vibration information to the input device 12. For example, the control unit 52 generates a vibration control signal for presenting vibration corresponding to the operation content or the presented information, and transmits the signal to the input device 12 via the communication unit 51.

[0166] In response to this, the control unit 102 of the input device 12 receives the vibration control signal via the communication unit 101. The vibration control unit 112 vibrates the vibration unit 105 based on the vibration control signal, thereby presenting vibration information to the user.

[0167] Thereafter, the process returns to step S31, and the processes from step S31 onwards are executed.

[0168] Note that the processing of each step in Fig. 13 is not limited to the order shown in Fig. 13. For example, step S31, step S32 and step S33, and step S34 and step S35 may be performed continuously in parallel.

[0169] <First Example of Information Presentation Method of Information Processing System 1> Next, a first example of a method in which the information processing system 1 presents presentation information will be described with reference to FIGS. 14 to 23 and in accordance with the flowchart of FIG.

[0170] The first example is an example in which the user uses the input device 12 to operate a virtual object displayed on the display unit 57 of the information processing device 11 .

[0171] FIG. 14 shows an example of the allocation of functions to each area of ​​the touch pad 103 of the input device 12. In FIG.

[0172] For example, a click operation in the central area 103C is assigned a "confirm" function for the target of the user's gaze, and a swipe operation in the central area 103C is assigned a "screen scroll" function.

[0173] For example, a click operation on the leftmost area 103L is assigned a "back" function for TV broadcasts, websites, etc. For example, a click operation on the rightmost area 103R is assigned a "forward" function for TV broadcasts, websites, etc.

[0174] For example, a "volume up / down" function or the like is assigned to an up / down swipe operation in the right-end area 103R.

[0175] For example, a "screen position movement" function is assigned to a click operation on the bottom end area 103B, and a "screen erasure" function is assigned to a double click operation on the bottom end area 103B.

[0176] 14 shows examples of display modes of the information processing device 11. The information processing device 11 transitions between five display modes, for example, "screen display," "menu display," "volume adjustment," "photography function," and "screen off."

[0177] In FIG. 14, the screen represents a virtual object.

[0178] The screen display mode is a display mode in which a virtual object is displayed on the display unit 57. The screen display mode includes two display modes: a single-screen display mode and a multi-screen display mode. The single-screen display mode is a display mode in which only one virtual object is displayed. The multi-screen display mode is a display mode in which multiple virtual objects are displayed simultaneously, as in the example of FIG. 1, for example.

[0179] When the central area 103C is pressed and held in the screen display mode (single screen display mode or multi-screen display mode), the mode transitions to the menu display mode.

[0180] In the menu display mode, as shown in Fig. 16, a menu 201, which is a virtual object, is displayed on the display unit 57. The menu 201 includes icons 211a to 211d.

[0181] Hereinafter, when there is no need to distinguish between the icons 211a to 211d, they will be simply referred to as icons 211.

[0182] The position of the cursor C1 in the menu 201 moves according to the gaze position S1. When the central area 103C is clicked, an icon 211 in the cursor C1 is selected, a process corresponding to the selected icon 211 is executed, and the display mode transitions to the screen display mode.

[0183] For example, when the icon 211 a is selected, an application for playing video is started, and the corresponding virtual object is displayed on the display unit 57 .

[0184] For example, when the icon 211 b is selected, a message application is started and a corresponding virtual object is displayed on the display unit 57 .

[0185] For example, when the icon 211 c is selected, a web browser is launched and the corresponding virtual object is displayed on the display unit 57 .

[0186] For example, when the icon 211 d is selected, a map application is started and a corresponding virtual object is displayed on the display unit 57 .

[0187] In the menu display mode, when the central area 103C is pressed and held, the mode transitions to the screen display mode, and the menu 201 disappears.

[0188] In the screen display mode, when the right edge area 103R is swiped up or down, the display mode transitions to the volume adjustment mode. For example, when the right edge area 103R is swiped up, the volume of content within the virtual object to be operated, which includes the gaze position (point of gaze), increases. On the other hand, when the right edge area 103R is swiped down, the volume of content within the virtual object to be operated, which includes the gaze position (point of gaze), decreases. After the swipe in the right edge area 103R stops, a timeout occurs after a predetermined time, and the display mode transitions to the screen display mode.

[0189] In the screen display mode, when the central area 103C is touched and a gesture of raising the operating hand is made, the mode transitions to the photographing mode and the photographing application is started. In the photographing mode, when the central area 103C is touched and a gesture of lowering the operating hand is made, the mode transitions to the screen display mode and the photographing application is ended.

[0190] The shooting modes will be described in detail later.

[0191] For example, when the bottom end area 103B is double-clicked in the screen display mode, the mode transitions to the screen-off mode. That is, all displayed virtual objects disappear. On the other hand, when the bottom end area 103B is double-clicked in the screen-off mode, the mode transitions to the screen display mode. In this case, for example, virtual objects that were displayed on the display unit 57 before the transition to the screen-off mode are re-displayed in the state before the transition.

[0192] Next, a specific example of an information presentation method of the information processing system 1 will be described with reference to FIGS.

[0193] For example, as shown in A of Fig. 17, when the gaze position S1 is located within the virtual object VO1, clicking the right edge area 103R switches the content within the virtual object VO1 to the next content while the position of the virtual object VO1 remains fixed, as shown in B of Fig. 17. On the other hand, as shown in B of Fig. 17, when the gaze position S1 is located within the virtual object VO1, clicking the right edge area 103R switches the content within the virtual object VO1 to the previous content while the position of the virtual object VO1 remains fixed, as shown in A of Fig. 17.

[0194] Here, previous and next contents refer to, for example, previous and next pages of a website, previous and next channels of a TV broadcast, and the like.

[0195] Furthermore, when the right edge area 103R is swiped upward, the volume of the content in the virtual object VO1 that includes the gaze position S1 increases. On the other hand, when the right edge area 103R is swiped downward, the volume of the content in the virtual object VO1 that includes the gaze position S1 decreases.

[0196] The display position of the virtual object is not fixed in real space, but is displayed based on the coordinate axes of the information processing device 11 itself so that the relative display position does not change even if the user moves. This allows the user to do things like viewing a website while walking.

[0197] On the other hand, if the user's position does not change and only the orientation of the head changes, the position of the virtual object in the display unit 57 is localized in real space without following the orientation of the user's head. Therefore, for example, as shown in A of Fig. 18 , the position of the virtual object VO1 may be displayed at a position shifted from the center of the display unit 57. In this case, for example, when the bottom edge area 103B is clicked, the virtual object VO1 moves to match the gaze position S1, as shown in B of Fig. 18 . For example, the virtual object VO1 moves so that the center of the virtual object VO1 approximately coincides with the gaze position S1.

[0198] Furthermore, if the orientation of the user's head changes while the bottom end area 103B is still clicked, the virtual object in the display unit 57 moves to follow the movement of the user's head. For example, the virtual object VO1 moves to follow the movement so that the center of the virtual object VO1 approximately coincides with the gaze position S1.

[0199] For example, as shown in A of Fig. 19, when the gaze position S1 is moved to the upper right vicinity of the virtual object VO1, a button VO2, which is a virtual object, is displayed in the upper right vicinity of the virtual object VO1. Then, when the central area 103C is clicked with the gaze position S1 positioned within the button VO2, the virtual object VO1 moves to the back as shown in B of Fig. 19. In other words, the virtual object VO1 is reduced in size. Also, the button VO2 disappears.

[0200] For example, as shown in FIG. 20, when the gaze position S1 is included within the virtual object VO1, if the central area 103C is swiped up or down, the content within the virtual object VO1 scrolls up or down.

[0201] For example, as shown in A of Fig. 21, when the gaze position S1 is included in the virtual object VO1, touching the central area 103C moves the cursor 312 to an item in the content within the virtual object VO1 that includes the gaze position S, as shown in B of Fig. 21. Furthermore, when the central area 103C is clicked, the content linked to the item within the cursor 312 is displayed within the virtual object VO1.

[0202] 22, when the operating hand is raised while the central area 103C is touched, the information processing device 11 transitions to a photography mode and an application for photography is started. Note that when the application for photography is started, the vibration unit 105 of the input device 12 may be vibrated to provide vibration feedback notifying the user that the application for photography is started.

[0203] 23A and 23B show examples of the display on the display unit 57 in the shooting mode. Specifically, a shooting frame 301, a focus frame 302, a scale 303, and a zoom bar 304, which are virtual objects, are displayed on the display unit 57.

[0204] The shooting frame 301 is a frame that indicates the shooting range. The focus frame 302 is a frame that indicates the focus position. The scale 303 is a scale that indicates the size of the image to be shot. The zoom bar 304 is a bar that indicates the set zoom magnification.

[0205] For example, if the user holds up their operating hand and twists their operating hand leftward while touching the central area 103C, the zoom magnification decreases and the shooting range indicated by the shooting frame 301 becomes smaller, as shown in A of Fig. 23. On the other hand, if the user holds up their operating hand and twists their operating hand rightward while touching the central area 103C, the zoom magnification increases and the shooting range indicated by the shooting frame 301 becomes larger, as shown in B of Fig. 23. Then, when the central area 103C is clicked, the subject within the shooting range indicated by the shooting frame 301 is photographed, and image data is generated.

[0206] On the other hand, for example, if the operating hand is moved down while the central area 103C is touched, the photographing mode ends and the screen display mode is entered. Also, with the photographing mode ending, the photographing application ends, and the photographing frame 301, the focus frame 302, the scale 303, and the zoom bar 304 disappear.

[0207] In this way, the user can photograph a desired subject at a desired zoom magnification by simply touching or clicking the touchpad 103 of the input device 12 with his or her thumb, or by raising or twisting the operating hand.

[0208] It should be noted that different applications may be activated and used depending on the combination of other areas of the touchpad 103 and the motion of the operating hand.

[0209] As described above, by dividing the area of ​​the touchpad 103 of the input device 12 and assigning different functions to each area, the user can perform a variety of operations using the small touchpad 103. Furthermore, as described above, erroneous detection of the operation area is suppressed, so the user can reliably perform the desired operation using the touchpad 103.

[0210] <Second Example of Information Presentation Method of Information Processing System 1> Next, a second example of a method in which the information processing system 1 presents presentation information will be described with reference to FIGS. 14 to 23 and in accordance with the flowchart of FIG.

[0211] In a second example, the input device 12 is used to operate an audio hierarchical menu including multiple levels and items.

[0212] For example, as shown in FIG. 24A, when the bottom area 103B is touched, the audio hierarchical menu is activated.

[0213] Next, while the bottom end area 103B is being touched, the operating hand is twisted counterclockwise or clockwise, as shown in A and B of Fig. 25, to sequentially specify items in the menu. For example, as shown in B of Fig. 24, while the bottom end area 103B is being touched, the operating hand is twisted counterclockwise or clockwise, to sequentially specify items A to D based on the angle of rotation from the time when the bottom end area 103B was touched.

[0214] Next, when the bottom area 103B is clicked, the specified item (hereinafter referred to as the "specified item") is selected, and a process corresponding to the selected specified item (hereinafter referred to as the "selected item") is performed. For example, the audio hierarchical menu moves to the next hierarchical level of the selected item, or a function corresponding to the selected item is executed.

[0215] Here, a specific example of an audio hierarchical menu will be described.

[0216] 26 shows an example of items in the first layer of the audio hierarchical menu. The first layer includes four items: "Start motion operation," "Schedule," "Message," and "Recording."

[0217] For example, when the bottom area 103B is touched, a sound effect indicating the start of the audio hierarchical menu (hereinafter referred to as a start sound) and a sound saying "Start motion operation" are output. In other words, the activation (start) of the audio hierarchical menu is controlled by the touch operation on the bottom area 103B.

[0218] Next, when the operating hand is twisted clockwise while touching the bottom end area 103B, the designated items move in the order of "Schedule," "Message," and "Recording." Conversely, when the operating hand is twisted counterclockwise while touching the bottom end area 103B, the designated items move in the order of "Recording," "Message," and "Schedule." In other words, the items move between the items in the first layer of the audio hierarchical menu. Note that the "Start motion operation" item is not designated except when the audio menu is started.

[0219] Then, a sound effect indicating that the designated item has been moved (hereinafter referred to as the "designated sound") and a voice indicating the designated item to which it has been moved are output. For example, when "Schedule" is designated, the designated sound is output, followed by the voice "Schedule."

[0220] In this way, movement between items in the audio hierarchical menu is controlled based on the touch operation on the bottom area 103B and the twisting operation of the operating hand.

[0221] Next, when the bottom area 103B is clicked, the specified item is selected, a sound effect indicating that the specified item has been selected (hereinafter referred to as a selection sound) is output, and the audio hierarchical menu transitions to the next hierarchical level (second hierarchical level) of the selected item. In other words, the click operation on the bottom area 103B controls the selection of an item in the audio hierarchical menu.

[0222] Next, when the finger is released from the touchpad 103 and the bottom area 103B is touched again within a predetermined time, a designation sound is output and designation of an item in the second hierarchical level begins. In other words, the touch operation on the bottom area 103B controls movement between levels in the audio hierarchical menu.

[0223] Then, as in the first layer, by twisting the operating hand counterclockwise or clockwise while the bottom end area 103B is still touched, an item in the second layer is designated based on the rotation angle based on the point in time when the bottom end area 103B is touched.

[0224] For example, as shown in Figure 27, when "Schedule" is selected in the first layer, schedules from the current time onwards can be specified in chronological order in the second layer. Item A corresponds to the schedule closest to the current time. Item B corresponds to the schedule second closest to the current time. Item C corresponds to the schedule third closest to the current time. Item D corresponds to the schedule fourth closest to the current time.

[0225] As the designated item moves, a designated sound and a voice relating to the schedule corresponding to the designated item (for example, a voice reading out the contents of the schedule) are output.

[0226] For example, when item A is specified, a specified sound is output, followed by a voice saying, "The regular project meeting will start at 10 o'clock in Shinagawa." When item B is specified, a specified sound is output, followed by a voice saying, "The regular software team meeting will start remotely at 12 o'clock." When item C is specified, a specified sound is output, followed by a voice saying, "There will be a meeting with the manufacturer in Shinagawa from 2 p.m." When item D is specified, a specified sound is output, followed by a voice saying, "There will be a farewell party for Tanaka-san in Ebisu from 6 p.m."

[0227] Thereafter, when the bottom area 103B is clicked, a voice message "Motion operation finished" is output, and the voice hierarchical menu is terminated.

[0228] For example, as shown in Figure 28, when "Messages" is selected in the first layer, messages received in the message application can be specified in the second layer in order from the most recent to the most recent. Item A corresponds to the most recent message (newly arrived message). Item B corresponds to the second most recent message. Item C corresponds to the third most recent message. Item D corresponds to an operation to move to the next message list.

[0229] As the designated item moves, a designated sound and a sound related to the message corresponding to the designated item (designated message) are output. For example, a sound indicating the arrival time and sender of the designated message, and a sound including a readout of the message content are output.

[0230] For example, when item A is specified, a designated sound is output, followed by a voice saying, "New message from Suzuki, are you free this weekend?" When item B is specified, a designated sound is output, followed by a voice saying, "Message from Takashi at 9:20, package arrived this morning." When item C is specified, a designated sound is output, followed by a voice saying, "Message from Miho at 8:30, pick up some milk on your way home today." When item D is specified, a designated sound is output, followed by a voice saying, "Next message list."

[0231] Thereafter, when the bottom area 103B is clicked with an item other than item D designated, a voice message "Motion operation finished" is output, and the voice hierarchical menu is terminated.

[0232] On the other hand, when the bottom area 103B is clicked while item D is specified, the next message in the message list is played. For example, messages preceding the message corresponding to the current item C are assigned to items A through C in order of most recent. Then, using the method described above, it becomes possible to output audio for the newly assigned message.

[0233] For example, as shown in Fig. 29, when "Record" is selected in the first layer, the timing for recording audio is specified in the second layer. Item A corresponds to recording audio from now. Item B corresponds to recording audio from 30 seconds ago to the present. Item C corresponds to recording audio from 1 minute ago to the present. Item D corresponds to recording audio from 3 minutes ago to the present.

[0234] Then, as the designated item moves, a designated sound and a voice corresponding to the designated item are output. For example, when item A is designated, a designated sound is output, followed by a voice saying "Recording now." When item B is designated, a designated sound is output, followed by a voice saying "Recording from 30 seconds ago." When item C is designated, a designated sound is output, followed by a voice saying "Recording from one minute ago." When item D is designated, a designated sound is output, followed by a voice saying "Recording from three minutes ago."

[0235] Thereafter, when the bottom area 103B is clicked while the item A is designated, a voice message saying "Recording has started" is output, and voice recording begins. That is, saving of the voice data collected via the microphone 56 in the storage unit 59 begins.

[0236] When the bottom area 103B is clicked while the item B is specified, a voice message saying "Recorded" is output. Then, of the voice data accumulated in the storage unit 59, the voice data from 30 seconds before to the present is saved.

[0237] When the bottom area 103B is clicked while the item C is specified, a voice message saying "Recorded" is output. Then, of the voice data accumulated in the storage unit 59, the voice data from one minute before to the present is saved.

[0238] When the bottom area 103B is clicked while the item D is specified, a voice message saying "Recorded" is output. Then, of the voice data accumulated in the storage unit 59, the voice data from three minutes ago to the present is saved.

[0239] In the first layer of the audio hierarchical menu, if the bottom area 103B is not clicked and the finger is released, the audio hierarchical menu closes. In the first layer of the audio hierarchical menu, if the bottom area 103B is not touched within a predetermined time after being clicked, the audio hierarchical menu times out and closes. In the second layer of the audio hierarchical menu, if the bottom area 103B is not clicked and the finger is released, the audio hierarchical menu closes.

[0240] In this way, the user can specify a desired item from a menu consisting of multiple layers and multiple items in a short time and with a low load.

[0241] For example, a user can operate the audio hierarchical menu and execute a predetermined function simply by touching or clicking the bottom area 103B and twisting the operating hand, without operating multiple buttons. Furthermore, by using a touch operation to "trigger" and a click operation to "confirm (select)," intuitive operation with fewer operational errors is realized. Furthermore, operation using the rotation of the wrist (twist operation) allows the user to operate without being restricted by the environment or situation as much as possible. Furthermore, by operating while listening to sound effects and voice readings of designated items, reliable operation is realized.

[0242] For example, at each level of the audio hierarchical menu, different background sounds (e.g., background sounds with different tones) may be output depending on the specified item. The background sounds are sounds output in the background of the above-mentioned audio. For example, when item A is specified, background sounds with a note of "C" may be output, and when item B is specified, background sounds with a note of "Re" may be output.

[0243] This allows the user to easily recognize the position of the specified item in each layer of the audio hierarchical menu.

[0244] Furthermore, for example, the background sound may be changed for each layer, for example, different background sounds may be output when item A in the first layer is specified and when item A in the second layer is specified.

[0245] This allows the user to easily recognize the hierarchical level specified in the audio hierarchical menu and the position within the hierarchical level.

[0246] Furthermore, for example, the background sound may be changed for each item specified in a higher hierarchy. For example, when "Schedule," "Message," or "Recording" is specified in the first hierarchy of the audio hierarchical menu, the background sound output in the second hierarchy may be changed.

[0247] This allows the user to easily recognize the contents of the item specified in the upper layer.

[0248] Also, for example, it may be possible to move to a higher level in the audio hierarchical menu by an operation different from the above-described operation.

[0249] Furthermore, for example, the audio hierarchical menu may be allocated to an area other than the bottom area 103B.

[0250] Also, for example, different audio hierarchical menus may be assigned to two or more areas among the central area 103C, the left edge area 103L, the right edge area 103R, and the bottom edge area 103B of the touchpad 103, respectively.

[0251] Alternatively, for example, an item may be designated by touching each area of ​​the touchpad 103 and then sliding a finger within the area instead of twisting the operating hand. For example, after touching the bottom end area 103B, the designated item may be moved based on the position of the finger by sliding the finger within the bottom end area 103B.

[0252] Furthermore, for example, an item may be designated by touching each area of ​​the touchpad 103 and then swiping a finger within the area. For example, by touching the bottom end area 103B and then swiping a finger within the bottom end area 103B, the designated item may move depending on the amount of displacement of the touch position during the swipe and the speed of the swipe operation.

[0253] Furthermore, for example, if the input device 12 is capable of outputting audio and executing various functions (for example, if the input device 12 is a ring-type information processing device), the input device 12 can independently control the audio hierarchical menu and execute the function corresponding to the specified or selected item.

[0254] Furthermore, for example, the touch operation and twist operation may be applied to operations other than the above-described item selection. For example, the volume may be increased / decreased by a similar operation. For example, different functions may be executed depending on the direction of the twist operation after the touch operation. For example, if the operating hand is twisted clockwise after touching the touchpad 103, a "back" function may be executed, and if the operating hand is twisted counterclockwise, a function corresponding to the "HOME" button may be executed.

[0255] In the above description, an example has been described in which an operation is initiated by touching the touchpad 103 as a trigger, but a touch sensor may be used instead of the touchpad 103 .

[0256] <Third Example of Information Presentation Method of Information Processing System 1> Next, with reference to FIGS. 30 to 37, a third example of a method in which the information processing system 1 presents presentation information will be described in accordance with the flowchart of FIG.

[0257] In the third example, as shown in FIG. 30, the above-described pinch operation, release operation, tap operation, and twist operation are performed with the thumb and index finger of the operating hand.

[0258] For example, pinching and tapping operations can be easily detected by placing a sensor on the pad of the thumb or index finger, or by using a glove-type input device that fits over each finger. However, the input device can be an obstacle to daily life.

[0259] In contrast to this, it would be a great advantage if pinch operations and tap operations could be detected using only the motion sensor 104 of the ring-shaped input device 12 .

[0260] Fig. 31 shows an example of the waveforms of the sensor values ​​of the acceleration sensor and gyro sensor of the motion sensor 104 when a pinch operation is performed. Fig. 32 shows an example of the waveforms of the sensor values ​​of the acceleration sensor and gyro sensor of the motion sensor 104 when a release operation is performed. Fig. 33 shows an example of the waveforms of the sensor values ​​of the acceleration sensor and gyro sensor of the motion sensor 104 when a tap operation is performed.

[0261] In each figure, the upper graph shows the detected acceleration values ​​(hereinafter referred to as acceleration values) of the acceleration sensor in the x-axis, y-axis, and z-axis directions. The horizontal axis represents frames, and the vertical axis represents acceleration (m / s ２ ) In each figure, the lower graph shows the detected values ​​of angular velocity (hereinafter referred to as angular velocity values) around the x-axis, y-axis, and z-axis of the gyro sensor. The horizontal axis represents frames, and the vertical axis represents angular velocity (rad / s).

[0262] These graphs show that the acceleration sensor can detect the impact when the pad of a finger touches or releases the touchscreen. A certain amount of impact also occurs during the release operation. In particular, by looking at the high-frequency vibration components of the acceleration, it becomes easier to detect the impact when the pad of a finger touches or releases the touchscreen, without being affected by macroscopic movements.

[0263] On the other hand, a gyro sensor can detect the rotation of the finger when bending and straightening it. This rotation is a macro angle change, so it is best to look at the low frequency component.

[0264] Therefore, the acceleration sensor detects the deformation of the finger when the finger pad comes into contact or separates, and the gyro sensor detects the subsequent rotation of the finger, thereby making it possible to detect pinch operations, release operations, and tap operations.

[0265] The position of the rotation direction (circumferential direction of the index finger) of the motion sensor 104 of the input device 12 relative to the back of the index finger differs depending on the wearing position of the input device 12 in the rotation direction. As a result, even if the movement of the index finger is the same, the angular velocity value of the gyro sensor changes depending on the position of the motion sensor 104.

[0266] For example, Figures 34 to 36 show examples of angular velocity values ​​around the x-axis, y-axis, and z-axis of the gyro sensor of the motion sensor 104 when the index finger is bent and then extended during a tap operation when the mounting position of the input device 12 in the rotational direction is changed.

[0267] Fig. 34 shows an example of angular velocity values ​​when the input device 12 is worn so that the motion sensor 104 is positioned directly above the index finger (on the back of the index finger). Fig. 35 shows an example of angular velocity values ​​when the input device 12 is worn so that the motion sensor 104 is positioned to the left of the back of the index finger (closer to the thumb). Fig. 35 shows an example of angular velocity values ​​when the input device 12 is worn so that the motion sensor 104 is positioned to the right of the back of the index finger (closer to the middle finger). The horizontal axis of the graph in each figure represents frames, and the vertical axis represents angular velocity (rad / s).

[0268] In this way, the changes in the angular velocity values ​​around the x-axis, y-axis, and z-axis during a tap operation differ depending on the mounting position of the input device 12.

[0269] In response to this, for example, the detection accuracy of pinch operations, release operations, and tap operations can be improved by detecting the mounting position of the input device 12 in the rotational direction and detecting the rotational direction when the index finger is bent and straightened based on the detection results.

[0270] <User Operation Detection Processing> Next, the user operation detection processing executed by the information processing device 11 will be described with reference to the flowchart of FIG.

[0271] This process is executed, for example, in step S33 of FIG.

[0272] In step S101, the user operation detection unit 61 acquires acceleration values. That is, the user operation detection unit 61 acquires acceleration values ​​in the x-axis direction, y-axis direction, and z-axis direction detected by the motion sensor 104 from motion data included in input information received from the input device 12.

[0273] In step S102, the user operation detection unit 61 applies a high-pass filter (HPF) to the acceleration value, i.e., extracts high-frequency components of the acceleration value in the x-axis direction, the y-axis direction, and the z-axis direction.

[0274] In step S103, the user operation detection unit 61 calculates the norm of the acceleration based on the high-frequency components of the acceleration values ​​in the x-axis direction, y-axis direction, and z-axis direction.

[0275] In step S104, the user operation detection unit 61 performs a process of detecting an acceleration peak. For example, the user operation detection unit 61 executes a process of detecting a maximum of one peak from a history of acceleration norms within a recent predetermined period (e.g., the past one second).

[0276] In step S105, the user operation detection unit 61 determines whether or not an acceleration peak has been detected based on the result of the process in step S104. If it is determined that an acceleration peak has been detected, the process proceeds to step S106.

[0277] In step S106, the user operation detection unit 61 performs a filtering process using deep learning. That is, the user operation detection unit 61 performs a filtering process using a classifier previously generated by deep learning to determine whether a user operation, either a pinch operation or a release operation, has been performed based on the waveforms of the acceleration values ​​in the x-axis direction, y-axis direction, and z-axis direction.

[0278] In step S107, the user operation detection unit 61 determines whether or not a user operation has been performed based on the processing in step S106. If it is determined that a user operation has been performed, the processing proceeds to step S108.

[0279] In step S108, the user operation detection unit 61 acquires angular velocity values ​​(gyro values). That is, the user operation detection unit 61 acquires angular velocity values ​​around the x-axis, y-axis, and z-axis detected by the motion sensor 104 from the motion data included in the input information received from the input device 12.

[0280] In step S109, the user operation detection unit 61 calculates the rotation direction of the index finger on which the input device 12 is worn, based on the acquired angular velocity value.

[0281] In this case, for example, as described above with reference to Figures 34 to 36, the rotation direction of the index finger is calculated taking into account the wearing position of the input device 12 in the rotation direction, thereby improving the detection accuracy of the rotation direction of the index finger.

[0282] In step S110, the user operation detection unit 61 determines whether the index finger has moved in a direction approaching the thumb based on the result of the process in step S109. If it is determined that the index finger has moved in a direction approaching the thumb, the process proceeds to step S111.

[0283] In step S111, the user operation detection unit 61 determines that a pinch operation has been performed.

[0284] Thereafter, the process returns to step S101, and the processes from step S101 onwards are executed.

[0285] On the other hand, if it is determined in step S110 that the index finger has moved in a direction away from the thumb, the process proceeds to step S112.

[0286] In step S112, the user operation detection unit 61 determines that a release operation has been performed.

[0287] Thereafter, the process returns to step S101, and the processes from step S101 onwards are executed.

[0288] On the other hand, if it is determined in step S107 that no user operation has been performed, the process returns to step S101, and the processes from step S101 onwards are executed.

[0289] If it is determined in step S105 that the acceleration peak has not been detected, the process returns to step S101, and the processes from step S101 onwards are executed.

[0290] In this manner, the pinch operation and the release operation can be detected using only the motion sensor 104 of the input device 12 .

[0291] It should be noted that, for example, if a pinch operation and a release operation are consecutive, that is, if a release operation is detected within a predetermined time after a pinch operation is detected, it is determined that a tap operation has been performed.

[0292] Furthermore, the twisting operation can be detected based on the angular velocity and acceleration of the main body of the input device 12 detected by the motion sensor 104 .

[0293] In the above description, an example has been given in which a filtering process using deep learning is performed to detect whether a user operation has been performed, but a filtering process using deep learning may also be performed when determining the type of user operation based on, for example, an angular velocity value. Both or only one of these filtering processes using deep learning may be performed.

[0294] It should be noted that the pinch operation, release operation, tap operation, and twist operation can be applied to various operations of the information processing device 11.

[0295] For example, when the information processing device 11 plays music, volume adjustment may be initiated by a knob operation, the volume may be increased or decreased based on the rotation angle and direction of a twist operation, and the volume may be determined by a release operation.

[0296] For example, a tap operation may start playback of a song, and a "double tap" may skip to the next song. Here, a "double tap" refers to a tap operation performed consecutively within a predetermined time.

[0297] For example, while the knob operation is being performed, the playback of music may be stopped, a transition may be made to a mode that captures external sounds, and when the knob operation ends, the playback of music may be resumed.

[0298] Also, for example, if the virtual object is a three-dimensional image (including the above-mentioned three-dimensional objects and UIs), the virtual object may be directly manipulated by the above-mentioned grabbing and releasing operations, and the display of the virtual object may be controlled.

[0299] As described above, by simply wearing the input device 12, pinch operations, release operations, tap operations, and twist operations can be detected without interfering with the user's daily life, thereby improving the operability of the information processing system 1.

[0300] It should be noted that, for example, in order to prevent erroneous detection of continuous pinching, the detection of pinching operation may be terminated if the pinching operation continues for a predetermined period of time or more.

[0301] For example, when two or more release operations are detected consecutively, the latter may be ignored as an erroneous detection.

[0302] For example, when consecutive pinch operations are detected, a pseudo release operation may be added in between to prevent the operations from contradicting each other.

[0303] The first to third examples of the information presentation method for information processing described above can be combined to the extent possible.

[0304] As described above, the information processing system 1 enables operation input to be performed more naturally, with less load, and with less hassle, during daily activities.

[0305] <<2. Modifications>> Next, modifications of the present technology will be described.

[0306] <Modifications Regarding Allocation of Processing Between Information Processing Device 11 and Input Device 12> The allocation of processing between the information processing device 11 and the input device 12 can be changed as appropriate.

[0307] For example, the input device 12 may be provided with a function equivalent to the user operation detection unit 61 of the information processing device 11, and the input device 12 may detect user operations based on touchpad data from the touchpad 103 and motion data from the motion sensor 104.

[0308] In this case, for example, the input information generation unit 111 generates input information including user operation information related to the detected user operation and transmits it to the information processing device 11. The information processing device 11 controls the presentation of the presentation information based on the user operation information included in the input information. Also, for example, all or part of the above-mentioned detection information may be included in the input information.

[0309] For example, the information processing device 11 and the input device 12 may share the responsibility of detecting a user operation. For example, the input device 12 may detect the operation area of ​​the touchpad 103, and the information processing device 11 may detect the type of operation performed on the operation area. For example, the input device 12 may detect the movement of the main body of the input device 12, and the information processing device 11 may detect or estimate a gesture made by the user's operating hand or finger based on the movement of the main body of the input device 12.

[0310] In this case, for example, the detection information included in the input information includes the detection result of the operation area and the detection result of the movement of the main body of the input device 12 .

[0311] <Modifications Regarding the Method of Wearing the Input Device 12> The input device 12 can be worn on a finger other than the thumb. For example, when the input device 12 is worn on a finger other than the index finger, the pinch operation, release operation, and the like are detected between the finger on which the input device 12 is worn and the thumb.

[0312] The input device 12 may be attached to the proximal joint (the joint at the base of the finger) or the middle joint (the joint in the middle of the finger). The user may also change the attachment position as appropriate depending on the usage situation.

[0313] The input device 12 may be worn on a part of the hand (for example, the palm) beyond the wrist of the user, as long as it is in a position that allows operation with the fingers.

[0314] <Modifications Related to Operation Input Sensor> The method of dividing the area of ​​the touchpad 103 is not limited to the example in Fig. 7. For example, an upper end area including the upper end of the touch sensor 133 may be provided, or part of the edge area may be removed.

[0315] Furthermore, it is not necessary to provide an edge area for each edge of the touch sensor 133. For example, an edge area may be provided that includes two or more edges (e.g., the bottom edge and the left edge) of the touch sensor 133. Furthermore, for example, an edge area may be provided that includes part of the edges of the touch sensor 133 (e.g., the bottom edge and part of the left edge, part of the right edge, etc.).

[0316] 38 and 39 show an example of the configuration of a touchpad 401, which is a modified example of the touchpad. In the figures, parts corresponding to the touchpad 103 in Fig. 11 are given the same reference numerals. Also, as in Fig. 11, the squares with a black square in the center of the touch sensor 133 indicate reaction nodes.

[0317] The touchpad 401 differs from the touchpad of FIG. 11 in that a metal housing 411 is provided instead of the base member 151, and the touch sensors 152L, 152R, and 152B are omitted.

[0318] The metal housing 411 has substantially the same shape as the base member 151, and the touch sensor 133 is provided on the top surface of the metal housing 411. Like the touch sensor 133, the metal housing 411 is connected to an electrostatic control IC 402 that can detect the electrostatic capacitance of each node, and forms one node (hereinafter referred to as the base node) as a whole. The edge portions and side surfaces of the top surface of the metal housing 411 are exposed, and these exposed portions form the edge areas (left end area, right end area, and bottom end area) of the touch pad 401.

[0319] For example, the base node reacts when the user's finger touches any of the edge areas, the left edge area, the right edge area, and the bottom edge area, of the touchpad 401. However, because the base node as a whole forms one node, it is not possible to detect which edge area has been touched by itself.

[0320] On the other hand, when a reaction of the base node is detected, it is possible to detect which edge area has been touched by aligning and detecting the reaction node of the touch sensor 133. For example, in the example of Fig. 39, the nodes on the right side of the touch sensor 133 are more likely to react. Therefore, in this case, it is detected that the right edge area has been touched.

[0321] In this way, the touch pad 401 can detect a touch on each edge area without providing a touch sensor for each edge area.

[0322] For example, an operation input sensor other than a touchpad may be used.

[0323] For example, when only the audio hierarchical menu is operated, an operation input sensor (for example, a pressure sensor) capable of detecting touch operations and click operations may be used instead of the touchpad.

[0324] For example, an operation input sensor capable of detecting a proximity operation performed by bringing a user's finger close to the operation input may be used. In this case, the operation input sensor may be capable of detecting both a proximity operation and a touch operation.

[0325] <Modifications Related to the Operation Target of the Input Device 12> An information processing device capable of displaying virtual objects other than the above-described glasses-type device may be used as the information processing device 11. For example, the information processing device 11 may be a smartphone or a tablet terminal, or may be a display device (wearable over one ear) that presents an image to one eye of the user.

[0326] For example, the object to be operated by the input device 12 is not limited to a virtual object superimposed on real space. For example, it may be a virtual object in a virtual space displayed on the information processing device 11. Furthermore, when the information processing device 11 is realized by a projector, the object to be operated may be a virtual object included in an image projected by the projector.

[0327] The input device 12 is not limited to operation input to the information processing device 11 as described above, and may be used for operation input to, for example, an audio output device. The audio output device may be various speakers worn by the user (neck speakers, earphone speakers, headphone speakers), or may be audio equipment installed in a room or a car.

[0328] For example, the input device 12 may be used in an information processing device other than a wearable device such as an HMD, etc. For example, the input device 12 may be used for remotely controlling an information processing device such as an information device or home appliance that uses a hierarchical menu.

[0329] For example, the input device 12 may be used alone, for example, as a device for communicating with others.

[0330] <<3. Others>> <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, a program constituting the software is installed in the computer of the information processing device 11 or the input device 12. Here, the computer includes a computer incorporated in dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0331] The program to be executed by the computer can be provided by being recorded on removable media such as package media, etc. The program can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0332] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0333] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0334] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0335] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0336] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0337] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0338] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0339] (1) An information processing device comprising: a communication unit that wirelessly communicates with an input device including a main body unit worn on a user's wrist, an operation input sensor that detects operation input to the main body unit, and a motion sensor that detects movement of the main body unit, and receives input information including at least one of detection information regarding a detection result of the operation input and the movement of the main body unit, and user operation information regarding a user operation detected based on the detection result of the operation input and the movement of the main body unit, and a presentation control unit that controls presentation of presentation information that is information to be presented to the user based on the user operation detected based on the detection information or the user operation information. (2) The information processing device described in (1), wherein the presentation information includes information regarding a hierarchical menu, and the presentation control unit controls movement between levels and between items in the hierarchical menu and selection of the item based on the user operation. (3) The information processing device described in (2), wherein the user operations include a touch operation and a click operation on the main body unit, and a twist operation of twisting the user's hand on which the main body unit is worn, and the presentation control unit controls movement between the items based on the touch operation and the twist operation, and controls selection of the item based on the click operation. (4) The information processing device described in (3), wherein the presentation control unit controls activation of the hierarchical menu and movement between the hierarchical levels based on the touch operation. (5) The information processing device described in any of (2) to (4), wherein the presentation control unit controls output of sound related to the item specified in the hierarchical menu. (6) The information processing device described in any of (2) to (5), wherein the main body unit is a ring type worn on the user's finger, and the operation input sensor includes a touch sensor and a switch, or a pressure sensor.(7) The information processing device according to any one of (1) to (6), wherein the main body is a ring-shaped body worn on a first finger of the user, the operation input sensor includes a touch sensor that can be touched by a second finger of the same hand as the first finger when the main body is worn on the first finger, and further includes a user operation detection unit that individually detects operations on a central area including a central region of the touch sensor and one or more edge areas including at least one edge of the touch sensor. (8) The information processing device according to (7), wherein the touch sensor is capable of individually detecting contact with a plurality of nodes, and the user operation detection unit detects the operation on the edge area based on at least one of a center of a touch area that is an area where the touch sensor is touched, the number or ratio of the nodes in the edge area among the nodes in the touch area, and the electrostatic capacitance of each of the nodes. (9) The information processing device according to any of (1) to (6), wherein the main body is a ring type worn on a first finger of the user, the motion sensor includes an acceleration sensor and a gyro sensor, and further includes a user operation detection unit that detects, based on the acceleration and angular velocity of the main body, at least one of a pinch operation of bringing the pad of the first finger into contact with the pad of a second finger of the same hand as the first finger, a release operation of separating the pad of the first finger from the pad of the second finger that are in contact with the pad of the first finger, a tap operation of bringing the pad of the first finger into contact with the pad of the second finger and then separating them within a predetermined time, and a twist operation of twisting the hand including the first finger and the second finger with the pad of the first finger in contact with the pad of the second finger. (10) The information processing device according to any of (1) to (9), further includes a gaze information acquisition unit that acquires gaze information of the user, and the presentation control unit controls presentation of the presentation information further based on the gaze information.(11) An information processing method in which an information processing device wirelessly communicates with an input device including a main body part worn on the wrist of a user, an operation input sensor that detects an operation input to the main body part, and a motion sensor that detects movement of the main body part, receives input information including at least one of detection information regarding a detection result of the operation input and the movement of the main body part, and user operation information regarding a user operation detected based on the detection result of the operation input and the movement of the main body part, and controls presentation of presentation information that is information to be presented to the user, based on the user operation detected based on the detection information or the user operation information. (12) An input device comprising: a main body part worn on the wrist of a user, the operation input sensor that detects an operation input to the main body part, a motion sensor that detects movement of the main body part, an input information generation unit that generates input information including at least one of detection information regarding a detection result of the operation input and the movement of the main body part, and user operation information regarding the user operation detected based on the detection result of the operation input and the movement of the main body part, and a communication unit that wirelessly communicates with an information processing device that controls the presentation of presentation information that is information to be presented to the user, and transmits the input information. (13) The input device according to (12), wherein the main body is a ring type that is worn on the user's finger, the operation input sensor includes a touch sensor and a switch or a pressure sensor, the user operations include touch operations and click operations on the main body, and twist operations of the user's hand on which the main body is worn, the presented information includes information related to a hierarchical menu, and movement between items in the hierarchical menu is controlled based on the touch operations and the twist operations, and selection of the items is controlled based on the click operations.(14) The input device according to (12) or (13), wherein the main body is a ring-shaped body worn on a first finger of the user, the operation input sensor includes a touch sensor that can be touched by a second finger of the same hand as the first finger when the main body is worn on the first finger, and further includes a user operation detection unit that individually detects operations on a central area including a central region of the touch sensor and one or more edge areas including at least one edge of the touch sensor. (15) The touch sensor is capable of individually detecting contact with a plurality of nodes, and the user operation detection unit detects the operation on the edge area based on at least one of a center of a touch area that is an area where the touch sensor is touched, the number or ratio of the nodes in the edge area among the nodes in the touch area, and the electrostatic capacitance of each of the nodes. (16) The input device according to (12) or (13), wherein the main body is a ring type that is worn on a first finger of the user, the motion sensor includes an acceleration sensor and a gyro sensor, and the input device further includes a user operation detection unit that detects, based on the acceleration and angular velocity of the main body, at least one of a pinch operation that brings the pad of the first finger into contact with the pad of a second finger of the same hand as the first finger, a release operation that separates the pad of the first finger from the pad of the second finger that are in contact, a tap operation that brings the pad of the first finger into contact with the pad of the second finger and then separates them within a predetermined time, and a twist operation that twists the hand including the first finger and the second finger while the pad of the first finger is in contact with the pad of the second finger. (17) An information processing method in which an input device including a main body worn on a user's wrist, an operation input sensor that detects operation input to the main body, and a motion sensor that detects movement of the main body generates input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body, and user operation information regarding user operation detected based on the detection results of the operation input and the movement of the main body, and wirelessly communicates with an information processing device that controls presentation of presentation information, which is information to be presented to the user, and transmits the input information.(18) An information processing system comprising: a main body unit worn on a user's wrist; an operation input sensor that detects an operation input to the main body unit; a motion sensor that detects movement of the main body unit; a user operation detection unit that detects a user operation based on the operation input detected by the operation input sensor and the movement of the main body unit detected by the motion sensor; and a presentation control unit that controls presentation of presentation information that is information to be presented to the user based on the user operation. (19) The presentation information includes information related to a hierarchical menu, and the presentation control unit controls movement between levels and between items in the hierarchical menu and selection of the items based on the user operation. (20) The information processing system according to (18) or (19), comprising: an input device that includes the main body unit, the operation input sensor, and the motion sensor; and an information processing device that includes the presentation control unit, wherein the user operation detection unit is provided in at least one of the input device and the information processing device.

[0340] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0341] REFERENCE SIGNS LIST 1 Information processing system, 11 Information processing device, 12 Input device, 51 Communication unit, 52 Control unit, 53 Gaze detection unit, 54 Outward camera, 55 Operation input unit, 56 Microphone, 57 Display unit, 58 Audio output unit, 61 User operation detection unit, 62 Presentation control unit, 101 Communication unit, 102 Control unit, 103 Touch pad, 104 Motion sensor, 105 Vibration unit, 111 Input information generation unit, 112 Vibration control unit, 131 Wireless communication microcomputer, 133 Touch sensor, 134 Tactile switch

Claims

1. An information processing device comprising: a main body worn on a user's wrist; an operation input sensor that detects operation input to the main body; and a communication unit that wirelessly communicates with an input device that has a motion sensor that detects movement of the main body, and receives input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body, and user operation information regarding a user operation detected based on the detection results of the operation input and the movement of the main body; and a presentation control unit that controls the presentation of presentation information, which is information to be presented to the user, based on the user operation detected based on the detection information or the user operation information.

2. The information processing device according to claim 1, wherein the presentation information includes information relating to a hierarchical menu, and the presentation control unit controls movement between levels and items in the hierarchical menu, as well as selection of the items, based on the user operation.

3. The information processing device according to claim 2, wherein the user operations include touch operations and click operations on the main body unit, and a twist operation of twisting the hand on which the user is wearing the main body unit, and the presentation control unit controls movement between the items based on the touch operations and the twist operations, and controls selection of the items based on the click operations.

4. The information processing device according to claim 3, wherein the presentation control unit controls activation of the hierarchical menu and movement between the hierarchical levels based on the touch operation.

5. The information processing device according to claim 2, wherein the presentation control unit controls output of a sound relating to the item specified in the hierarchical menu.

6. The information processing device according to claim 2, wherein the main body is a ring type that is worn on the user's finger, and the operation input sensor includes a touch sensor and a switch, or a pressure sensor.

7. The information processing device according to claim 1, wherein the main body is ring-shaped and worn on a first finger of the user, the operation input sensor includes a touch sensor that can be contacted by a second finger of the same hand as the first finger when the main body is worn on the first finger, and further includes a user operation detection unit that individually detects operations on a central area including a central region of the touch sensor and one or more edge areas including at least one edge of the touch sensor.

8. The information processing device according to claim 7, wherein the touch sensor is capable of individually detecting contact with multiple nodes, and the user operation detection unit detects an operation on the edge area based on at least one of the center of a touch area, which is the area where the touch sensor is touched, the number or ratio of the nodes in the touch area among the nodes in the edge area, and the capacitance of each of the nodes.

9. The information processing device according to claim 1, further comprising a user operation detection unit that detects, based on the acceleration and angular velocity of the main body unit, at least one of a pinch operation in which the pad of the first finger is brought into contact with the pad of a second finger of the same hand as the first finger, a release operation in which the pad of the first finger and the pad of the second finger that are in contact are separated, a tap operation in which the pad of the first finger is brought into contact with the pad of the second finger and then separated within a predetermined time, and a twist operation in which the hand including the first finger and the second finger is twisted while the pad of the first finger is in contact with the pad of the second finger.

10. An information processing device as described in claim 1, further comprising a gaze information acquisition unit that acquires gaze information of the user, wherein the presentation control unit controls the presentation of the presentation information further based on the gaze information.

11. An information processing method in which an information processing device wirelessly communicates with an input device that has a main body worn on the user's wrist, an operation input sensor that detects operation input to the main body, and a motion sensor that detects movement of the main body, receives input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body, and user operation information regarding a user operation detected based on the detection results of the operation input and the movement of the main body, and controls the presentation of presentation information, which is information to be presented to the user, based on the user operation detected based on the detection information or the user operation information.

12. An input device comprising: a main body unit worn on a user's wrist; an operation input sensor that detects an operation input to the main body unit; a motion sensor that detects movement of the main body unit; an input information generation unit that generates input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body unit, and user operation information regarding a user operation detected based on the detection results of the operation input and the movement of the main body unit; and a communication unit that wirelessly communicates with an information processing device that controls the presentation of presentation information, which is information to be presented to the user, and transmits the input information.

13. The input device according to claim 12, wherein the main body is ring-shaped and worn on the user's finger, the operation input sensor includes a touch sensor and a switch or a pressure sensor, the user operations include a touch operation and a click operation on the main body, and a twist operation of twisting the user's hand on which the main body is worn, the presented information includes information regarding a hierarchical menu, and movement between items in the hierarchical menu is controlled based on the touch operation and the twist operation, and selection of the item is controlled based on the click operation.

14. The input device according to claim 12, wherein the main body is ring-shaped and worn on a first finger of the user, the operation input sensor includes a touch sensor that can be contacted by a second finger of the same hand as the first finger when the main body is worn on the first finger, and further includes a user operation detection unit that individually detects operations on a central area including a central region of the touch sensor and one or more edge areas including at least one edge of the touch sensor.

15. The input device according to claim 14, wherein the touch sensor is capable of individually detecting contact with a plurality of nodes, and the user operation detection unit detects an operation on the edge area based on at least one of the center of a touch area, which is the area where the touch sensor is touched, the number or ratio of the nodes in the touch area among the nodes in the edge area, and the capacitance of each of the nodes.

16. The input device according to claim 12, further comprising a user operation detection unit that detects, based on the acceleration and angular velocity of the main body unit, at least one of a pinch operation in which the pad of the first finger is brought into contact with the pad of a second finger of the same hand as the first finger, a release operation in which the pad of the first finger and the pad of the second finger that are in contact are separated, a tap operation in which the pad of the first finger is brought into contact with the pad of the second finger and then separated within a predetermined time, and a twist operation in which the hand including the first finger and the second finger is twisted while the pad of the first finger is in contact with the pad of the second finger.

17. An information processing method in which an input device including a main body worn on a user's wrist, an operation input sensor that detects operation input to the main body, and a motion sensor that detects movement of the main body generates input information including at least one of detection information regarding the detection results of the operation input and the movement of the main body, and user operation information regarding user operation detected based on the detection results of the operation input and the movement of the main body, and wirelessly communicates with an information processing device that controls the presentation of presentation information, which is information presented to the user, to transmit the input information.

18. An information processing system comprising: a main body unit worn on a user's wrist; an operation input sensor that detects operation input to the main body unit; a motion sensor that detects movement of the main body unit; a user operation detection unit that detects user operation based on the operation input detected by the operation input sensor and the movement of the main body unit detected by the motion sensor; and a presentation control unit that controls the presentation of presentation information, which is information presented to the user, based on the user operation.

19. An information processing system as described in claim 18, wherein the presentation information includes information regarding a hierarchical menu, and the presentation control unit controls movement between levels and items in the hierarchical menu, as well as selection of the items, based on the user operation.

20. The information processing system according to claim 18, comprising: an input device having the main body, the operation input sensor, and the motion sensor; and an information processing device having the presentation control unit, wherein the user operation detection unit is provided in at least one of the input device and the information processing device.

Images