Information processing device, information processing method, and program

Abstract

The present invention makes it easy for a user to recognize that a received notification is from a different host if there are multiple hosts. Provided is an information processing device equipped with a control unit that sets haptic signals corresponding to notification information transmitted by a plurality of hosts as different haptic signals on the basis of a determination result, which is a result of determining host types of the plurality of hosts.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0002] In recent years, technology for providing notifications such as tactile feedback to users has been developed. For example, Patent Document 1 discloses a technology for causing a slave device to detect a contact state between a patient and a medical device in a master-slave system and causing a master device to feed back the contact state to an operator (user) such as a surgeon. In addition, Patent Document 1 discloses a tactile feedback device that provides easily perceptible feedback to an operator.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-192130 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] Here, the tactile feedback device that provides tactile feedback notifications to the user does not take into account the presence of multiple processing entities responsible for outputting notification information representing notifications to the tactile feedback device.

[0008] Therefore, the present disclosure proposes a technology that takes into account a case where there are a plurality of hosts as processing entities and makes it easier for a user to recognize notifications from different hosts.

[0009] Solution to the problem

[0010] According to the present disclosure, an information processing device is provided, which includes a control unit that converts a tactile signal corresponding to notification information sent by each of a plurality of hosts into a different tactile signal based on a determination result of an identification of a host type of the plurality of hosts.

[0011] Furthermore, according to the present disclosure, there is provided an information processing method including converting a tactile signal corresponding to notification information sent by each of a plurality of hosts into a different tactile signal based on a determination result of identification of a host type according to the plurality of hosts.

[0012] In addition, according to the present disclosure, a program is provided for causing a computer to function as a control unit, which performs the following processing: based on a determination result of an identification of a host type according to multiple hosts, a tactile signal corresponding to notification information sent by each of multiple hosts is converted into a different tactile signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram for describing an overview of an information processing system according to an embodiment of the present disclosure.

[0014] Figure 2 is a diagram for describing an example of objects displayed on the information processing device 10 and the terminal device 20 .

[0015] Figure 3 is an explanatory diagram for describing an example of host type identification performed by the determination unit 123 .

[0016] Figure 4 is an explanatory diagram for describing an example of host type identification performed by the determination unit 123 .

[0017] Figure 5 is an explanatory diagram for describing an example of host type identification performed by the determination unit 123 .

[0018] Figure 6 is an explanatory diagram for describing an example of host type identification performed by the determination unit 123 .

[0019] Figure 7 is an explanatory diagram for describing an example of host type identification performed by the determination unit 123 .

[0020] Figure 8 is an explanatory diagram showing an example in which the conversion unit 124 converts the frequency of the haptic signal.

[0021] Fig. 9 is an explanatory diagram showing an example in which the conversion unit 124 converts the amplitude of the haptic signal.

[0022] Fig.10 is an explanatory diagram showing an example in which the conversion unit 124 converts the waveform of the haptic signal.

[0023] Fig.11 is an explanatory diagram showing an example in which the conversion unit 124 converts a haptic signal into a fade-out signal.

[0024] Fig.12 is an explanatory diagram showing an example in which the conversion unit 124 converts the amplitude of the haptic signal into 0.

[0025] Fig.13 is an explanatory diagram showing an example in which the conversion unit 124 converts a tactile signal into a predetermined signal.

[0026] Fig.14 is an explanatory diagram showing an example in which the conversion unit 124 adds a buffer time before a haptic signal.

[0027] Fig.15is an explanatory diagram showing an example in which the conversion unit 124 changes the output destination of the haptic signal.

[0028] Fig.16 is an explanatory diagram showing an example in which the conversion unit 124 converts the frequency or amplitude of a haptic signal.

[0029] Fig.17 is an explanatory diagram showing an example of frequency band conversion in a case where the conversion unit 124 converts haptic signals from a plurality of hosts into haptic signals having non-overlapping frequency bands.

[0030] Fig.18 is an explanatory diagram showing an example of amplitude conversion in a case where the conversion unit 124 converts haptic signals from a plurality of hosts into haptic signals having non-overlapping amplitudes.

[0031] Fig.19 is an explanatory diagram illustrating an example in which the conversion unit 124 converts the amplitudes of the second haptic signal and the subsequent haptic signal when the same haptic signal is output from the same host within a certain period of time.

[0032] Fig. 20 is an explanatory diagram showing an example in which the conversion unit 124 changes the output timing of the haptic signal.

[0033] Fig.21 is a table showing the correspondence between the notification content indicated by the haptic signal and the signal transmitted from the information processing device 10 to the haptic feedback device 30 in a case where the host type is the main host or the sub-host.

[0034] Fig. 22 is an explanatory diagram for describing an example of tactile feedback in a case where a virtual desktop screen and a virtual object displayed by the information processing device 10 are displayed on the terminal device 20 .

[0035] Fig.23 is an explanatory diagram for describing an example in which tactile feedback is provided by the annular input device 30B including a pair of annular input devices.

[0036] Fig.24 is a sequence diagram showing an example of the flow of operation processing of the information processing system according to the present embodiment.

[0037] Fig.25 : is an explanatory diagram showing the configuration of the information processing system according to the second embodiment of the present disclosure and the internal configuration of the terminal device 40 .

[0038] Fig.26 is an explanatory diagram showing the configuration of an information processing system and the internal configuration of a tactile feedback device 30 according to a third embodiment of the present disclosure.

[0039] Fig. 27 is a sequence diagram illustrating an example of an operation processing flow of the information processing system according to the third embodiment of the present disclosure.

[0040] Fig.28 is an explanatory diagram showing the configuration of an information processing system and the internal configuration of a tactile feedback device 30 according to a fourth embodiment of the present disclosure.

[0041] Fig.29 is a block diagram showing an example of the information processing device 90 . DETAILED DESCRIPTION

[0042] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals to avoid redundant description.

[0043] Note that the description will be given in the following order.

[0044] 1. Overview of information processing system according to an embodiment (first embodiment) of the present disclosure

[0045] 2. Configuration Example of Information Processing Device 10

[0046] 3. Specific examples of tactile feedback

[0047] 4. Example of Operation Processing According to First Embodiment

[0048] 5. Second Implementation

[0049] 6. Third Implementation

[0050] 7. Fourth Implementation

[0051] 8. Hardware Configuration

[0052] 9. Supplement

[0053] <<1. Overview of Information Processing System According to Embodiment (First Embodiment) of the Present Disclosure>>

[0054] Figure 1 1 is a diagram for describing an overview of an information processing system according to an embodiment of the present disclosure. Figure 1 As shown in , the information processing system according to the present embodiment includes an information processing device 10 (an example of an information processing device), a terminal device 20 (an example of a second information processing device), and a tactile feedback device 30.

[0055] The information processing device 10 provides information to the user in response to the user's operation. In the present embodiment, visual information is used as an example of information provided to the user. Other examples of information provided to the user include auditory information provided by an audio output device. Figure 1 An example is shown in which the information processing device 10 is a personal computer (PC). In the present embodiment, a case where the information processing device 10 is a PC will be mainly described, but the information processing device 10 may also be a smartphone, a tablet terminal, or the like.

[0056] like Figure 1 As shown in , the information processing apparatus 10 includes an operation input unit 150 that receives an operation of a user. Figure 1 A mouse 150A and a keyboard 150B are shown as examples of components configured to receive an operation of a user. The mouse 150A and the keyboard 150B may be connected to the information processing device 10 by a wired or wireless connection. The user operates the operation input unit 150 while checking visual information provided by the information processing device 10. The information processing device 10 is connected to the terminal device 20, and various information is exchanged.

[0057] The terminal device 20 is a device that is worn on the user's head and provides a virtual space (virtual reality: VR) to the user. The terminal device 20 includes an image display unit (an example of a display unit) for each of the left eye and the right eye. The terminal device 20 is also capable of displaying different images to each of the left eye and the right eye, and can provide a three-dimensional virtual object by displaying an image with parallax to the left eye and the right eye.

[0058] Figure 1 An example is shown in which the terminal device 20 is a non-transparent head mounted display (HMD) covering the user's field of view. In this embodiment, the case where the terminal device 20 is an HMD will be mainly described, but the terminal device 20 may be, for example, a smartphone used in conjunction with augmented reality (AR) glasses or goggles.

[0059] The terminal device 20 is connected with the information processing device 10 and the tactile feedback device 30, and exchanges various information. The terminal device 20 is controlled by an application (also referred to as an HMD application or HMD app herein) executed by the information processing device 10 to operate the terminal device 20. For example, the information processing device 10 executes the HMD app to control the output of an image to be displayed on the terminal device 20. In addition, the information processing device 10 executes the HMD app to construct a virtual space to be provided by the terminal device 20.

[0060] The user can use the tactile feedback device 30 to input an operation to the virtual object displayed on the terminal device 20. The tactile feedback device 30 detects the user's operation input or the user's action through a button, a sensor, etc., and sends a signal indicating the user's operation input or a signal indicating the user's action to the information processing device 10 via the terminal device 20. In addition, the tactile feedback device 30 may include a sensor such as an infrared (IR) light emitting element, and the sensor may be allowed to detect the three-dimensional coordinates of the tactile feedback device 30. The three-dimensional coordinates of the tactile feedback device 30 may also be detected by a camera device or the like included in the terminal device 20.

[0061] Figure 1 An example is shown in which the terminal device 20 is connected to an input device such as a controller 30A, a ring-shaped input device 30B, a pen-shaped input device 30C, and a six-degree-of-freedom (6DoF) mouse 30D as examples of the tactile feedback device 30. In the present embodiment, an input device including a tactile feedback unit that gives a user tactile stimulation is used as an example of the tactile feedback device 30. Note that although Figure 1 An example in which the terminal device 20 is connected to a plurality of tactile feedback devices 30 is shown, but the number of tactile feedback devices 30 connected to the terminal device 20 is not limited to a specific number, and the terminal device 20 may be connected to one tactile feedback device 30. Hereinafter, details of each input device as an example of the tactile feedback device 30 will be described.

[0062] The controller 30A is an input device held by the user's hand. The controller 30A may have a plurality of buttons that the user can press. The user may perform a selection operation, a confirmation operation, a scroll operation, etc. on a virtual object displayed on the terminal device 20 by pressing a corresponding button of the controller 30A. In addition, the controller 30A may include a touch sensor and a motion sensor. Note that the controller 30A is not limited to being held by the user's hand, and may be worn on a part of the body such as an elbow, an arm, a knee, or an ankle.

[0063] The ring-shaped input device 30B is a ring-shaped input device to be worn on the user's finger. For example, the ring-shaped input device 30B is worn on the user's index finger. The ring-shaped input device 30B may include a touch sensor and a motion sensor. The user can perform an operation on a virtual object displayed on the terminal device 20 by a tap operation or a slide operation on the touch sensor or by moving the user's finger wearing the ring-shaped input device 30B.

[0064] In addition, the annular input device 30B may be a device including a pair of annular input devices that can be worn on fingers of both hands of the user. In this case, each of the pair of annular input devices may include a touch sensor and a motion sensor. In addition, each of the pair of annular input devices may communicate with the terminal device 20, or one of the pair of annular input devices may communicate with the terminal device 20 and send the communication content to the other annular input device.

[0065] Furthermore, the ring-shaped input device 30B can be used as an auxiliary interface of the information processing device 10. For example, a cursor displayed on the information processing device 10 can be controlled by movement of a finger of a user wearing the ring-shaped input device 30B.

[0066] The pen-shaped input device 30C is a pen-shaped input device to be held by the user's hand. The pen-shaped input device 30C may include a touch sensor and a motion sensor. The user may perform an operation on a virtual object displayed on the terminal device 20 by a tap operation or a slide operation on the touch sensor or by moving the pen-shaped input device 30C. For example, the user may be able to write characters or draw pictures on a virtual space displayed on the terminal device 20 by moving the pen-shaped input device 30C like writing or drawing.

[0067] In addition, the pen-shaped input device 30C can be used as an auxiliary interface of the information processing device 10, such as a pen-type tablet computer. For example, the user may be able to write characters or draw pictures on the screen displayed on the information processing device 10 by moving the pen-shaped input device 30C like writing or drawing.

[0068] The 6DoF mouse 30D is a mouse capable of detecting six degrees of freedom, including movement along the X-axis, Y-axis, and Z-axis and rotation around the X-axis, Y-axis, and Z-axis. For example, the 6DoF mouse 30D may sense information about the six degrees of freedom using an inertial measurement unit (IMU). A virtual object displayed on the terminal device 20 may be operated based on the sensing result from the 6DoF mouse 30D.

[0069] In addition, the 6DoF mouse 30D may include a plurality of buttons that can be pressed by the user. The user may perform a selection operation, a confirmation operation, a scroll operation, etc. on a virtual object displayed on the terminal device 20 by moving the 6DoF mouse 30D or pressing a corresponding button.

[0070] Furthermore, the 6DoF mouse 30D can be used as a mouse for operating a screen displayed on the information processing device 10. The 6DoF mouse 30D can switch an operation target between the screen displayed on the information processing device 10 and a virtual object displayed on the terminal device 20 in response to a user's action. For example, in a case where the 6DoF mouse 30D detects a planar movement of the 6DoF mouse 30D, the 6DoF mouse 30D can be used as a mouse for operating a screen displayed on the information processing device 10.

[0071] The details of an example of the tactile feedback device 30 have been described. The tactile feedback device 30 provides various notifications including tactile feedback to the user through vibration based on a signal transmitted from the information processing device 10. Note that vibration is an example of tactile feedback, and tactile feedback may be provided, for example, by thermal sensation (warmth, thermal stimulation), wind pressure, electrostatic force, force feedback, etc. other than vibration. The information processing device 10 generates a tactile signal corresponding to notification information indicating a notification to the user, and transmits the tactile signal to the tactile feedback device 30 via the terminal device 20.

[0072] In response to the operation performed by the tactile feedback device 30 on the object displayed on the screen of the information processing device 10, the information processing device 10 may generate a tactile signal, for example, as tactile feedback. In addition, in response to the operation performed by the tactile feedback device 30 on the virtual object displayed on the terminal device 20, a tactile signal may be generated on the HMD app as tactile feedback.

[0073] (Question review)

[0074] Here, the user can input an operation using the tactile feedback device 30 while checking various objects (display objects) displayed on the information processing device 10 or the terminal device 20 . Figure 2 is a diagram for describing an example of objects displayed on the information processing device 10 and the terminal device 20 .

[0075] In order to create a computer-aided design (CAD) model, the user inputs an operation on an object displayed on the information processing device 10 or the terminal device 20. For example, the user operates a 2D CAD model displayed on the information processing device 10, such as Figure 2 In addition, the user can operate the 3D CAD model displayed on the terminal device 20, such as Figure 2 In addition, in order to create a CAD model, the user can check a 2D object such as a blueprint or specification displayed on the terminal device 20, such as Figure 2 As shown in the right figure.

[0076] As described above, it is conceivable that the user frequently switches the object to be operated or checked between the object displayed on the information processing device 10 and the object displayed on the terminal device 20. Here, any object can be operated or checked using the same tactile feedback device 30. In such a case, it is assumed that the tactile feedback device 30 receives tactile signals corresponding to notification information output by a plurality of hosts as processing entities such as the information processing device 10 or the HMD app, and provides tactile feedback to the user.

[0077] In the case where there are multiple hosts as processing entities for outputting notification information, the user may not be able to recognize the multiple notification information provided by the tactile feedback device 30 as notifications output from different hosts. In particular, in the case where the multiple hosts generate the same or similar tactile signals, it is difficult for the user to recognize the difference in vibration provided by the tactile feedback device 30 based on the tactile signals. Therefore, the user may mistakenly recognize the notification information output from a certain host as a notification output from a different host.

[0078] Therefore, the information processing device 10 according to the present embodiment identifies the host type and controls the conversion processing of the tactile signal based on the host type. For example, the information processing device 10 identifies the host type, that is, whether the host is the information processing device 10 or the HMD app. Then, the information processing device 10 sends the tactile signal to the tactile feedback device 30 without performing any conversion on the tactile signal with the information processing device 10 as the host. On the other hand, the information processing device 10 performs conversion processing on the tactile signal with the HMD app as the host, and sends the resulting signal to the tactile feedback device 30.

[0079] The tactile feedback device 30 provides tactile feedback using the signal transmitted through the above process, so as to allow the user to recognize multiple notifications as tactile feedback from different hosts.

[0080] The overview of the information processing system according to the embodiment of the present disclosure has been described above. Next, a specific configuration of the information processing device 10 included in the information processing system according to the present embodiment will be described with reference to the drawings.

[0081] <<2. Configuration Example of Information Processing Device 10>>

[0082] Figure 3 1 is a block diagram showing an example of the configuration of the information processing device 10 according to the present embodiment. Figure 3 As shown in , the information processing device 10 includes a communication unit 110 , a control unit 120 , a storage unit 130 , a display unit 140 , and an operation input unit 150 .

[0083] (Communication unit 110)

[0084] The communication unit 110 establishes a communication connection with an external device and sends and receives data. For example, the communication unit 110 sends data to the terminal device 20 and receives data from the terminal device 20. The communication unit 110 can establish a communication connection with the terminal device 20 or the like through, for example, a wired or wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0085] For example, the communication unit 110 receives a signal indicating a motion of a user detected by the tactile feedback device 30 from the tactile feedback device 30 via the terminal device 20. In addition, the communication unit 110 transmits a tactile signal generated by a generation unit 121 or an app execution unit 122 to be described later and a signal obtained by converting the tactile signal by a conversion unit 124 to be described later to the tactile feedback device 30 via the terminal device 20.

[0086] (Control Unit 120)

[0087] The control unit 120 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing device 10 according to various programs. For example, the control unit 120 executes a 2D app (a 2D application to be executed by the information processing device 10) such as a 2D CAD tool or a browser, and a user can operate a 2D CAD model using the 2D app. The control unit 120 executes the 2D app to control the screen display on the display unit 140. For example, the app execution unit 122 displays the UI of the 2D app on the display unit 140 to be described later. In addition, the control unit 120 can also display the UI of the 2D app on the terminal device 20. In addition, the display screen and audio generated by the 2D app can be output to the image display unit, the audio output unit, etc. included in the terminal device 20.

[0088] The control unit 120 is implemented by, for example, an electronic circuit such as a central processing unit (CPU) or a microprocessor. In addition, the control unit 120 may include a read-only memory (ROM) that stores programs to be used, operating parameters, etc., and a random access memory (RAM) that temporarily stores parameters that are changed as needed, etc.

[0089] The control unit 120 according to the present embodiment may also function as the generation unit 121 , the app execution unit 122 , the determination unit 123 , and the conversion unit 124 .

[0090] (Generation Unit 121)

[0091] The generating unit 121 generates a tactile signal. The generating unit 121 generates the tactile signal, for example, based on a signal indicating a user's action detected by the tactile feedback device 30, which is received by the communication unit 110. More specifically, the generating unit 121 generates the tactile signal based on a signal indicating a user's action on an object displayed on the display unit 140, which is detected by the tactile feedback device 30. Such a tactile signal is generated to provide tactile feedback to the user. The tactile feedback provided to the user is an example of notification information.

[0092] In addition, the generating unit 121 may process the 2D app. For example, the generating unit 121 may generate notification information about the 2D app, such as haptic feedback, in response to an operation performed by a user on the 2D app.

[0093] In addition, the generation unit 121 may generate a tactile signal in response to a message reception from an external device via the communication unit 110 or a battery charge reduction of the information processing device 10, etc. Such a tactile signal is generated to notify the user of message reception, battery charge of the information processing device 10, etc. Notification to the user is an example of notification information. Details of an example of a tactile signal generated by the generation unit 121 will be described later.

[0094] Note that the tactile signal may include multiple parameter values ​​related to the vibration. The multiple parameters include amplitude, frequency, output period or waveform (rectangular wave, sine wave, etc.). Upon receiving the tactile signal including multiple parameter values, the tactile feedback device 30 outputs a vibration with multiple parameters to provide tactile feedback to the user.

[0095] The generation unit 121 may generate a tactile signal based on a tactile signal pattern stored in a storage unit 130 to be described later. More specifically, the generation unit 121 may select a tactile signal pattern suitable for a signal indicating an action of the user from the tactile signal patterns stored in the storage unit 130 to generate a tactile signal indicated by the pattern. In addition, the tactile signal pattern may be stored in a storage unit included in the terminal device 20 or a storage unit included in the tactile feedback device 30. In this case, the communication unit 110 receives the tactile signal pattern from the terminal device 20. Then, the generation unit 121 generates a tactile signal based on the tactile signal pattern received by the communication unit 110.

[0096] (APP execution unit 122)

[0097] The app execution unit 122 executes an HMD app (HMD application executed by the information processing device 10 ) that provides the user with an experience in a virtual space and enables the user to edit a virtual object.

[0098] For example, the app execution unit 122 executes an HMD app that provides the user with an experience in a virtual space and enables the user to edit a virtual object. The app execution unit 122 executes the HMD app to control outputs such as screen display on and audio output from the terminal device 20. In addition, the app execution unit 122 constructs a virtual space to be displayed on the terminal device 20. The HMD app is an application that provides the user with an experience in a virtual space by using information received from the terminal device 20 by the communication unit 110 and information received from the tactile feedback device 30 by the communication unit 110 via the terminal device 20. Specifically, the HMD app may be a 3D CAD application capable of operating a 3D CAD model or the like. In the case where the HMD app is a 3D CAD application, the user can edit the 3D CAD object, i.e., the virtual object displayed on the terminal device 20, by interacting with the 3D CAD object using the tactile feedback device 30. In the case where the terminal device 20 is an HMD capable of providing VR, a virtual space including a 3D CAD object is displayed on the terminal device 20. On the other hand, in the case where the terminal device 20 is a video see-through HMD or see-through AR HMD capable of displaying AR, the terminal device 20 does not necessarily need to display a virtual space, but may display virtual objects on the image display unit such that the virtual objects are located in a real space.

[0099] Screen display, audio output, and the like for the HMD app do not necessarily need to be performed by the terminal device 20, and screen display on the display unit 140 of the information processing device 10 or audio output from the audio output unit of the information processing device 10 may be performed. In this case, it is not necessary to display the virtual object as if the virtual object exists in the virtual space or the real space displayed by the terminal device 20, but the virtual object may be displayed in 2D on the display unit 140 of the information processing device 10. For example, in the case where the information processing device 10 is a PC, the virtual object may be displayed in 2D that can be perceived three-dimensionally on the display as the display unit 140. The user can edit the virtual object constructed by the HMD app based on the operation of the PC by operating the virtual object displayed on the display using the operation input unit 150 to be described later such as a mouse.

[0100] In addition, the app execution unit 122 generates a tactile signal related to the HMD app. For example, the app execution unit 122 generates a tactile signal based on a signal indicating a user's action detected by the tactile feedback device 30. More specifically, the app execution unit 122 generates a tactile signal based on a signal indicating a user's action on a virtual object displayed on the terminal device 20 and detected by the tactile feedback device 30. Such a tactile signal is generated to provide tactile feedback to the user. The tactile feedback provided to the user is an example of notification information.

[0101] In addition, the app execution unit 122 may generate a tactile signal based on a decrease in the battery level of the terminal device 20, etc. Such a tactile signal is generated to provide the user with various notifications such as message reception or the battery level of the information processing device 10. Various notifications to the user are examples of notification information. Details of examples of tactile signals generated by the app execution unit 122 will be described later.

[0102] The app execution unit 122 may generate a tactile signal based on a tactile signal pattern stored in a storage unit 130 to be described later. More specifically, the app execution unit 122 may select a tactile signal pattern suitable for a signal indicating an action of the user from the tactile signal patterns stored in the storage unit 130 to generate a tactile signal indicated by the pattern. In addition, the tactile signal pattern may be stored in a storage unit included in the terminal device 20 or a storage unit included in the tactile feedback device 30. In this case, the communication unit 110 receives the tactile signal pattern from the terminal device 20. Then, the app execution unit 122 generates a tactile signal based on the tactile signal pattern received by the communication unit 110.

[0103] (Determination unit 123)

[0104] The determination unit 123 identifies the host as the source of outputting notification information to the user to obtain the host type as a determination result. For example, the determination unit 123 may identify the host based on whether the tactile signal is generated by the generation unit 121 or the app execution unit 122, and use the identification result as the host type determination result. More specifically, in the case where the generation unit 121 generates the tactile signal, the determination unit 123 may identify the information processing device 10 as the host. In addition, in the case where the tactile signal is generated by the app execution unit 122, the determination unit 123 may identify the HMD app or HMD as the host.

[0105] In addition, the determination unit 123 may identify the host based on whether the application that has generated the notification information is an HMD app executed by the PC or a 2D app executed by the PC. For example, in the case where the application that has generated the notification information is an HMD app, the determination unit 123 may identify the HMD app or the HMD as the host. In addition, in the case where the application that has generated the notification information is a 2D app executed by the PC, the determination unit 123 may identify the PC as the host.

[0106] In addition, the determination unit 123 can determine whether the host is primary as the host type based on the host priority. For example, the determination unit 123 can determine the priority of each of the multiple hosts and identify the host type with the highest priority as the primary host. In addition, the determination unit 123 can identify the host types other than the host with the highest priority as sub-hosts.

[0107] Note that the determining unit 123 does not need to determine the priorities of all multiple hosts, but can, for example, only determine the host with the highest priority. In this case, the determining unit 123 identifies the host type with the highest priority as the main host, and identifies other host types as sub-hosts.

[0108] Furthermore, after determining the priorities of all the plurality of hosts, the determination unit 123 may sequentially identify the host having the second highest priority as sub-host 1 , and the host having the third highest priority as sub-host 2 .

[0109] The determination unit 123 may determine the priorities of the plurality of hosts so that the higher the frequency of use of the host by the user, the higher the priority assigned to the host. In addition, the determination unit 123 may determine the priorities of the plurality of hosts so that the higher the performance of the host, the higher the priority assigned to the host. Note that the performance may be evaluated based on values ​​such as resolution or processing speed, but the evaluation method is not limited to any particular method.

[0110] In addition, the determination unit 123 may dynamically identify the host type. For example, the determination unit 123 may determine the host priority based on the execution location of the application last used by the user.

[0111] For example, in the case where the app execution unit 122 executes the application that the user used last time, the determination unit 123 may identify the HMD or the HMD app as the host with the highest priority, that is, as the primary host. That is, in the case where it is determined based on the user's operation on the display of the HMD that the user is using the HMD application executed by the app execution unit 122, the determination unit 123 may identify the HMD or the HMD app as the host with the highest priority, that is, as the primary host. In addition, in the case where the application that the user used last time is executed by a unit other than the app execution unit 122, the determination unit 123 may identify the information processing device 10 as the primary host.

[0112] Furthermore, when it is determined based on the user's operation on the display of the display unit 140 that the user is using a 2D app, that is, when the control unit 120 is executing a 2D app, the determination unit 123 may identify the information processing device 10 as the host.

[0113] In addition, in the case where the tactile feedback device 30 is not a device for operating an object displayed on the display unit 140 but a device for operating a virtual object displayed on the terminal device 20, the determination unit 123 may determine the host priority based on the device last operated by the user. For example, in the case where the device last operated by the user is the tactile feedback device 30, the determination unit 123 may identify the HMD app as the host with the highest priority, that is, as the primary host. For example, in the case where the device last operated by the user is the operation input unit 150, the determination unit 123 may identify the information processing device 10 as the host with the highest priority, that is, as the primary host.

[0114] In addition, in the case where the user uses the 6DoF mouse 30D to operate the object displayed on the display unit 140 and the virtual object displayed on the terminal device 20, the determination unit 123 can determine the host priority based on the use state of the mouse detected by the 6DoF mouse 30D. For example, in the case where the use state indicating the planar movement of the 6DoF mouse 30D is detected, the determination unit 123 can identify the information processing device 10 as the host with the highest priority, that is, as the primary host. In addition, in the case where the use state of the 6DoF mouse 30D using the 6DoF function is detected, the determination unit 123 can identify the HMD app as the host with the highest priority, that is, as the primary host.

[0115] In addition, when the user wears the terminal device 20, the determination unit 123 can identify the HMDapp as the host with the highest priority, that is, as the primary host. Note that the terminal device 20 uses a sensor to detect whether the user wears the terminal device 20, and sends the detection result to the information processing device 10. The determination unit 123 determines the priority based on the detection result.

[0116] Here, we will refer to Figures 4 to 7 An example of host type identification performed by the determination unit 123 is described. Here, an example in which the determination unit 123 dynamically identifies a host type based on the host priority will be described. Figures 4 to 7 is an explanatory diagram for describing an example of host type identification performed by the determination unit 123 .

[0117] exist Figure 4 In the example, the user U wears the terminal device 20 and uses the controller 30A to operate a virtual object displayed on the terminal device 20 through the HMD app. At this time, the controller 30A sends a signal indicating the action of the user U to the information processing device 10 via the terminal device 20. When the communication unit 110 of the information processing device 10 receives the signal, the app execution unit 122 generates a tactile signal as feedback based on the signal.

[0118] At this time, the determination unit 123 identifies the HMD app as the host. In addition, since the user U wears the terminal device 20, the determination unit 123 can identify the HMD app as the host with the highest priority, that is, as the primary host.

[0119] On the other hand, Figure 5 , the user U operates the screen displayed on the information processing device 10 using the 6DoF mouse 30D. At this time, the 6DoF mouse 30D transmits a signal indicating the operation input of the user U to the information processing device 10 via the terminal device 20. When the communication unit 110 receives the signal, the generation unit 121 generates a tactile signal based on the signal.

[0120] At this time, the determination unit 123 identifies the information processing device 10 as the host. In addition, since the application last executed by the user U was executed by the control unit 120 other than the app execution unit 122, the determination unit 123 identifies the information processing device 10 as the host with the highest priority, that is, as the primary host.

[0121] Note that even when the user wears the terminal device 20, when the terminal device 20 displays the screen displayed by the information processing device 10, the determination unit 123 can also identify the information processing device 10 as the host with the highest priority, that is, as the main host.

[0122] exist Figure 6 , the user U uses the controller 30A to operate the virtual desktop screen VS of the information processing device 10 displayed on the terminal device 20. At this time, the determination unit 123 obtains information indicating that the virtual desktop screen VS of the information processing device 10 is displayed on the terminal device 20 from the app execution unit 122. Then, in the case where a tactile signal is generated as feedback on the operation performed on the object displayed on the virtual desktop screen VS, the determination unit 123 identifies the information processing device 10 as the host. As described above, since the virtual desktop screen VS of the information processing device 10 is displayed on the terminal device 20, the priority of the information processing device 10 becomes the highest, and the determination unit 123 identifies the information processing device 10 as the host as the primary host.

[0123] Note that when the terminal device 20 displays a virtual object together with the virtual desktop screen VS, the determination unit 123 may set the priority of the HMD app to the highest and identify the HMD app as the main host. For example, for a tactile signal indicating feedback on an operation performed by the user U on the virtual desktop screen VS, the determination unit 123 identifies the information processing device 10 as the host. Then, the determination unit 123 identifies the host type of the host as a sub-host based on the priority of the information processing device 10. In addition, for a tactile signal indicating feedback on an operation performed by the user U on the virtual object, the determination unit 123 identifies the HMD app as the host. Then, the determination unit 123 identifies the host type of the host as the main host based on the priority of the HMD app.

[0124] In addition, when the user pushes the image display unit of the terminal device 20 to a position that does not block the field of view of the user U while fixing the terminal device 20 on the head (turning the image display unit upward), the determination unit 123 can set the priority of the information processing device 10 to the highest, and identify the information processing device 10 as the main host, and identify the terminal device 20 as the sub-host. Note that the terminal device 20 uses a sensor to detect whether the user has turned the terminal device 20 upward, and sends the detection result to the information processing device 10. The determination unit 123 obtains the detection result received through the communication unit 110, and determines the priority based on the detection result.

[0125] Figure 7 The user U shown in FIG. 1 flips up the image display unit 200 of the terminal device 20, views the screen displayed on the information processing device 10, and uses the 6DoF mouse 30D to operate the screen displayed on the information processing device 10. For the tactile signal indicating feedback for the screen operation, the determination unit 123 identifies the information processing device 10 as the host. As described above, since the user U flips up the image display unit 200, the priority of the information processing device 10 becomes the highest, and the determination unit 123 identifies the information processing device 10 as the host as the primary host.

[0126] A specific example of a method for identifying a host type by the determination unit 123 has been described above. The determination unit 123 may determine whether the tactile feedback device 30 is in an environment where the tactile feedback device 30 can obtain notification information from multiple hosts based on information received by the communication unit 110 from the tactile feedback device 30 via the terminal device 20. In a case where the determination unit 123 determines that the tactile feedback device 30 cannot obtain notification information from multiple hosts, that is, in a case where the tactile feedback device 30 is in an environment where the tactile feedback device 30 obtains notification information from only one host and provides notification information, the determination unit 123 may identify the host type of the one host as a primary host.

[0127] (Conversion unit 124)

[0128] The conversion unit 124 controls a conversion process of the haptic signal generated by the generation unit 121 or the app execution unit 122 based on the host type identified by the determination unit 123 .

[0129] The conversion unit 124 may control a conversion process of a haptic signal corresponding to each notification information received from a plurality of hosts at different timings or at timings at least partially overlapped in time, based on the host type of each host.

[0130] As an example, the conversion unit 124 may control the conversion process of the haptic signal based on the determination result from the determination unit 123 as to whether the host is the information processing device 10 or the HMD app.

[0131] As another example, the conversion unit 124 may control the conversion process of the haptic signal based on the host type according to the priority identified by the determination unit 123. In this embodiment, an example in which the conversion unit 124 controls the conversion process of the haptic signal based on the host type according to the priority identified by the determination unit 123 will be mainly described.

[0132] The conversion unit 124 can perform various types of conversion processing on the tactile signal. For example, the conversion unit 124 can convert the frequency, amplitude, output period, or waveform included in the tactile signal. In addition, the conversion unit 124 does not need to perform conversion processing on the tactile signal corresponding to the notification information received from at least one of the multiple hosts. For example, the conversion unit 124 does not need to convert the tactile signal output from the host whose host type is the main host. Therefore, the user is able to identify each of the multiple notifications as tactile feedback from a different host without compromising the ease with which the user can identify the content of the notification from the host with a higher priority.

[0133] - Example of conversion process

[0134] Here, we will refer to Figures 8 to 20 An example of conversion processing performed by the conversion unit 124 is described. Figure 8 is an explanatory diagram showing an example in which the conversion unit 124 converts the frequency of the haptic signal. Figure 8 An example is shown in which the conversion unit 124 converts the frequency of a haptic signal output from a master whose master type is a sub-master by a factor of 2. Note that when converting the frequency, the conversion unit 124 may convert the output period in addition to the frequency to maintain the number of waves included in the haptic signal. Figure 8 An example is shown in which the conversion unit 124 converts the output period into half.

[0135] More specifically, the conversion unit 124 converts a haptic signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 output from a host whose host type is a sub-host into a haptic signal with a frequency of 200 Hz, an output period of 5 ms, and an amplitude of 1.0.

[0136] Fig. 9 is an explanatory diagram showing an example in which the conversion unit 124 converts the amplitude of the haptic signal. Fig. 9 An example is shown in which the conversion unit 124 converts the amplitude of a tactile signal output from a host whose host type is a sub-host into half. More specifically, the conversion unit 124 converts a tactile signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 output from a host whose host type is a sub-host into a tactile signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 0.5.

[0137] Fig.10 is an explanatory diagram showing an example in which the conversion unit 124 converts the waveform of the haptic signal. Fig.10 An example is shown in which the conversion unit 124 converts the waveform of a tactile signal output from a host whose host type is a sub-host from a sine wave to a rectangular wave. More specifically, the conversion unit 124 converts a tactile signal with a frequency of 100 Hz, an output period of 10 ms, an amplitude of 1.0, and a sine wave waveform output from a host whose host type is a sub-host into a tactile signal with a frequency of 100 Hz, an output period of 10 ms, an amplitude of 1.0, and a rectangular wave waveform.

[0138] Fig.11 is an explanatory diagram showing an example in which the conversion unit 124 converts a haptic signal into a fade-out signal. Fig.11 An example is shown in which the conversion unit 124 converts a tactile signal output from a host whose host type is a sub-host into a fade-out signal. More specifically, the conversion unit 124 converts a tactile signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 output from a host whose host type is a sub-host into a fade-out signal with a frequency of 100 Hz, an output period of 30 ms, and a gradually decreasing amplitude. For example, the conversion unit 124 converts the tactile signal into a signal with an output period between 0 ms and 10 ms and an amplitude of 1.0, into a signal with an output period between 10 ms and 20 ms and an amplitude of 0.75, and into a signal with an output period between 20 ms and 30 ms and an amplitude of 0.5.

[0139] In addition, the conversion unit 124 may convert the amplitude included in the haptic signal into zero. Fig.12 is an explanatory diagram showing an example in which the conversion unit 124 converts the amplitude included in the haptic signal into 0. Fig.12An example is shown in which the conversion unit 124 converts the amplitude of a haptic signal output from a master whose master type is a sub-master into 0.

[0140] More specifically, the conversion unit 124 converts a tactile signal having a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 output from a host whose host type is a sub-host into a tactile signal having a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 0. Note that at this time, even if the signal obtained by the conversion performed by the conversion unit 124 is transmitted to the tactile feedback device 30 through the communication unit 110, the tactile feedback device 30 does not provide tactile feedback. When the conversion unit 124 performs such a conversion process, only notifications from a host having a higher priority are provided to the user, thereby reducing the possibility that the user mistakenly recognizes the output destination of the notification information indicating the notification.

[0141] In the case where a host whose host type is a main host and a host whose host type is a sub host output the same haptic signal simultaneously or substantially simultaneously, the conversion unit 124 may convert the amplitude of the haptic signal output from the host whose host type is a sub host into zero.

[0142] In particular, in the case where there are various types of haptic signals that a host whose host type is a main host can output, the conversion unit 124 may convert the amplitude of a haptic signal output from a host whose host type is a sub host into 0. Therefore, confusion between haptic feedback using a haptic signal output from a host whose host type is a main host and haptic feedback using a haptic signal output from a host whose host type is a sub host or a signal obtained by converting the haptic signal may be prevented.

[0143] In addition, the conversion unit 124 may convert the tactile signal into a predetermined signal. In particular, when the frequency band that the tactile feedback device 30 can output is a narrow band, the conversion unit 124 may convert the tactile signal into a predetermined signal.

[0144] Fig.13 is an explanatory diagram showing an example in which the conversion unit 124 converts a tactile signal into a predetermined signal. Fig.13 An example is shown in which the conversion unit 124 converts a tactile signal output from a host whose host type is a sub-host into a predetermined signal. More specifically, the conversion unit 124 converts a tactile signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 output from a host whose host type is a sub-host into a tactile signal with a frequency of 100 Hz, an output period of 20 ms, and an amplitude of 0.25, i.e., a predetermined signal.

[0145] At this time, the predetermined signal may be a signal different from each tactile signal that can be output by the host whose host type is the main host. In addition, the conversion unit 124 may perform control to display the conversion from the tactile signal output by the host whose host type is the sub-host to the predetermined signal on the screen output by the host control as the main host. For example, the conversion unit 124 may perform control to display "Provide vibration notification from different hosts in vibration mode 1" on the screen.

[0146] In addition, the conversion unit 124 can convert the tactile signal by adding a buffer time before the tactile signal. In the case where multiple hosts send tactile signals at the same time, the conversion unit 124 can add a buffer time before the tactile signal output from the host whose host type is a sub-host to ensure that the tactile feedback provided by the tactile feedback device 30 using the multiple tactile signals does not overlap in timing.

[0147] The buffer time may be an output period of a tactile signal output from a host whose host type is a primary host, or may be a time obtained by adding a certain time to the output period. When the certain time is added to the output period, tactile feedback using a plurality of tactile signals is distinguished, allowing a user to easily recognize that a plurality of types of tactile feedback are different from each other.

[0148] Fig.14 is an explanatory diagram showing an example in which the conversion unit 124 adds a buffer time before a haptic signal. Fig.14 An example is shown in which the conversion unit 124 adds a buffer time before a haptic signal output from a host whose host type is a sub-host to ensure that the haptic signal output from the host whose host type is a sub-host and the haptic feedback do not overlap.

[0149] More specifically, the conversion unit 124 converts a tactile signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 output from a host whose host type is a subhost into a tactile signal with a frequency of 100 Hz and an output period of 20 ms, into a tactile signal with an output period between 0 ms and 10 ms and an amplitude of 0, and into a tactile signal with an output period between 10 ms and 20 ms and an amplitude of 1.

[0150] Furthermore, the conversion unit 124 may change an output destination of the haptic signal. Fig.15 is an explanatory diagram showing an example in which the conversion unit 124 changes the output destination of the haptic signal. Fig.15An example of a case where the tactile signal before conversion is output to both of the pair of annular input devices 30B is shown. An example of a case where the conversion unit 124 changes the output destination of the tactile signal output from the master whose host type is the main master to the annular input device worn on the right hand of the pair of annular input devices 30B is shown. On the other hand, the conversion unit 124 changes the output destination of the ae tactile signal output from the master whose host type is the sub-master to the annular input device worn on the left hand of the pair of annular input devices 30B.

[0151] Furthermore, the conversion unit 124 may perform different conversion processing on each haptic signal output from the plurality of hosts based on a varying combination of the above-described conversion processing. Fig.16 is an explanatory diagram showing an example in which the conversion unit 124 converts the frequency or amplitude of a haptic signal. Fig.16 An example is shown in which the conversion unit 124 converts the amplitude of the haptic signal output from the host whose host type is sub-host 1 into half and converts the frequency of the haptic signal output from the host whose host type is sub-host 2 into twice.

[0152] More specifically, the conversion unit 124 converts a haptic signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 output from a host whose host type is sub-host 1 into a haptic signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 0.5.

[0153] Furthermore, the conversion unit 124 converts the tactile signal with a frequency of 100 Hz, an output period of 10 ms, and an amplitude of 1.0 outputted from the host of the host type sub-host 2 into a tactile signal with a frequency of 200 Hz, an output period of 5 ms, and an amplitude of 1.0. Note that the conversion unit 124 is shown here in a manner similar to that of reference 1. Figure 8 The manner of describing the case is an example in which not only the frequency but also the output period is converted.

[0154] Furthermore, the conversion unit 124 may convert haptic signals from a plurality of hosts into haptic signals having non-overlapping frequency bands. Fig.17 is an explanatory diagram showing an example of frequency band conversion in a case where the conversion unit 124 converts haptic signals from a plurality of hosts into haptic signals having non-overlapping frequency bands.

[0155] Fig.17 An example is shown in which a frequency band of a haptic signal that a host whose host type is a main host can output and a frequency band of a haptic signal that a host whose host type is a sub host can output are consistent and each of the frequency bands is converted.

[0156] More specifically, the frequency band of the tactile signal that can be output by the host as the main host and the frequency band of the tactile signal that can be output by the host whose host type is the sub-host are within the range of 150 Hz to 250 Hz. The conversion unit 124 converts the frequency of the tactile signal output from the host as the main host to ensure that the frequency band of the tactile signal falls within the range of 150 Hz to 180 Hz. The conversion unit 124 converts the frequency of the tactile signal output from the host as the sub-host to ensure that the frequency band of the tactile signal falls within the range of 220 Hz to 250 Hz.

[0157] Furthermore, conversion unit 124 may convert haptic signals from a plurality of hosts into haptic signals having non-overlapping amplitudes. Fig.18 is an explanatory diagram showing an example of amplitude conversion in a case where the conversion unit 124 converts haptic signals from a plurality of hosts into haptic signals having non-overlapping amplitudes.

[0158] Fig.18 An example is shown in which the amplitude of a haptic signal that a host whose host type is a main host can output is consistent with the amplitude of a haptic signal that a host whose host type is a sub host can output, and each of the amplitudes is converted.

[0159] More specifically, the amplitude of the tactile signal that can be output by a host whose host type is a main host and the amplitude of the tactile signal that can be output by a host whose host type is a sub-host are within the range of 0 to 1.0. The conversion unit 124 converts the frequency of the tactile signal output from the host as the main host to ensure that the amplitude of the tactile signal falls within the range of 0.6 to 1.0. Note that the conversion unit 124 does not convert the amplitude of the tactile signal with an amplitude of 0. In addition, the conversion unit 124 converts the amplitude of the tactile signal output from the host as the sub-host to ensure that the amplitude of the tactile signal falls within the range of 0 to 0.4.

[0160] In addition, when the same tactile signal is output from the same host within a certain period of time, the conversion unit 124 can convert the amplitude of the second tactile signal and the subsequent tactile signal. For example, when the same tactile signal is output from a host whose host type is a sub-host within a certain period of time (e.g., n seconds), the conversion unit 124 can convert the amplitude of the tactile signal to ensure that the amplitude of the second tactile signal and the subsequent tactile signal gradually decreases.

[0161] Fig.19 is an explanatory diagram showing an example in which, when the same haptic signal is output from the same host within a certain period of time, the conversion unit 124 converts the amplitudes of the second haptic signal and the subsequent haptic signals. Fig.19An example is shown in which, in a case where the same haptic signal is output from a host whose host type is a sub-host within n seconds, the amplitude of the haptic signal is converted to ensure that the amplitude of the second haptic signal and subsequent haptic signals gradually decreases.

[0162] Furthermore, in the case where haptic signals from a plurality of hosts overlap, the conversion unit 124 may change the output timing of the haptic signals. Fig. 20 1 is an explanatory diagram showing an example in which the conversion unit 124 changes the output timing of the tactile signal. Fig. 20 As shown in , the conversion unit 124 may convert the output timing of the haptic signal output from the master whose host type is the slave master to ensure that the haptic signal is output n seconds after the haptic signal is output from the master whose host type is the main master.

[0163] Although an example of the conversion process performed by the conversion unit 124 has been described above, the conversion process performed by the conversion unit 124 is not limited to such an example. For example, the conversion unit 124 may perform a conversion process obtained by combining the above-mentioned conversion methods. In addition, the conversion unit 124 may perform the conversion process under different methods. For example, the conversion unit 124 may convert the tactile signal into a signal obtained by adding a predetermined signal before the tactile signal.

[0164] In addition, the conversion unit 124 can convert the signal obtained by converting the tactile signal output from the host into a tactile signal having a mode different from any mode of the tactile signal from other hosts based on the mode of the tactile signal that each of the multiple hosts can output. For example, the conversion unit 124 can select a conversion processing method and convert the tactile signal to ensure that the signal obtained by converting the tactile signal output from the host is converted into a tactile signal having a mode different from any mode of the tactile signal from other hosts. Note that in this case, the storage unit 130, which will be described later, stores the mode of the tactile signal that each of the multiple hosts can output.

[0165] Furthermore, an example has been mainly described in which the conversion unit 124 converts only a haptic signal output from a host whose host type is a sub-host, but the conversion unit 124 may convert a haptic signal output from a host whose host type is a main host.

[0166] In addition, the conversion unit 124 may convert the tactile signal into a signal other than the tactile signal, such as sound or light. For example, the conversion unit 124 may convert the tactile signal output from the host whose host type is a sub-host into an audio signal. The audio signal may be a signal having the same waveform as the tactile signal, or may be a predetermined audio signal.

[0167] In particular, in the case where there are various types of tactile signals that can be output by a host whose host type is a main host, the conversion unit 124 can convert a tactile signal output from a host whose host type is a sub-host into an audio signal. Therefore, confusion between tactile feedback using a tactile signal output from a host whose host type is a main host and tactile feedback using a tactile signal output from a host whose host type is a sub-host or a signal obtained by converting a tactile signal can be prevented.

[0168] (Storage unit 130)

[0169] The storage unit 130 is realized by a ROM that stores programs, operation parameters, and the like to be used for the processing of the control unit 120 , and a RAM that temporarily stores parameters and the like that are changed as needed.

[0170] (Display unit 140)

[0171] The display unit 140 has a function of displaying various screens such as an operation screen. The display unit 140 may be implemented by, for example, a liquid crystal display (LCD) device, an organic light emitting diode (OLED) device, etc. In addition, the display unit 140 may be a so-called optical see-through display having optical transparency, or may be a projector that projects an image into a real space.

[0172] (Operation Input Unit 150)

[0173] The operation input unit 150 receives an operation from a user and outputs input information to the control unit 220. For example, the operation input unit 150 is realized by various input devices such as a touch panel, a button, a switch, a mouse, and a keyboard.

[0174] The specific configuration of the information processing device 10 has been described above. Note that the configuration of the information processing device 10 according to the present embodiment is not limited to Figure 3 For example, the information processing device 10 may be implemented by a plurality of devices. In addition, at least a part of the functions of the control unit 120 may be implemented by a terminal device 20 or a server (not shown) on the network.

[0175] <<3. Specific examples of tactile feedback>>

[0176] Next, we will refer to Figure 21 to Figure 23 A specific example of tactile feedback implemented by the information processing device 10 according to the present embodiment is described.

[0177] Fig.21 : is a table showing the correspondence between the notification content indicated by the haptic signal and the signal transmitted from the information processing device 10 to the haptic feedback device 30 in a case where the host type is the main host or the sub-host.

[0178] Fig.21 The following example is shown, in which, when the host type of the host is a main host, the conversion unit 124 does not convert the tactile signal, and when the host type of the host is a sub-host, the wavelength of the tactile signal is converted by the conversion unit 124 and sent to the tactile feedback device 30. That is, Fig.21 The signal shown in FIG. 1 in the case where the host type is the primary host is a haptic signal generated by the generating unit 121 or the app executing unit 122. In addition, Fig.21 The signal shown in FIG. 1 in the case where the host type is a sub-host is a signal obtained by conversion performed by the conversion unit 124. Note that for a case where it is not desired to output a tactile signal, “—” is shown, but a tactile signal may be output, converted by the conversion unit 124, and sent to the tactile feedback device 30.

[0179] Fig.21 Examples are shown of notification content indicated by a tactile signal, a mouse hover on an object displayed on the display unit 140 or a user interface (UI) included in a virtual object displayed on the terminal device 20, a UI confirmation operation (clicking on the UI), an error indicating that the user's action is incorrect, receiving a message from the outside, a reduction in battery power, and switching of an operation target by the tactile feedback device 30 (controller).

[0180] The notification of battery power reduction relates to the battery power of the information processing device 10, the terminal device 20, or the tactile feedback device 30. Notification information indicating the notification of battery power reduction of the information processing device 10 is generated by the generation unit 121. Notification information indicating the notification of battery power reduction of the terminal device 20 is generated by the app execution unit 122.

[0181] Notification information indicating a notification of a reduction in battery power of the tactile feedback device 30 is generated by the generation unit 121 or the app execution unit 122. For example, in a case where an object displayed on the display unit 140 of the information processing device 10 is operated by the tactile feedback device 30, notification information indicating a notification of a reduction in battery power of the tactile feedback device 30 may be generated by the generation unit 121. On the other hand, in a case where a virtual object displayed on the terminal device 20 is operated by the tactile feedback device 30, notification information indicating a notification of a reduction in battery power of the tactile feedback device 30 may be generated by the app execution unit 122.

[0182] Here, an example of a case where a user hovers a mouse over a UI included in an object displayed on the display unit 140 will be described. In this case, the generation unit 121 outputs a sine wave with a wave number of 1 as a tactile signal. Since the determination unit 123 identifies the information processing device 10 as a host and identifies the information processing device 10 as a primary host, the tactile signal is sent to the tactile feedback device 30 without being converted by the conversion unit 124.

[0183] Note that here, an example is shown in which, in the case where the user operates the UI, a signal with an amplitude greater than 0 is sent to the tactile feedback device 30 regardless of the host type of the host; however, in the case where the host is the information processing device 10, a signal with an amplitude of 0 may be sent. That is, in the case where the host is the information processing device 10, there is no need to provide tactile feedback to the user.

[0184] More specifically, as feedback for operating the UI displayed on the information processing device 10, the tactile signal generated by the generation unit 121 can be converted into a signal with an amplitude of 0 by the conversion unit 124. This is because: in the case where the user operates the UI displayed by the information processing device 10, the user can easily recognize the UI to be operated without tactile feedback, compared with the case where the user operates the UI displayed by the terminal device 20.

[0185] Note that the notification content is not limited to Fig.21 The notification content shown in the figure can also notify events such as drag operations on the UI, collisions between virtual objects and a cursor, movement of virtual objects, changes in the size of virtual objects, changes in the angle of virtual objects, interference between virtual objects, or notifications about software updates.

[0186] For example, in the case where the user operates the tactile feedback device 30 to change the size of the virtual object, the app execution unit 122 can generate a tactile signal to ensure that feedback is provided when the zoom level becomes an integer. In addition, in the case where the user cannot visually recognize that the virtual objects collide with each other, the app execution unit 122 can generate a tactile signal indicating interference between the virtual objects.

[0187] Note that in the case where a user operates an object displayed on the display unit 140 and performs three-dimensional operations by a circular input device, a 6DoF mouse, etc., the generation unit 121 can generate notification information about the object, similar to the notification information indicating a notification about a virtual object generated by the app execution unit 122.

[0188] Next, we will refer to Fig. 22 Another example of haptic feedback is described. Fig. 22is an explanatory diagram for describing an example of tactile feedback in a case where a virtual desktop screen and a virtual object displayed by the information processing device 10 are displayed on the terminal device 20 .

[0189] The user can operate both the virtual desktop screen VS and the virtual object O of the information processing device 10 displayed on the terminal device 20 using the controller 30A.

[0190] For example, for a tactile signal indicating feedback on an operation performed by the user U on the virtual desktop screen VS, the determination unit 123 identifies the information processing device 10 as the host. On the other hand, for a tactile signal indicating feedback on an operation performed by the user U on the virtual object O, the determination unit 123 identifies the HMD app as the host. In addition, as described above, since the terminal device 20 displays virtual objects in addition to the virtual desktop screen VS, the determination unit 123 sets the priority of the HMD app to the highest. Therefore, when identifying the host type, in the case where the host is the HMD app, the determination unit 123 identifies the host as the main host, and in the case where the host is the information processing device 10, the determination unit 123 identifies the host as the sub-host.

[0191] Here, the app execution unit 122 emits light as a virtual laser beam in the direction indicated by the controller 30A in the virtual space. The light may be displayed on the virtual space to be recognizable to the user, or need not be displayed. The app execution unit 122 sets the collision position between the light and the object in the virtual space as the pointer position to be operated by the user.

[0192] However, in the case where there are multiple locations where the light collides with the object, the app execution unit 122 cannot determine which object the user wants to operate. Fig. 22 , the light collides with the virtual object O and the virtual desktop screen VS. Therefore, the app execution unit 122 cannot determine whether the user wants to operate the virtual object O or the virtual desktop screen VS. Therefore, the app execution unit 122 can set the collision position between the object located closer to the controller 30A in the virtual space and the light as the pointer position. For example, in Fig. 22 In , the virtual object O is set as the pointer position.

[0193] In addition, in the case where there are multiple positions where the light collides with the object, the app execution unit 122 may generate a haptic signal indicating a sense of friction as a haptic signal indicating feedback of the operation of the virtual object O. The haptic signal indicating a sense of friction is a signal that provides a rough feeling to the user. As long as there are multiple positions where the light collides with the object, the app execution unit 122 may generate a continuous waveform to continuously provide a sense of friction.

[0194] Next, the tactile feedback to the annular input device 30B will be described. In the case of sending a tactile signal or a signal obtained by conversion to the annular input device 30B, the signal may be sent to the annular input device in a pair of annular input devices that performs an operation on an object displayed on the display unit 140 or the terminal device 20. Note that in the case of performing an operation using two input devices, the signal may be sent to both annular input devices, or the signal may be sent to any one of the annular input devices.

[0195] In addition, the destination of the signal may be determined based on the distance between the annular input device 30B and the display unit 140 or the terminal device 20. The distance may be the distance between the annular input device 30B and the display unit 140 or the image display unit of the terminal device 20, or may be the distance between the annular input device 30B and a virtual object in a virtual space. For example, the signal may be transmitted to one of a pair of annular input devices that is at a shorter distance from the display unit 140.

[0196] Fig.23 is an explanatory diagram for describing an example in which tactile feedback is provided by the annular input device 30B including a pair of two annular input devices. Fig.23 A case is shown in which the distance between the annular input device 30B worn on the left hand H1 and the display unit 140 is shorter than the distance between the annular input device 30B worn on the right hand H2 and the display unit 140 .

[0197] The control unit 120 calculates the distance from the display unit 140 based on the information about the position of the annular input device 30B received by the communication unit 110 from the annular input device 30B via the terminal device 20. The control unit 120 determines that the distance between the annular input device worn on the left hand H1 and the display unit 140 is shorter than the distance between the annular input device worn on the right hand H2 and the display unit 140. Then, the control unit 120 performs control to output the tactile signal or the signal obtained by conversion to the annular input device worn on the left hand H1.

[0198] <<4. Example of Operation Processing According to First Embodiment>>

[0199] Next, we will refer to Fig.24 The operation processing flow of the information processing system according to the present embodiment is described. Fig.24 is a sequence diagram showing an example of the flow of operation processing of the information processing system according to the present embodiment.

[0200] like Fig.24As shown in , the tactile feedback device 30 detects the user's motion (S104). The tactile feedback device 30 sends a signal indicating the detected motion to the terminal device 20 (S108). The terminal device 20 sends the signal indicating the motion received from the tactile feedback device 30 to the information processing device 10 (S112).

[0201] When the communication unit 110 of the information processing device 10 receives the signal indicating the action, the generation unit 121 or the app execution unit 122 generates a tactile signal as feedback (FB) corresponding to the action ( S116 ).

[0202] Next, the determination unit 123 determines whether the tactile feedback device 30 is in an environment where notification information can be obtained from multiple hosts (step S120). In the case where the determination unit 123 determines that the tactile feedback device 30 is in an environment where notification information can be obtained only from one host (S120 / No), the communication unit 110 sends the generated tactile signal to the terminal device 20 (S124). The terminal device 20 sends the received tactile signal to the tactile feedback device 30 (S128), and the process proceeds to S144.

[0203] On the other hand, when the determination unit 123 determines that the tactile feedback device 30 is in an environment capable of acquiring notification information from a plurality of hosts ( S120 / Yes), the determination unit 123 determines whether the host type is a main host or a sub-host ( S132 ).

[0204] In the case where the host type is the primary host (S132 / primary host), the communication unit 110 transmits the generated haptic signal to the terminal device 20 (S136). The terminal device 20 transmits the haptic signal received from the haptic feedback device 30 to the haptic feedback device 30 (S140).

[0205] Next, the haptic feedback device 30 determines whether a haptic signal has been received (S144). In the case where a haptic signal has been received (S144 / Yes), the haptic feedback device 30 performs haptic feedback using the haptic signal (S148). That is, the haptic feedback device 30 performs the FB received from the main host.

[0206] On the other hand, in the case where the host type is a sub-host (S132 / sub-host), the conversion unit 124 converts the generated tactile signal (S152). The communication unit 110 transmits the signal obtained by the conversion performed by the conversion unit 124 to the terminal device 20 (S156). The terminal device 20 transmits the received signal as the signal obtained by the conversion to the tactile feedback device 30 (S160).

[0207] The tactile feedback device 30 provides tactile feedback using the received signal as the signal obtained by the conversion (S164). That is, the tactile feedback device 30 executes the FB received from the sub-host.

[0208] <<5. Second Embodiment>>

[0209] Next, a second embodiment of the present disclosure will be described. In the above-mentioned first embodiment, an example of controlling the operation of the terminal device 20 by the HMD app running on the information processing device 10 has been described. However, the overall operation of the terminal device, such as the control of the generation and display of virtual objects and the generation and conversion of tactile signals, can be controlled by a control unit included in the terminal device. In addition, the overall operation of the terminal device can be controlled by the HMD app executed by the control unit included in the terminal device. In the second embodiment of the present disclosure, an example in which the overall operation of the terminal device is controlled by the control unit included in the terminal device will be described.

[0210] Note that among the components included in the information processing system according to the second embodiment of the present disclosure, the components that are different from the components included in the information processing system according to the first embodiment of the present disclosure will be mainly described, and the detailed description of the components that are the same as those of the information processing system according to the first embodiment of the present disclosure will be omitted.

[0211] Will refer to Fig.25 A second embodiment of the present disclosure is described. Fig.25 1 is an explanatory diagram showing the configuration of the information processing system according to the second embodiment of the present disclosure and the internal configuration of the terminal device 40. In the second embodiment of the present disclosure, the tactile feedback device 30 is connected to the information processing device 10 and the terminal device 40, and various information is exchanged. In addition, the user can input an operation from the tactile feedback device 30 to an object displayed on the information processing device 10 or a virtual object displayed on the terminal device 40.

[0212] The tactile feedback device 30 sends a signal indicating an operation input of the user or a signal indicating an action of the user to the information processing device 10 or the terminal device 40. More specifically, the tactile feedback device 30 sends a signal indicating an operation input or an action on an object displayed on the information processing device 10 to the information processing device 10. On the other hand, the tactile feedback device 30 sends a signal indicating an operation input or an action on a virtual object displayed on the terminal device 40 to the terminal device 40.

[0213] In addition, the tactile feedback device 30 transmits information indicating the connection status with the information processing device 10 and the terminal device 40 to the information processing device 10 and the terminal device 40. In addition, the tactile feedback device 30 receives information used by the determination unit 123 of the information processing device 10 to determine the host priority, such as the wearing status of the terminal device 40 worn by the user, from the terminal device 40, and transmits the information to the information processing device 10. The determination unit 123 of the information processing device 10 determines the host priority based on the information obtained through the communication between the communication unit 110 and the tactile feedback device 30.

[0214] Next, details of the terminal device 40 (an example of an information processing device) according to the second embodiment of the present disclosure will be described. Fig.25 As shown in , the terminal device 40 includes a communication unit 410 , a control unit 420 , a storage unit 430 , an image display unit 440 , and an operation input unit 450 .

[0215] (Communication unit 410)

[0216] The communication unit 410 establishes a communication connection with an external device and sends and receives data. For example, the communication unit 110 sends data to the tactile feedback device 30 and receives data from the tactile feedback device 30. The communication unit 410 can establish a communication connection with the tactile feedback device 30, etc., for example, through a wired or wireless LAN, Wi-Fi, Bluetooth, etc.

[0217] For example, the communication unit 410 receives a signal indicating a user's motion detected by the tactile feedback device 30 from the tactile feedback device 30. In addition, the communication unit 410 transmits a tactile signal generated by a generation unit 421 to be described later or a signal obtained by conversion performed by a conversion unit 424 to be described later to the tactile feedback device 30.

[0218] (Control Unit 420)

[0219] The control unit 420 functions as an arithmetic processing device and a control device, and controls the overall operation in the terminal device 40 according to various programs. For example, the control unit 420 controls outputs such as screen display and audio output by the image display unit 440 to be described later. In addition, the control unit 420 constructs a virtual space to be displayed on the image display unit 440.

[0220] For example, the control unit 420 is implemented by an electronic circuit such as a CPU or a microprocessor. In addition, the control unit 420 may include a ROM and a RAM, the ROM storing programs to be used, operating parameters, etc., and the RAM temporarily storing parameters that are changed as needed.

[0221] Furthermore, the control unit 420 according to the present embodiment may also function as the generating unit 421 , the determining unit 423 , and the converting unit 424 .

[0222] (Generation Unit 421)

[0223] The generating unit 421 generates a tactile signal. For example, the generating unit 421 generates a tactile signal based on a signal indicating a user's action detected by the tactile feedback device 30. The tactile signal generated by the generating unit 421 is similar to the tactile signal generated by the app executing unit 122 of the information processing device 10 according to the first embodiment of the present disclosure.

[0224] (Determination Unit 423)

[0225] The determination unit 423 identifies the host as the output source of the haptic signal generated by the generation unit 421 to obtain the host type as a determination result. The determination unit 423 identifies the host type of the host in substantially the same manner as the determination unit 123 of the information processing device 10. For example, in the case where the application that has generated the notification information is an HMD app, the determination unit 423 may identify the HMD app or the HMD as the host.

[0226] (Conversion unit 424)

[0227] The conversion unit 424 controls the conversion process of the haptic signal generated by the generation unit 421 based on the host type identified by the determination unit 423. The conversion unit 424 controls the conversion process of the haptic signal in a manner similar to the conversion unit 124 of the information processing device 10.

[0228] (Storage unit 430)

[0229] The storage unit 430 is realized by a ROM that stores programs, operation parameters, and the like to be used for the processing of the control unit 420 , and a RAM that temporarily stores parameters and the like that are changed as needed.

[0230] (Image display unit 440)

[0231] For example, in the case where the terminal device 40 is configured as an HMD, the image display unit 440 includes a left screen and a right screen respectively fixed to the left eye and the right eye of the user, and displays the left eye image and the right eye image. For example, the image display unit 440 includes a display panel such as a liquid crystal display or an organic electroluminescent (EL) display or a laser scanning display such as a retinal projection display. In addition, the image display unit 440 may include an imaging optical system that enlarges and projects the display screen to form an enlarged virtual image with a predetermined viewing angle on the pupil of the user.

[0232] (Operation Input Unit 450)

[0233] The operation input unit 450 receives an operation from the user and outputs input information to the control unit 420. The operation input unit 450 is implemented by a button, a sensor, etc., and receives, for example, an operation input from the user such as turning on or off the power of the terminal device 40 or adjusting the brightness of the image display unit 440.

[0234] According to the second embodiment described above, the information processing device 10 and the terminal device 40 as hosts cause their own conversion units to convert the generated tactile signals. With such a configuration, even when the information processing device 10 does not serve as multiple hosts, it is possible for the user to recognize notifications from different hosts. Note that the information processing device 10 according to this embodiment does not need to have the function of the app execution unit 122.

[0235] <<6. Third embodiment>>

[0236] Next, a third embodiment of the present disclosure will be described. In the above-mentioned first embodiment, an example of performing conversion processing of tactile signals by the conversion unit 124 of the information processing device 10 has been described. Note that the conversion processing of tactile signals can be performed by the tactile feedback device 50. In the third embodiment of the present disclosure, an example of performing conversion processing of tactile signals by the tactile feedback device 50 (an example of an information processing device) will be described.

[0237] Note that among the components included in the information processing system according to the third embodiment of the present disclosure, the components that are different from the components included in the information processing system according to the first embodiment of the present disclosure will be mainly described, and the detailed description of the components that are the same as those of the information processing system according to the first embodiment of the present disclosure will be omitted.

[0238] Will refer to Fig.26 A third embodiment of the present disclosure is described. Fig.26 is an explanatory diagram showing the configuration of an information processing system according to a third embodiment of the present disclosure and the internal configuration of a tactile feedback device 50. In the third embodiment of the present disclosure, the communication unit 110 of the information processing device 10 transmits a tactile signal generated by the generation unit 121 or the app execution unit 122 and a determination result from the determination unit 123 to the tactile feedback device 50. The tactile feedback device 50 performs a conversion process on the tactile signal based on the received tactile signal and the determination result.

[0239] <6-1. Configuration Example of Tactile Feedback Device 50>

[0240] Hereinafter, the details of the tactile feedback device 50 according to the third embodiment of the present disclosure will be described. Fig.26 As shown in , the tactile feedback device 50 includes an operation input unit 510 , a communication unit 520 , a control unit 530 and an output unit 540 .

[0241] (Operation Input Unit 510)

[0242] The operation input unit 510 is realized by a button, a sensor, etc., and detects an operation input of a user or a user's action. Examples of the input operation or action detected by the operation input unit 510 are as described in the first embodiment.

[0243] (Communication unit 520)

[0244] The communication unit 520 establishes a communication connection with an external device and sends and receives data. For example, the communication unit 520 sends data to the terminal device 20 and receives data from the terminal device 20. The communication unit 520 can establish a communication connection with the terminal device 20, etc., for example, through a wired or wireless LAN, Wi-Fi, Bluetooth, etc.

[0245] For example, the communication unit 520 transmits a signal indicating the action of the user detected by the operation input unit 510 to the information processing device 10 via the terminal device 20. In addition, the communication unit 520 receives a tactile signal generated by the generation unit 121 or the app execution unit 122 of the information processing device 10 and a determination result as a result of the host type being identified by the determination unit 123 from the information processing device 10 via the terminal device 20.

[0246] (Control Unit 530)

[0247] The control unit 530 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing device 10 according to various programs. For example, the control unit 530 is implemented by an electronic circuit such as a CPU or a microprocessor. In addition, the control unit 530 may include a ROM and a RAM, the ROM storing programs to be used, operating parameters, etc., and the RAM temporarily storing parameters and the like that are changed as needed.

[0248] The control unit 530 according to the present embodiment can also function as the conversion unit 531 and the output control unit 532 .

[0249] (Conversion unit 531)

[0250] The conversion unit 531 controls the conversion process of the haptic signal received by the communication unit 520 based on the host type indicated by the determination result received by the communication unit 520. The conversion unit 531 controls the conversion process of the haptic signal in a manner similar to the conversion unit 124 of the information processing device 10 according to the first embodiment.

[0251] (Output control unit 532)

[0252] The output control unit 532 controls the output of vibration from the output unit 540 to be described later, the vibration being indicated by a signal obtained through conversion performed by the conversion unit 531 or a tactile signal received by the communication unit 520. For example, in a case where the determination result received by the communication unit 520 includes information indicating that the host type is a primary host, the output control unit 532 may control the output unit 540 to output the vibration indicated by the tactile signal received by the communication unit 520. In addition, in a case where the tactile signal is converted by the conversion unit 531, the output control unit 532 may control the output unit 540 to output the vibration indicated by the signal obtained through conversion performed by the conversion unit 531.

[0253] (Output unit 540)

[0254] The output unit 540 is an example of a tactile feedback unit that outputs vibration (provides tactile feedback) based on the tactile signal or the signal obtained through conversion performed by the conversion unit 531 under the control of the output control unit 532 .

[0255] <6-2. Example of Operation Processing According to Third Embodiment>

[0256] Next, we will refer to Fig. 27 An operation processing flow of the information processing system according to the third embodiment of the present disclosure is described. Fig. 27 is a sequence diagram illustrating an example of an operation processing flow of the information processing system according to the third embodiment of the present disclosure.

[0257] like Fig. 27 As shown in , the operation input unit 510 of the tactile feedback device 50 detects the user's action (S204). The communication unit 520 sends a signal indicating the detected action to the terminal device 20 (S208). The terminal device 20 sends the signal indicating the action received from the tactile feedback device 50 to the information processing device 10 (S212).

[0258] When the communication unit 110 of the information processing device 10 receives the signal indicating the action, the generation unit 121 or the app execution unit 122 generates a haptic signal as a FB corresponding to the action ( S216 ).

[0259] Next, the determination unit 123 determines whether the tactile feedback device 50 is in an environment where notification information can be obtained from multiple hosts (step S220). If the determination unit 123 determines that the tactile feedback device 50 is in an environment where notification information can be obtained from only one host (S220 / No), the communication unit 110 sends the generated tactile signal to the terminal device 20 (S224). The terminal device 20 sends the received tactile signal to the tactile feedback device 50 (S228).

[0260] Next, the control unit 530 of the tactile feedback device 50 determines whether the communication unit 520 has received a tactile signal from the terminal device 20 (S232). In the case where the communication unit 520 has received a tactile signal (S232 / Yes), the output control unit 532 controls the output unit 540 to provide tactile feedback using the tactile signal (S236). That is, the output unit 540 executes the FB received from the main host.

[0261] In a case where the communication unit 520 has not received the haptic signal ( S232 / No), the process proceeds to S252 .

[0262] On the other hand, when the determination unit 123 determines that the tactile feedback device 50 is in an environment capable of acquiring notification information from a plurality of hosts ( S220 / Yes), the determination unit 123 determines whether the host type is a main host or a sub-host ( S240 ).

[0263] Next, the communication unit 110 transmits the host type determination result and the generated haptic signal to the terminal device 20 (S244). The terminal device 20 transmits the determination result and the haptic signal received from the haptic feedback device 50 to the haptic feedback device 50 (S248).

[0264] In the case where the host type indicated by the determination result received by the communication unit 520 is a sub-host (S252 / sub-host), the conversion unit 531 of the tactile feedback device 50 converts the tactile signal received by the communication unit 520 (S256). Then, the output control unit 532 controls the output unit 540 to provide tactile feedback using the signal obtained by the conversion (S260). That is, the output unit 540 executes the FB received from the sub-host.

[0265] On the other hand, in the case where the host type indicated by the determination result received by the communication unit 520 is the master host (S252 / master host), the output control unit 532 controls the output unit 540 to provide tactile feedback (S148) using the tactile signal received by the communication unit 520. That is, the output unit 540 executes the FB received from the master host.

[0266] <<7. Fourth embodiment>>

[0267] The third embodiment of the present disclosure has been described above. Next, the fourth embodiment of the present disclosure will be described. In the third embodiment of the present disclosure, an example in which the host type is identified by the determination unit 123 of the information processing device 10 has been described. However, the host type may be identified by a tactile feedback device.

[0268] In addition, in the third embodiment of the present disclosure, an example has been described in which a tactile signal is generated by the information processing device 10. However, a tactile signal may be generated by both the information processing device 10 as a host and the terminal device 20. In a fourth embodiment of the present disclosure, an example in which a host type is identified by a tactile feedback device and a tactile signal is generated by each host will be described.

[0269] Note that among the components included in the information processing system according to the fourth embodiment of the present disclosure, the components that are different from the components included in the information processing system according to the third embodiment of the present disclosure will be mainly described, and the detailed description of the components that are the same as the information processing system according to the third embodiment of the present disclosure will be omitted.

[0270] Will refer to Fig.28 A fourth embodiment of the present disclosure is described. Fig.28 1 is an explanatory diagram showing the configuration of an information processing system according to a fourth embodiment of the present disclosure and the internal configuration of a tactile feedback device 60. In the fourth embodiment of the present disclosure, the tactile feedback device 60 communicates with the information processing device 10 and the terminal device 40. Here, the terminal device 40 according to the fourth embodiment of the present disclosure has a function similar to that of the terminal device 40 according to the second embodiment of the present disclosure. That is, the overall operation of the terminal device 40 is controlled by the control unit 420 included in the terminal device 40.

[0271] When transmitting the tactile signal generated by the generating unit 121, the communication unit 110 of the information processing device 10 according to the present embodiment transmits identification information indicating that the host is the information processing device 10 together with the tactile signal to the tactile feedback device 60. In addition, when transmitting the tactile signal generated by the generating unit 421, the communication unit 410 of the terminal device 40 transmits identification information indicating that the host is the terminal device 40 together with the tactile signal to the tactile feedback device 60.

[0272] Next, the details of the tactile feedback device 60 according to the fourth embodiment of the present disclosure will be described. Fig.26As shown in , the tactile feedback device 60 includes an operation input unit 610, a communication unit 620, a control unit 630, and an output unit 640. In addition, the control unit 630 according to the present embodiment may also be used as a conversion unit 631, an output control unit 632, and a determination unit 633. Here, the operation input unit 610, the output control unit 632, and the output unit 640 have similar functions to the operation input unit 510, the output control unit 532, and the output unit 540 of the tactile feedback device 50 according to the third embodiment of the present disclosure, and thus their description will be omitted.

[0273] (Communication unit 620)

[0274] The communication unit 620 transmits and receives data to and from the information processing device 10 and the terminal device 40. For example, the communication unit 620 transmits a signal indicating the action of the user detected by the operation input unit 610 to the information processing device 10 or the terminal device 40. More specifically, the communication unit 620 transmits a signal indicating an operation input or action on an object displayed on the information processing device 10 to the information processing device 10. On the other hand, the communication unit 620 transmits a signal indicating an operation input or action on a virtual object displayed on the terminal device 40 to the terminal device 40. In addition, the communication unit 620 receives identification information including information indicating whether the host is the information processing device 10 or the terminal device 40 from the information processing device 10 or the terminal device 40.

[0275] In addition, the tactile feedback device 60 receives information used by the determination unit 633 to determine the master priority from the terminal device 40, such as the wearing state of the terminal device 40 worn by the user.

[0276] (Determination Unit 633)

[0277] The determination unit 633 identifies the host as the output source of the tactile signal generated by the generation unit 121 of the information processing device 10 or the generation unit 421 of the terminal device 40 to obtain the host type as the determination result. The determination unit 633 can use the identification information obtained by the communication unit 620 as the host type determination result. In addition, the determination unit 633 can determine the host priority in a manner similar to the determination unit 123 of the information processing device 10 according to the first embodiment, and use the priority as the type. At this time, the determination unit 633 can determine the host priority based on the information for determining the host priority received by the communication unit 620.

[0278] (Conversion unit 631)

[0279] The conversion unit 631 controls the conversion process of the haptic signal received by the communication unit 620 based on the host type indicated by the determination result from the determination unit 633. The conversion unit 631 controls the conversion process of the haptic signal in a manner similar to the conversion unit 124 of the information processing device 10 according to the first embodiment.

[0280] According to the fourth embodiment described above, the host type determination and the tactile signal conversion process are performed by the tactile feedback device 60. With such a configuration, the user can recognize notifications from different hosts regardless of the type of the connected information processing device or terminal device.

[0281] (Replenish)

[0282] Note that in the fourth embodiment described above, an example has been used in which the host type is identified by the determination unit 633 of the tactile feedback device 60, but the host type may be identified by both the information processing device 10 and the terminal device 40. In this case, the determination results of the information processing device 10 and the terminal device 40 are transmitted from the information processing device 10 and the terminal device 40 to the tactile feedback device 60. Then, the conversion unit 631 controls the conversion process of the tactile signal based on the host type indicated by the determination result received by the communication unit 620.

[0283] Furthermore, according to the fourth embodiment described above, an example has been described in which the tactile feedback device 60 is connected to the information processing device 10 and the terminal device 40, and the control unit itself included in the terminal device 40 controls the overall operation of the terminal device. However, the terminal device does not need to be connected to the tactile feedback device 60, and may be controlled by an application (HMD app) executed by the information processing device 10 to operate the terminal device in a manner similar to the terminal device 20 according to the first embodiment. In this case, the communication unit 110 of the information processing device 10 transmits identification information indicating that the host is the HMD app together with the tactile signal to the tactile feedback device 60 when outputting the tactile signal generated by the app execution unit 122.

[0284] <<8. Hardware Configuration Example>>

[0285] Each embodiment of the present disclosure has been described above. The above information processing is realized by the cooperation of software and hardware. Hereinafter, a hardware configuration example that can be applied to the information processing device 10, the terminal device 20, the tactile feedback device 30, the terminal device 40, the tactile feedback device 50, and the tactile feedback device 60 will be described.

[0286] Fig.29is a block diagram showing an example of the information processing device 90. Note that the hardware configuration example of the information processing device 90 to be described below is merely an example of the hardware configuration of the information processing device 10, the terminal device 20, the tactile feedback device 30, the terminal device 40, the tactile feedback device 50, and the tactile feedback device 60. Therefore, the information processing device 10, the terminal device 20, the tactile feedback device 30, the terminal device 40, the tactile feedback device 50, and the tactile feedback device 60 do not necessarily have Fig.29 In addition, it is not necessary to set all the hardware components shown in the information processing device 10, the terminal device 20, the tactile feedback device 30, the terminal device 40, the tactile feedback device 50 and the tactile feedback device 60. Fig.29 Some of the hardware components shown in .

[0287] like Fig.29 As shown in , the information processing device 90 includes a CPU 901, a read-only memory (ROM) 903, and a RAM 905. In addition, the information processing device 90 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Instead of the CPU 901 or in addition to the CPU 901, the information processing device 90 may include a processing circuit such as a graphics processing unit (GPU), a digital signal processor (DSP), or an application-specific integrated circuit (ASIC).

[0288] The CPU 901 functions as an arithmetic processing device and a control device, and controls all or part of the operations in the information processing device 90 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs used by the CPU 901, operating parameters, and the like. The RAM 905 temporarily stores programs used in the execution of the CPU 901, parameters changed as needed during execution, and the like. The CPU 901, the ROM 903, and the RAM 905 are connected to each other via a host bus 907, which includes an internal bus such as a CPU bus. In addition, the host bus 907 is connected to an external bus 911 such as a peripheral component interconnect / interface (PCI) bus via a bridge 909.

[0289] For example, the CPU 901 may cooperate with the ROM 903 , the RAM 905 , and software to realize the functions of the control unit 120 , the control unit 330 , and the control unit 420 .

[0290] The input device 915 is, for example, a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a joystick, etc. In addition, the input device 915 may include a microphone for detecting a user's voice. The input device 915 may be, for example, a remote control device using infrared or other radio waves, or may be an external connection device 929 such as a mobile phone suitable for operation of the information processing device 90. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the CPU 901. By operating the input device 915, the user inputs various data to the information processing device 90 or gives instructions for performing processing operations.

[0291] In addition, the input device 915 may include an imaging device and a sensor. The imaging device is a device that generates a captured image by imaging a real space using various components, such as an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) and a lens for controlling image formation of a subject image on the imaging element. The imaging device can capture a still image, or can capture a moving image.

[0292] Examples of the sensor include various types of sensors such as a range finder, an accelerometer, a gyroscope, a geomagnetic sensor, a vibration sensor, a light sensor, and a sound sensor. The sensor acquires, for example, information about the state of the information processing device 90 itself, such as the posture of the housing of the information processing device 90, and information about the environment around the information processing device 90, such as the brightness and noise around the information processing device 90. In addition, the sensor may include a global positioning system (GPS) sensor that receives a GPS signal to measure the latitude, longitude, and altitude of the device.

[0293] The output device 917 includes a device that can visually or auditorily notify the user of the acquired information. The output device 917 may be, for example, a display device such as a liquid crystal display (LCD) or an organic electroluminescent (EL) display, an audio output device such as a speaker or a headset, etc. In addition, the output device 917 may include a plasma display panel (PDP), a projector, a hologram, a printer device, etc. The output device 917 outputs the result of the processing performed by the information processing device 90 as text or visual data such as an image, or outputs the result as a sound such as a voice or audio. In addition, the output device 917 may include a lighting device that brightens the surrounding environment, etc.

[0294] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 90. The storage device 919 includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage device 919 stores programs or various types of data to be executed by the CPU 901, various types of data acquired from the outside, and the like.

[0295] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is internally or externally attached to the information processing apparatus 90. The drive 921 reads information recorded in the attached removable recording medium 927, and outputs the information to the RAM 905. In addition, the drive 921 writes records to the attached removable recording medium 927.

[0296] The connection port 923 is a port for directly connecting a device to the information processing device 90. The connection port 923 may be, for example, a universal serial bus (USB) port, an IEEE1394 port, a small computer system interface (SCSI) port, etc. In addition, the connection port 923 may be an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI (registered trademark)) port, etc. By connecting an external connection device 929 to the connection port 923, various types of data can be exchanged between the information processing device 90 and the external connection device 929.

[0297] The communication device 925 is, for example, a communication interface including a communication device for connecting to a local area network or a communication network with a wireless base station, etc. The communication device 925 may be, for example, a communication card for a wired or wireless LAN, Bluetooth (registered trademark), Wi-Fi, or a wireless USB (WUSB). In addition, the communication device 925 may be a router for optical communication, a router for an asymmetric digital subscriber line (ADSL), a modem for various types of communication, etc. For example, the communication device 925 uses a predetermined protocol such as TCP / IP to send and receive signals through the Internet or using other communication devices, etc. In addition, the local area network to which the communication device 925 is connected or the communication network with a base station is a network connected in a wired or wireless manner, and examples of the network include the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, etc.

[0298] <<9.Supplement>>

[0299] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. Obviously, a person of ordinary skill in the technical field of the present disclosure can think of various changes or modifications within the scope of the technical concept described in the claims, and it can be naturally understood that these changes or modifications also fall within the technical scope of the present disclosure.

[0300] For example, in the above-mentioned embodiment, an example has been described in which the information processing device 10 communicates with the tactile feedback device 30 via the terminal device 20, but the present technology is not limited to such an example. For example, the information processing device 10 may be directly connected to the tactile feedback device 30, and the terminal device 20 and the tactile feedback device 30 do not need to be connected. In this case, the terminal device 20 and the tactile feedback device 30 exchange various information via the information processing device 10.

[0301] In addition, in the above embodiment, the example in which the terminal device 20 is an HMD or the like worn on the user's head and provides a virtual space to the user is mainly described, but the present disclosure is not limited to such an example. For example, the terminal device 20 may be a mobile entity such as a vehicle or a train.

[0302] In addition, in the above embodiment, an example has been described in which the tactile feedback device 30 has the function of an operation input unit, but the tactile feedback device 30 does not need to include an operation input unit. For example, the tactile feedback device 30 may be a glove, wristband, watch, etc. that can provide tactile feedback.

[0303] In addition, in the above-mentioned embodiment, an example in which the tactile signal includes a parameter value representing vibration has been described, but the tactile signal is not limited to such an example. For example, the tactile signal may be a waveform representing vibration. In this case, the conversion unit 124 may perform conversion processing by performing pitch shifting or filtering on the tactile signal or performing fast Fourier transform (FFT) processing on the tactile signal.

[0304] In addition, one or more computer programs may be created to enable hardware such as a CPU, ROM, and RAM built into the above-mentioned information processing device 10, terminal device 20, or tactile feedback device 30 to exhibit the functions of the information processing device 10, terminal device 20, or tactile feedback device 30. In addition, a computer-readable storage medium storing one or more computer programs is also provided.

[0305] In addition, the effects described in this specification are only exemplary or illustrative, not restrictive. That is, in addition to or instead of the above effects, the technology according to the present disclosure can show other effects that are obvious to those skilled in the art from the description of this specification.

[0306] Note that the following configurations also fall within the technical scope of the present disclosure.

[0307] (1) An information processing device includes a control unit that converts a tactile signal corresponding to notification information sent by each of a plurality of hosts into a different tactile signal based on a determination result of an identification of a host type according to the plurality of hosts.

[0308] (2) The information processing device according to (1) above, wherein the control unit controls a conversion process of a haptic signal corresponding to the notification information.

[0309] (3) The information processing device according to (1) or (2) above, wherein:

[0310] A plurality of display objects corresponding one to one with the plurality of hosts are displayed on a virtual space displayed on a display unit of the second information processing device,

[0311] The notification information is generated based on the user's operation on each of the display objects using a tactile feedback device, and

[0312] The tactile signal is provided by the tactile feedback device.

[0313] (4) The information processing device according to (3) above, wherein each of the hosts is an application executed by at least one of the information processing device and the second information processing device.

[0314] (5) An information processing device according to any one of (1) to (4) above, wherein the control unit obtains a host type based on the host priority as the determination result, and controls the conversion processing of the tactile signal according to the host type based on the host priority.

[0315] (6) The information processing apparatus according to (5) above, wherein the determination result includes, as the host type based on priority, a result of determining whether each of the hosts is a primary host or a sub-host.

[0316] (7) An information processing device according to any one of (1) to (6) above, wherein the control unit obtains identification information for identifying each of the hosts as the determination result, and controls the conversion processing of the tactile signal according to the identification information.

[0317] (8) The information processing device according to (7) above, wherein the identification information includes information indicating whether each of the hosts is an HMD (Head Mounted Display) or a PC (Personal Computer).

[0318] (9) The information processing device according to (7) above, wherein the identification information includes information indicating whether each of the hosts is executed on a PC to run an application of an HMD or the PC.

[0319] (10) The information processing device according to (6) above, wherein, when it is determined based on the user's operation on the display of the HMD that the user is using an application executed on the PC to run the HMD, the control unit identifies the HMD application as a main host and controls conversion processing of a tactile signal corresponding to notification information generated in response to the user's operation.

[0320] (11) An information processing device according to (6) or (10) above, wherein, when it is determined based on the user's operation on the display of the PC that the user is using a 2D application executed on the PC, the control unit identifies the PC as a main host and controls the conversion processing of a tactile signal corresponding to the notification information generated in response to the user's operation.

[0321] (12) The information processing device according to (2) above, wherein the conversion process includes a process of converting a frequency, an amplitude, an output period, or a waveform included in the tactile signal.

[0322] (13) The information processing device according to (2) above, wherein the control unit controls conversion processing of the haptic signals from the plurality of hosts to convert the haptic signals into haptic signals having non-overlapping frequency bands.

[0323] (14) The information processing device according to (2) above, wherein the control unit controls conversion processing of haptic signals from a plurality of hosts to convert the haptic signals into haptic signals having non-overlapping amplitudes.

[0324] (15) The information processing device according to any one of (1) to (14) above, the information processing device serving as a host further including a communication unit, wherein:

[0325] The control unit controls:

[0326] as a host outputting the notification, and

[0327] A process of sending the notification from the communication unit to a tactile feedback device.

[0328] (16) The information processing device according to any one of (1) to (15) above, the information processing device serving as a tactile feedback device further comprising a communication unit and at least one tactile feedback unit, wherein:

[0329] The communication unit receives the notification from each of the hosts, and

[0330] The control unit inputs the notification to the at least one tactile feedback unit.

[0331] (17) The information processing device according to (3) or (4) above, wherein the tactile signal is transmitted to the tactile feedback device via the second information processing device.

[0332] (18) An information processing method comprising:

[0333] Based on a determination result of identification according to the host type of the plurality of hosts, a haptic signal corresponding to notification information transmitted by each of the plurality of hosts is converted into a different haptic signal.

[0334] (19) A program for causing a computer to:

[0335] A control unit performs a process of converting a haptic signal corresponding to notification information transmitted by each of the plurality of hosts into a different haptic signal based on a determination result of identification according to host types of the plurality of hosts.

[0336] Reference numerals list

[0337] 10Information processing device

[0338] 20Terminal device

[0339] 30Haptic feedback device

[0340] 110 Communication Unit

[0341] 120 control unit

[0342] 121Generation Unit

[0343] 122App execution unit

[0344] 123 Determine unit

[0345] 124 conversion units

[0346] 130 storage units

[0347] 140 display units

[0348] 150 Operation input unit

Claims

1. An information processing device, comprising a control unit, which converts a tactile signal corresponding to notification information sent by each of a plurality of hosts into a different tactile signal based on a determination result of an identification of a host type according to a plurality of hosts.

2. The information processing device according to claim 1, wherein: The control unit controls a conversion process of a haptic signal corresponding to the notification information.

3. The information processing device according to claim 1, wherein: A plurality of display objects corresponding one to one with the plurality of hosts are displayed on a virtual space displayed on a display unit of the second information processing device, The notification information is generated based on the user's operation on each of the display objects using a tactile feedback device, and The tactile signal is provided by the tactile feedback device.

4. The information processing device according to claim 3, wherein: Each of the hosts is an application executed by at least one of the information processing device and the second information processing device.

5. The information processing device according to claim 1, wherein: The control unit acquires a host type based on the priority of the hosts as the determination result, and controls a conversion process of the haptic signal according to the host type based on the priority of the hosts.

6. The information processing device according to claim 5, wherein: The determination result includes: a result obtained by determining whether each of the hosts is a main host or a sub-host as a host type based on priority.

7. The information processing device according to claim 1, wherein: The control unit acquires identification information for identifying each of the hosts as the determination result, and controls a conversion process of the haptic signal according to the identification information.

8. The information processing device according to claim 7, wherein: The identification information includes information indicating whether each of the hosts is an HMD (Head Mounted Display) or a PC (Personal Computer).

9. The information processing device according to claim 7, wherein: The identification information includes information indicating whether each of the hosts is executed on a PC to run an application of an HMD or the PC.

10. The information processing device according to claim 6, wherein: In a case where it is determined based on an operation of a display of the HMD by the user that the user is using an application executed on a PC to run the HMD, the control unit identifies the HMD application as a primary host and controls conversion processing of a haptic signal corresponding to notification information generated in response to the operation of the user.

11. The information processing device according to claim 6, wherein: When it is determined based on the user's operation on the display of the PC that the user is using a 2D application executed on the PC, the control unit identifies the PC as a main host and controls conversion processing of a tactile signal corresponding to notification information generated in response to the user's operation.

12. The information processing device according to claim 2, wherein: The conversion process includes a process of converting a frequency, an amplitude, an output period, or a waveform included in the haptic signal.

13. The information processing device according to claim 2, wherein: The control unit controls conversion processing of haptic signals from the plurality of hosts to convert the haptic signals into haptic signals having non-overlapping frequency bands.

14. The information processing device according to claim 2, wherein: The control unit controls a conversion process of haptic signals from the plurality of processing entities to convert the haptic signals into haptic signals having non-overlapping amplitudes.

15. The information processing device according to claim 1, the information processing device serving as a host further comprising a communication unit, wherein The control unit controls: as a host outputting the notification, and A process of sending the notification from the communication unit to a tactile feedback device.

16. The information processing device according to claim 1, used as a tactile feedback device further comprising a communication unit and at least one tactile feedback unit, wherein: The communication unit receives the notification from each of the hosts, and The control unit inputs the notification to the at least one tactile feedback unit.

17. The information processing device according to claim 3, wherein: The tactile signal is sent to the tactile feedback device via the second information processing device.

18. An information processing method, comprising: Based on a determination result of identification according to the host type of the plurality of hosts, a haptic signal corresponding to notification information transmitted by each of the plurality of hosts is converted into a different haptic signal.

19. A program for causing a computer to: A control unit performs a process of converting a haptic signal corresponding to notification information transmitted by each of the plurality of hosts into a different haptic signal based on a determination result of identification according to host types of the plurality of hosts.

Images