Information processing system, control method for information processing system, and program

Abstract

To provide an information processing system, a control method for the information processing system, and a program that allow a user to distinguish a sound from a specific sound source while leaving the sounds from each sound source as they are, even when a plurality of sound sources are present in a direction of interest that is of interest to the user.SOLUTION: An information processing system 1000 comprises: display means (a display unit 107) capable of displaying an image of a mixed reality space; playback means (an audio output unit 105) capable of playing back sounds from a plurality of sound sources to a user; vibration applying means (a vibration unit 117) for applying vibration to the user; control means (a control unit 111) for controlling displaying a virtual object that can be linked to each sound source on the display means; and selection means for selecting the virtual object (an object selection instruction unit 116). The vibration applying means applies vibration to the user in accordance with the intensity of the sound from the sound source associated with the virtual object if the virtual object is selected.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program. [Background technology]

[0002] Conventionally, methods for distinguishing a specific sound source even under multiple sound sources include, for example, a method using a directional microphone, a method controlling the volume of a sound source located in the direction in which the user's face is facing, and a method of playing only a sound source selected by the user. In addition, for example, Patent Document 1 discloses a technology for distinguishing a specific sound source by increasing the volume delivered to an object of interest of the user. Patent Document 2 discloses a technology for distinguishing a specific sound source by bringing a ring-shaped speaker worn on a finger close to the ear. Non-Patent Document 1 discloses a technology for distinguishing a specific sound source by applying sound vibrations from the skin even in a noisy environment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-163649 A [Patent Document 2] Patent No. 3771207 [Non-patent literature]

[0004] [Non-Patent Document 1] Yusuke Torikai, "Presentation of skin vibration during sound reception", [online], Waseda University, School of Fundamental Science and Engineering, Department of Representation Engineering, 2012 Graduation Thesis, [October 13, 2023], Internet〈URL:http: / / www.ias.sci.waseda.ac.jp / GraduationThesis / 2012_summary / 1W090367_s.pdf〉 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, when there are multiple sound sources in a direction of interest that the user is interested in, it may be difficult to distinguish a sound from a specific sound source among these sound sources. Also, when trying to distinguish a sound from a specific sound source and only reproducing the sound from that sound source, the sounds from the remaining sound sources are blocked, and as a result, there is a risk that the sounds from all sound sources cannot be heard as a whole.

[0006] The present invention has been made in consideration of the above problems. The present invention aims to provide an information processing system that can distinguish a sound from a specific sound source while leaving the sounds from each sound source as they are, even when multiple sound sources are present in a direction of interest that is of interest to the user. Similarly, the present invention aims to provide a control method and a program for an information processing system that can distinguish a sound from a specific sound source while leaving the sounds from each sound source as they are, even when multiple sound sources are present in a direction of interest that is of interest to the user. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the information processing system of the present invention comprises a display means capable of displaying an image of at least one of a real space and a virtual space, a playback means capable of playing sounds from a plurality of sound sources to a user viewing the image displayed on the display means, a vibration imparting means for imparting vibration to the user, a control means for controlling the display means to display virtual objects that can be linked to each of the sound sources together with the image, and a selection means for selecting the virtual object displayed on the display means, wherein when the virtual object is selected by the selection means, the vibration imparting means imparts vibration to the user in accordance with the strength of sound from a sound source among the plurality of sound sources that is linked to the virtual object. Effect of the Invention

[0008] According to the present invention, even when a plurality of sound sources are present in a direction of interest that is of interest to the user, the user can distinguish a sound from a specific sound source while leaving the sounds from each sound source unchanged. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of an information processing system according to a first embodiment. [Diagram 2] 4 is a flowchart showing a process executed by an information processing device of the information processing system. [Diagram 3] FIG. 13 is a diagram showing a screen for switching whether or not vibration is applied in accordance with the intensity of a sound. [Figure 4] 11 is a diagram illustrating whether or not a virtual object is selected. FIG. [Diagram 5] 13 is a flowchart showing a process executed by an information processing device of an information processing system according to a second embodiment. [Figure 6] FIG. 13 is a diagram for explaining a region of interest. [Figure 7] FIG. 13 is a diagram for explaining a region of interest. [Figure 8] 13 is a flowchart showing a process executed by an information processing device of an information processing system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exert a similar function. In addition, any configuration may be added. In addition, any two or more configurations (features) of each embodiment can be combined.

[0011] First Embodiment Hereinafter, the first embodiment will be described with reference to Figs. 1 to 4. Fig. 1 is a block diagram showing an example of a hardware configuration of an information processing system according to the first embodiment. The information processing system 1000 shown in Fig. 1 has a display device (first device) 101, an information processing device (third device) 102, and an operation device (second device) 103. The display device 101 to the operation device 103 are configured separately from each other. In addition, the display device 101 to the operation device 103 are connected to each other so that they can communicate with each other. The connection method is not particularly limited, and may be, for example, a wired connection method or a wireless connection method.

[0012] The display device 101 has a sound recording unit (sound collection means) 104, a sound output unit (playback means) 105, an imaging unit 106, and a display unit (display means) 107. The sound recording unit 104 collects sounds from multiple sound sources. For example, the sound recording unit 104 can capture sounds around the display device 101 and sounds such as the voice of the user himself as sound data. The sound data is output to a sound recording processing unit 108 of the information processing device 102. As a result, the sound data is input to the sound recording processing unit 108. Note that the sound recording unit 104 is not particularly limited, and examples thereof include a directional microphone and a lapel microphone. In addition, the number of sound recording units 104 arranged is one in the configuration shown in FIG. 1, but is not limited to this and may be multiple. The sound output unit 105 outputs the sound data input to the sound output instruction unit 109 via the sound recording processing unit 108 and the control unit 111 of the information processing device 102 as sound. As a result, sounds from each sound source can be played simultaneously or individually (playback process). The audio output unit 105 is not particularly limited, and may be, for example, a speaker, an earphone, a headphone, etc. The number of audio output units 105 arranged is one in the configuration shown in FIG. 1, but is not limited to this and may be more than one. The sound recording unit 104 and the audio output unit 105 may be composed of wireless earphones having a microphone function, etc. The imaging unit 106 captures an image of the real space as a still image or a video. This image of the real space (captured image) is output to the control unit 111 of the information processing device 102. The imaging unit 106 can capture an image of the real space in the line of sight from the user's eyes. In this case, the imaging unit 106 has a stereo camera composed of two cameras fixed at a distance from each other.

[0013] The display unit 107 can display an image of at least one of the real space and the virtual space (display step). In this embodiment, the display unit 107 displays an image of a mixed reality space (MR space) obtained by fusing an image of the real space captured by the imaging unit 106 with a virtual space rendered by computer graphics (CG), that is, a synthesized image. This image is visually recognized by the user. The display unit 107 may have two displays arranged corresponding to the left and right eyes of the user, respectively. In this case, an MR image for the left eye is displayed on the display corresponding to the left eye of the user, and an MR image for the right eye is displayed on the display corresponding to the right eye of the user. Note that the image displayed on the display unit 107 is not limited to an MR image, and may be, for example, a VR image or an AR image. In the information processing system 1000, the display device 101 having the above configuration can be applied to, for example, an HMD (head-mounted display device). In addition, devices to which the display device 101 can be applied are not limited to HMDs, but also include, for example, HHDs (handheld displays) that allow images to be observed by looking through them while held in the hand like binoculars, tablet terminals, smartphones, etc.

[0014] The operation device 103 is an operation device for video games that can obtain, for example, a value according to the tilt direction of a stick, a value indicating a pressed state of a button, and the like. The operation device 103 is not limited to an operation device for video games, and may be, for example, an operation device that is detachably attached to the user's torso or hand, or a ring-shaped (finger ring-shaped) operation device that is detachably attached to the user's finger. The information processing device 102 may be configured to be disposed within the display device 101, unlike the configuration shown in FIG. 1. The operation device 103 has an object selection instruction unit (selection means) 116, a vibration unit (vibration applying means) 117, and a position and orientation calculation unit 120. The object selection instruction unit 116 outputs a value according to a user's operation, such as a value according to the tilt direction of a stick or a value indicating a pressed state of a button, as operation information to the information processing device 102. The operation information includes information that a virtual object displayed on the display unit 107 together with an image of a mixed reality space has been selected by the object selection instruction unit 116. The virtual objects are associated with each sound source. The association between the virtual objects and the sound sources may be stored in advance in the data storage unit 113 of the information processing device 102, or may be appropriately performed on a setting screen (setting means) displayed on the display unit 107. The shape of the virtual object is not particularly limited, and may be, for example, an icon imitating a sound source. The object selection instruction unit 116 can select a virtual object based on the line of sight of the user recognized by the line of sight recognition unit (not shown). For example, if there is a virtual object in the line of sight of the user, the virtual object is selected. In addition, if the movement (gesture) of the user's hand is recognized from the image of the real space captured by the imaging unit 106, the object selection instruction unit 116 can select a virtual object based on the gesture. For example, if there is a gesture in which the user touches a virtual object in the image of the mixed reality space, the virtual object is selected. The vibration unit 117 applies vibration to the user according to a vibration instruction from the information processing device 102 (vibration instruction unit 114) (vibration application process). The vibrating section 117 is not particularly limited, and it is preferable to use a vibrating element having a relatively short response time, such as a piezoelectric element.This allows the user to get a sensation (vibration sensation) of small vibrations being transmitted from the vibration unit 117. Note that as the vibration unit 117, a linear vibrator (LRA), an eccentric motor (ERM), or the like can also be used.

[0015] The position and orientation calculation unit 120 calculates the position and orientation (hereinafter sometimes referred to as "position and orientation") of the operation device 103 in the world geodetic system (geographic coordinate system). Note that the position and orientation calculation unit 120 is not limited to calculating both the position and orientation of the operation device 103, and it is sufficient if it is capable of calculating at least one of the position and the orientation. The position and orientation calculation unit 120 outputs information regarding the position and orientation of the operation device 103 to the object selection processing unit 115 of the information processing device 102. Then, this position and orientation information of the operation device 103 is stored in the data storage unit 113 of the information processing device 102. The position and orientation calculation unit 120 is not particularly limited, and examples thereof include an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor. The position and orientation calculation unit 120 may have a plurality of such sensors. The position and orientation calculation unit 120 may have, for example, an imaging unit that captures an image and an imaging processing unit that processes the captured image captured by the imaging unit. The image capturing processing unit can calculate the position and orientation of the operation device 103 by extracting feature points of the captured image and executing SLAM (Simultaneous Localization and Mapping) processing on the feature points. Furthermore, the position and orientation calculation unit 120 may calculate the position and orientation of the operation device 103 in conjunction with an optical sensor installed in real space. Furthermore, the position and orientation calculation unit 120 may output values ​​(values ​​output from a sensor) for calculating the position and orientation of the operation device 103, images, and the like to the information processing device 102. In this case, the position and orientation calculation unit 121 of the information processing device 102 can calculate the position and orientation of the operation device 103.

[0016] The information processing device 102 includes a sound recording processing unit 108, a sound output instruction unit 109, an image synthesis unit 110, a control unit (control means) 111, an image generation unit 112, a data storage unit 113, a vibration instruction unit 114, an object selection processing unit 115, and a position and orientation calculation unit 121. The information processing device 102 is not particularly limited, and examples thereof include desktop and notebook personal computers, tablet terminals, and smartphones. The control unit 111 controls the entire information processing device 102. For example, the control unit 111 can perform control to display a virtual object together with an image on the display unit 107 (control step). In addition, the control unit 111 can control the position and orientation of a UI (graphics) displayed on the display unit 107 based on the position and orientation information of the operation device 103 stored in the data storage unit 113. The position of the UI is indicated by three-dimensional coordinate information according to a Cartesian coordinate system of three axes, for example, the X-axis, the Y-axis, and the Z-axis. Also, when the UI is a virtual ray emitted from the user's hand, the position of the UI is, for example, at least one of the start position and the end position of the ray. Also, the orientation of the UI corresponds to the orientation of the UI in a three-dimensional virtual space. When the UI is a ray, the orientation of the UI corresponds to, for example, the direction in which the ray extends. The control unit 111 acquires sound source data that the user is supposed to hear from the data storage unit 113, and synthesizes the sound source data as necessary. This synthesized sound source data is output to the sound output unit 105 of the display device 101 via the sound output instruction unit 109. This allows the sound output unit 105 to perform sound output based on the synthesized sound source data. Also, the control unit 111 may perform control according to a voice command when controlling the entire information processing device 102. In this case, the control unit 111 analyzes the voice data from the sound recording unit 104 acquired by the sound recording processing unit 108, and as a result, if it is a voice command, performs control according to the command content.

[0017] The position and orientation calculation unit 121 calculates the position and orientation of the imaging unit 106 in the world geodetic system. Specifically, the position and orientation calculation unit 121 extracts a marker assigned to the world geodetic system from an image of real space captured by the imaging unit 106. Then, the position and orientation calculation unit 121 calculates the position and orientation of the imaging unit 106 in the world geodetic system based on the position and orientation of the extracted marker. Information on the position and orientation of the imaging unit 106 is stored in the data storage unit 113. In addition, the position and orientation calculation unit 121 calculates the position and orientation of the operation device 103 in the world geodetic system based on the position and orientation information of the operation device 103 acquired from the position and orientation calculation unit 120 of the operation device 103 or the image of real space captured by the imaging unit 106. When the position and orientation of the operation device 103 is calculated based on the position and orientation information of the operation device 103 in the position and orientation calculation unit 120, a difference (error) may occur between the position and orientation of the operation device 103 calculated by the position and orientation calculation unit 120 and the actual position and orientation of the operation device 103. For example, in a method of calculating the position and orientation using a combination of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor, errors of the sensors are accumulated. As a result, an error occurs with respect to the actual position and orientation. In addition, in this method, it may be difficult to calculate the position and orientation of the operation device 103. In addition, in a method of calculating the position and orientation using an optical sensor installed in a real space, the optical sensor may be blocked by another real object. In this case, an error may occur with respect to the actual position and orientation. Or, even in this method, it may be difficult to calculate the position and orientation. In such a case, the position and orientation calculation unit 121 can extract a marker attached to the operation device 103 from an image of the real space captured by the imaging unit 106, and thereby calculate the position and orientation of the operation device 103 with high accuracy based on the position and orientation of the marker. At this time, the position and orientation calculation unit 121 may use all or part of the calculation result of the position and orientation acquired from the position and orientation calculation unit 120 of the operation device 103. Note that the calculation method of the position and orientation calculation unit 121 is not limited to a method using a marker, and may be, for example, a method using SLAM processing. The position and orientation calculation unit 121 stores information on the position and orientation of the operation device 103 in the data storage unit 113 via the control unit 111 .

[0018] When the operation device 103 is not captured in the image of the real space captured by the imaging unit 106, the position and orientation calculation unit 121 can calculate the position and orientation of the operation device 103 based on the value acquired from the position and orientation calculation unit 120 of the operation device 103. The position and orientation calculation unit 121 may calculate the position and orientation of the operation device 103 based on the value acquired from the position and orientation calculation unit 120 of the operation device 103 and other information. For example, when the operation device 103 is not captured in the image of the real space, if the operation device 103 has an acceleration sensor or the like, the position and orientation calculation unit 121 may calculate the position and orientation of the operation device 103 based on the detection result of the acceleration sensor or the like. For example, the position and orientation calculation unit 121 adds the position of the operation device 103 at a past time when the operation device 103 was captured in the image of the real space and the movement amount of the operation device 103 from that time calculated from the acceleration sensor. The position and orientation calculation unit 121 may calculate the current position and orientation of the operation device 103 based on the result of this addition. When the operation device 103 is not captured in an image in real space, the position and orientation calculation section 121 may calculate only the orientation of the operation device 103, and calculation of the position and orientation of the operation device 103 may be difficult.

[0019] When a virtual object displayed on the display unit 107 is selected by the object selection instruction unit 116 of the operation device 103 (selection step), the object selection processing unit 115 reads out sound source data associated with the virtual object stored in the data storage unit 113. At this time, the vibration instruction unit 114 instructs the vibration unit 117 to apply vibration to the user in accordance with the strength of the sound emitted by the sound source data. This provides the user with a vibration sensation that matches the strength of the sound. In the information processing system 1000, when a plurality of virtual objects are displayed on the display unit 107, it is possible to select one of the plurality of virtual objects, and at least two of them (selection as a virtual object group) can also be selected. In this case, if appropriate sound source data associated with the virtual object group is stored in the data storage unit 113, the vibration unit 117 applies vibration to the user in accordance with the strength of the sound of the sound source data. Furthermore, if appropriate sound source data associated with the virtual object group is not stored in the data storage unit 113, a synthetic sound obtained by synthesizing sounds from sound sources associated with each virtual object included in the virtual object group is generated by the control unit 111. The vibration unit 117 applies vibration to the user according to the strength of the synthetic sound. As described above, the selection of a virtual object is possible based on the user's line of sight. In this case, for example, if the display area displayed on the display unit 107 is divided into a plurality of areas, and one of the areas is in the line of sight of the user, the virtual objects included in the one area can be selected collectively.

[0020] The image generating unit 112 acquires the data of the virtual space stored in the data storage unit 113 via the control unit 111. The data of the virtual space includes data related to each virtual object constituting the virtual space, data related to a light source that illuminates the virtual space, and the like. The image generating unit 112 generates an image of the virtual space based on the data of the virtual space. The image generating unit 112 also acquires information on the position and orientation of the imaging unit 106 calculated by the position and orientation calculation unit 121 from the data storage unit 113 via the control unit 111. The image generating unit 112 also acquires information on the position and orientation of the UI controlled by the control unit 111 from the data storage unit 113 via the control unit 111. The image generating unit 112 generates an image of the virtual space according to the position and orientation of the imaging unit 106. Note that a technique for generating an image of the virtual space according to the position and orientation of the imaging unit 106 is a well-known technique, and therefore a detailed description thereof will be omitted here. The image synthesis unit 110 synthesizes an image of the virtual space generated by the image generation unit 112 with an image of the real space captured by the imaging unit 106 to generate an MR image. In this MR image, the image of the virtual space generated by the image generation unit 112 may be an image representing the entire virtual space, or an image representing a part of the virtual space. In addition, the image synthesis unit 110 may generate an MR image by performing affine transformation on the image, or may generate an MR image by allocating the image to a parametric curved surface. Then, the image synthesis unit 110 outputs the MR image to the display unit 107. As described above, the data storage unit 113 stores various information. The data storage unit 113 includes a RAM, a HDD, and the like. In addition to various information, the data storage unit 113 also stores a program. The program includes a program for causing a computer to execute each unit and each means (a control method of the information processing system) of the information processing system 1000. The program is not limited to being stored in the information processing device 102 (data storage unit 113), but may be stored in a distributed manner in the devices from the display device 101 to the operation device 103, for example.

[0021] Fig. 2 is a flowchart showing the processing executed by the information processing device of the information processing system. Fig. 3 is a diagram showing a screen for switching on / off the application of vibration according to the intensity of a sound. Fig. 4 is a diagram for explaining on / off the selection of a virtual object. As shown in Fig. 2, in step S201, control unit 111 initializes the selection information of the virtual object and the user's line of sight information (area of ​​interest information) stored in data storage unit 113. This updates these pieces of information to their initial states. After step S201 is executed, the process proceeds to step S202.

[0022] In step S202, the control unit 111 judges whether or not the vibration unit 117 applies vibration according to the strength of the sound. Thus, in this embodiment, the control unit 111 has a function as a judgment unit that judges whether or not vibration according to the strength of the sound is applied. Note that in the information processing device 102, a part that functions as a judgment unit may be provided separately from the control unit 111. The judgment in step S202 is made based on the display state (switching result) of the screen 300 shown in FIG. 3. The screen 300 is a screen displayed on the display unit 107, and serves as a switching unit that switches whether or not vibration according to the strength of the sound is applied by the vibration unit 117, that is, a setting unit that sets voice emphasis by vibration. Hereinafter, applying vibration according to the strength of the sound by the vibration unit 117 may be referred to as "applying vibration with strength of sound". The screen 300 includes a check box 301 and a check box 302. Checking the check box 301 switches to applying vibration with strength of sound. Checking the check box 302 switches to not applying vibration with strength of sound. Then, if it is determined in step S202 that vibration is to be applied, that is, if the check box 301 is checked, the process proceeds to step S203. On the other hand, if it is determined in step S202 that vibration is not to be applied, that is, if the check box 302 is checked, the process proceeds to step S206. Note that the determination in step S202 is not limited to being made based on the screen 300 shown in FIG. 3, and for example, the information processing system 1000 may set a default value to apply sound volume vibration or not apply sound volume vibration. For example, if the default value is set to apply sound volume vibration, changing the default value can be omitted if sound volume vibration is to be applied as it is. On the other hand, if the setting is made to not apply sound volume vibration, the default value is changed to that effect. Note that the default value is stored in advance in the data storage unit 113, and is read by the control unit 111 and used in the determination in step S202.

[0023] In step S203, the control unit 111 judges whether the operation device 103 is valid. This judgment is made based on whether the operation device 103 is used by the user or not. In addition, in the case where the operation device 103 is a device that is detachably attached to the user, the judgment in step S203 is made based on whether the operation device 103 is attached to the user or not. In either case, a sensor that detects the user's contact with the operation device 103, a sensor that detects a change in the position and orientation of the operation device 103, or the like can be used for such judgment. In addition, information required for the judgment in step S203 may be stored in the data storage unit 113 in advance, and may be read by the control unit 111 at the judgment timing.

[0024] In step S204, the control unit 111 judges whether or not the selection information of the virtual object (information about the virtual object) stored in the data storage unit 113 has changed as a result of the user selecting the virtual object. Thus, in this embodiment, the control unit 111 has a function as a detection unit that detects whether or not the selection information of the virtual object has changed. Note that in the information processing device 102, a part having a function as a detection unit may be provided separately from the control unit 111. The judgment in step S204 is possible by comparing IDs indicating the virtual objects. For example, as shown in FIG. 4(A), it is assumed that there are OBJ1 selected previously and OBJ2 selected this time as virtual objects. If the ID of OBJ1 and the ID of OBJ2 are the same, it is judged that the selection information of the virtual object has not changed. On the other hand, if the ID of OBJ1 and the ID of OBJ2 are different, it is judged that the selection information of the virtual object has changed. Also, as shown in FIG. 4(B), it is assumed that there are OBJ1 and OBJ3 selected previously and OBJ2 selected this time as virtual objects. In this case, the number of virtual objects selected previously was two, whereas the number of virtual objects selected currently is one, so it is determined that the selection information of the virtual objects has changed. Also, as shown in FIG. 4C, it is assumed that the virtual objects are OBJ1, OBJ2, and OBJ3 selected previously, and OBJ1 and OBJ2 selected currently. In this case, OBJ3 is included in the previously selected virtual object, but is not included in the currently selected virtual object, so it is determined that the selection information of the virtual objects has changed. Then, if it is determined that the selection information of the virtual objects has changed as a result of the determination in step S204 (the result of detection by the detection means), the process proceeds to step S205. On the other hand, if it is determined that the selection information of the virtual objects has not changed as a result of the determination in step S204, the process returns to step S202, and the subsequent steps are executed in sequence.

[0025] In step S205, the control unit 111 applies sound volume vibration. Specifically, the control unit 111 reads out sound volume data associated with the currently selected virtual object from among the multiple sound volume data from the data storage unit 113, and extracts volume volume information from the sound volume data. Then, the control unit 111 transmits a vibration instruction to the vibration unit 117 via the vibration instruction unit 114. This vibration instruction is an instruction to apply vibration to the user based on the extracted volume information, that is, to apply sound volume vibration. The vibration unit 117 can apply sound volume vibration according to the vibration instruction. This allows the user to feel the vibration from the vibration unit 117 being transmitted in small increments. In addition, the control unit 111 transmits the sound volume data associated with the currently selected virtual object to the audio output unit 105 via the audio output instruction unit 109. As a result, while the vibration unit 117 applies sound intensity vibration, the audio output unit 105 can output sound based on the sound source data transmitted via the audio output instruction unit 109, that is, can play sound. Then, by combining the application of sound intensity vibration and audio output, even when a plurality of sound sources are present in a direction of interest that is of interest to the user, the sound from each sound source is left as it is, and the sound from a specific sound source (a sound source associated with a virtual object) is emphasized. This allows the sound from a specific sound source to be distinguished. In addition, the audio output unit 105 can adjust the volume of the sound source associated with the virtual object. This allows the sound from a specific sound source to be better distinguished.

[0026] In addition, it is preferable that the control unit 111 synchronizes the application of the loudness vibration and the audio output to prevent a timing discrepancy between the two. If the timing is discrepant, the user may feel a sense of incongruity between the sense of vibration and the sense of hearing the sound. For example, it is assumed that the vibration timing of the vibration unit 117 is delayed by D [msec] from the audio output timing of the audio output unit 105. In this case, the control unit 111 can synchronize the timing of the application of the loudness vibration and the audio output by issuing an instruction to apply the loudness vibration and waiting for D [msec] before issuing an instruction to output the audio. This can eliminate the sense of incongruity felt by the user. Note that the control unit 111 can also synchronize the timing of the application of the loudness vibration and the audio output by issuing an instruction to apply the loudness vibration D [msec] before issuing an instruction to output the audio. Note that it is preferable that the vibration unit 117 applies vibration while in contact with the user's finger (fingertip) or the like. It is known that fingers have a relatively sharp sense, and are a preferable part to be vibration-applied. Furthermore, the vibration section 117 may be one that applies vibrations to the user at multiple points. Furthermore, the vibration section 117 may be one that generates air conduction sound or bone conduction sound.

[0027] In step S206, the control unit 111 determines whether the vibration unit 117 of the operation device 103 is operating and vibrating. If it is determined in step S206 that the operation device is vibrating, the process proceeds to step S207. On the other hand, if it is determined in step S206 that the operation device is not vibrating, the process ends.

[0028] In step S207, the control unit 111 stops the operation of the vibration unit 117 of the operation device 103, that is, stops the vibration. After step S207 is executed, the process ends.

[0029] <Second embodiment> Hereinafter, the second embodiment will be described with reference to FIG. 5 to FIG. 7, but the differences from the above-mentioned embodiment will be mainly described, and the description of the similar matters will be omitted. FIG. 5 is a flowchart showing the process executed by the information processing device of the information processing system according to the second embodiment. FIG. 6 and FIG. 7 are diagrams for explaining the region of interest, respectively. The process of steps S501 to S507 in the flowchart shown in FIG. 5 is the same as the process of steps S201 to S207 in the flowchart shown in FIG. 2. In addition, the process of steps S508 to S510 after the determination in step S504 in the flowchart shown in FIG. 5 is different from the flowchart shown in FIG. 2. As shown in FIG. 5, in step S504, the control unit 111 determines whether or not the selection information of the virtual object stored in the data storage unit 113 has changed as a result of the user selecting a virtual object. If it is determined that the selection information of the virtual object has changed as a result of the determination in step S504, the process proceeds to step S505. On the other hand, if it is determined that the selection information of the virtual object has not changed as a result of the determination in step S504, the process proceeds to step S508.

[0030] In step S508, the control unit 111 determines whether the vibration unit 117 of the operation device 103 is operating in accordance with the output from the sound source associated with the virtual object and is vibrating. Here, when the vibration unit 117 is vibrated in step S505, the control unit 111 sets a flag and stores the flag information in the data storage unit 113. The flag information is updated by dropping the flag when it is no longer necessary to vibrate the vibration unit 117 in accordance with the output from the sound source associated with the virtual object. In step S508, the control unit 111 reads the flag information from the data storage unit 113 and can determine whether the vibration is occurring based on whether the flag is set. Then, if it is determined that the vibration is occurring as a result of the determination in step S508, the process returns to step S502, and the subsequent steps are executed in sequence. On the other hand, if it is determined that the vibration is not occurring as a result of the determination in step S508, the process proceeds to step S509.

[0031] In step S509, the control unit 111 judges whether the user's area of ​​interest has changed. Specifically, the control unit 111 receives gaze information from a gaze detection unit (not shown) of the display device 101 at a predetermined interval, and identifies an area of ​​interest that is of interest to the user based on the gaze information. For example, as shown in FIG. 6(A), it is assumed that there are a coordinate range 1 that was of interest last time and a coordinate range 2 that is of interest this time as areas of interest. Information on such areas of interest is stored in the data storage unit 113. Note that the coordinate range may be a range of a polar coordinate system centered on the user, or a range of an orthogonal coordinate system. Also, as shown in FIG. 6(B), it is assumed that there are an area option 1 that was of interest last time and an area option 2 that is of interest this time as areas of interest. Each area option is one area option among a plurality of area options determined in advance. For example, as shown in FIG. 7, when an area displayed on the display unit 107 of the display device 101 is divided into five areas, namely, upper, lower, left, right, and center, these five areas are predetermined area options. The control unit 111 can read information about the region of interest from the data storage unit 113 and determine whether there is a change between the previous region of interest and the current region of interest. If the result of the determination in step S509 is that the user's region of interest has changed, that is, that there is a change, the process proceeds to step S510. On the other hand, if the result of the determination in step S509 is that the user's region of interest has not changed, that is, that there is no change, the process returns to step S502, and the subsequent steps are executed in sequence.

[0032] In step S510, the control unit 111 identifies a group of virtual objects included in the current region of interest, and performs the same process as in step S505, that is, the addition of sound intensity vibration, for each virtual object in the group of virtual objects. After performing step S510, the process returns to step S502, and the subsequent steps are sequentially performed.

[0033] As described above, in this embodiment, the information processing device 102 is configured to be capable of executing a process of determining whether a virtual object has been selected and a process of determining whether a region of interest has changed. This allows, for example, more detailed application of sound intensity vibration in response to a change in the user's line of sight.

[0034] <Third embodiment> Hereinafter, the third embodiment will be described with reference to FIG. 8, but the differences from the above-mentioned embodiment will be mainly described, and the description of the similar matters will be omitted. FIG. 8 is a flowchart showing the process executed by the information processing device of the information processing system according to the third embodiment. As shown in FIG. 8, in step S801, the control unit 111 acquires the volume α collected by the sound recording unit 104 through the sound recording processing unit 108, for example, in a state where sound volume vibration is applied. Then, the control unit 111 judges whether the volume α is equal to or greater than the threshold value α0. The threshold value α0 is stored in the data storage unit 113 in advance. Moreover, it is preferable that the threshold value α0 can be changed as appropriate. Then, if it is judged that the volume α is equal to or greater than the threshold value α0 as a result of the judgment in step S801, the process proceeds to step S802. On the other hand, if it is judged that the volume α is not equal to or greater than the threshold value α0 as a result of the judgment in step S801, the process ends.

[0035] In step S802, the control unit 111 determines to stop applying the loudness vibration, and issues an instruction to that effect to the vibration unit 117 via the vibration instruction unit 114. This allows the vibration unit 117 to stop applying the loudness vibration in accordance with the instruction to stop applying the loudness vibration. Furthermore, the control unit 111 stores in the data storage unit 113 which sound source data the vibration was synchronized with, in preparation for resuming the application of the loudness vibration. After executing step S802, the process proceeds to step S803.

[0036] In step S803, when a predetermined time has elapsed after execution of step S802, the control unit 111 acquires the volume α collected by the recording unit 104 via the recording processing unit 108. Then, the control unit 111 judges whether or not the volume α is less than the threshold value α0. If it is determined in step S803 that the volume α is less than the threshold value α0, the process proceeds to step S804. On the other hand, if it is determined in step S803 that the volume α is not less than the threshold value α0, the process ends.

[0037] In step S804, the control unit 111 determines whether to resume the application of the volume vibration, and issues an instruction to that effect to the vibration unit 117 via the vibration instruction unit 114. This allows the vibration unit 117 to apply volume vibration according to the volume of the sound acquired in step S803 in accordance with the instruction to resume the application of the volume vibration. After executing step S804, the process ends.

[0038] As described above, in this embodiment, the information processing device 102 is configured to be able to determine whether or not to apply loudness vibration based on the volume of the sound collected by the recording unit 104. This allows, for example, the application of loudness vibration and its stop to be repeated. It is preferable that the determination of the volume α in steps S801 and S803 is performed when the volume α is continuously acquired for a predetermined time. It is also preferable that the threshold value α0 in step S801 and the threshold value α0 in step S803 are the same, but they may be different. In addition, when the reproduction of the sound is completed after the determination in step S804, the application of loudness vibration may be stopped. In this case, the sound source data stored in the data storage unit 113 in step S802 may be erased.

[0039] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors of a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (for example, an ASIC) for realizing one or more functions. In addition, the operation device 103 has both the object selection instruction unit 116 and the vibration unit 117 in the configuration shown in FIG. 1, but is not limited thereto, and may have, for example, one of the object selection instruction unit 116 and the vibration unit 117. In this case, the other can be arranged in a device different from the operation device 103 (for example, the display device 101).

[0040] In the information processing system 1000, the information processing device 102 may be a server. In this case, the information processing system 1000 may be in a state in which, for example, the server (information processing device 102) exists outside Japan, and the terminal device composed of the display device 101 and the operation device 103 exists inside Japan. Even in this case, each file or data is transmitted from the server to the terminal device, and the terminal device can receive each file or data. Even if the server exists outside Japan in this way, the transmission and reception (transmission and reception) of files and data in the information processing system 1000 is performed as a whole, that is, without a separate operation by the user of the terminal device. Since the system functions by a terminal device in Japan receiving each file or data, the transmission and reception can be considered to have been performed inside Japan. Also, in the information processing system 1000, for example, even if the server exists outside Japan and the terminal device exists inside Japan, the terminal device can perform the main function of the information processing system 1000 (the function of applying vibration according to the strength of sound). In addition, the effect of this function (the effect of being able to distinguish a sound from a specific sound source while leaving the sounds from each sound source as they are even when multiple sound sources are present in the direction of interest of the user) can be realized within Japan. For example, even if the server is located outside of Japan, if the terminal device constituting the information processing system 1000 is located within Japan, the information processing system 1000 can be used within Japan using the terminal device. And the use of the information processing system 1000 can affect the economic benefits of, for example, the patentee.

[0041] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A display means capable of displaying an image of at least one of a real space and a virtual space; a playback means for playing back sounds from a plurality of sound sources to a user viewing the image displayed on the display means; a vibration applying means for applying vibration to the user; a control means for controlling the display means to display a virtual object that can be linked to each of the sound sources together with the image; a selection means for selecting the virtual object displayed on the display means, The information processing system is characterized in that, when the virtual object is selected by the selection means, the vibration applying means applies vibration to the user in accordance with the strength of a sound from a sound source linked to the virtual object among the multiple sound sources. (Configuration 2) The information processing system according to configuration 1, wherein the playback means plays back a sound from a sound source associated with the virtual object while applying vibrations in accordance with the intensity of the sound from the sound source using the vibration applying means. (Configuration 3) The information processing system according to configuration 2, wherein the playback means is capable of adjusting a volume of a sound source associated with the virtual object. (Configuration 4) The information processing system according to any one of configurations 1 to 3, wherein the selection means is capable of selecting the virtual object based on a line of sight of the user. (Configuration 5) A detection unit detects a change in information about the virtual object when the virtual object is selected by the selection unit, The information processing system according to any one of configurations 1 to 4, characterized in that when the change is detected as a result of detection by the detection means, the vibration applying means applies vibrations that correspond to the intensity of the sound. (Configuration 6) The control means is capable of controlling the display means to display a plurality of the virtual objects, 6. The information processing system according to any one of configurations 1 to 5, wherein the selection means is capable of selecting at least two of the plurality of virtual objects. (Configuration 7) The information processing system according to Configuration 6, wherein when the at least two virtual objects are selected by the selection means, the vibration applying means generates a synthetic sound by synthesizing sounds from sound sources linked to the respective virtual objects, and applies vibration to the user in accordance with the strength of the synthetic sound. (Configuration 8) The information processing system according to any one of configurations 1 to 7, further comprising a determination unit that determines whether or not the vibration applying unit should apply vibration in accordance with the intensity of the sound. (Configuration 9) A switching means is provided for switching whether or not the vibration is applied by the vibration applying means in accordance with the strength of the sound, The information processing system according to configuration 8, wherein the determination means determines whether or not to cause the vibration applying means to apply vibration in accordance with the intensity of the sound, based on a switching result of the switching means. (Configuration 10) A sound collecting means for collecting sounds from the plurality of sound sources is provided, The information processing system according to configuration 8, wherein the determination means determines whether or not to cause the vibration applying means to apply vibration in accordance with the volume of the sound collected by the sound collecting means. (Configuration 11) The information processing system described in Configuration 10, characterized in that when the vibration imparting means is imparting vibrations in accordance with the loudness of the sound and the volume of the sound collected by the sound collecting means is equal to or greater than a threshold value, the judgment means determines to stop imparting vibrations in accordance with the loudness of the sound by the vibration imparting means. (Configuration 12) The information processing system described in Configuration 11, characterized in that the judgment means determines to resume applying vibrations in accordance with the loudness of the sound by the vibration imparting means when the volume of the sound collected by the sound collecting means is less than a threshold value while the vibration imparting means has stopped applying vibrations in accordance with the loudness of the sound. (Configuration 13) The information processing system according to any one of configurations 1 to 12, wherein the vibration applying means applies vibration while in contact with the user's finger. (Configuration 14) A first device having the display means and the playback means; A second device configured separately from the first device and having the vibration applying means and the selection means; 14. The information processing system according to any one of configurations 1 to 13, further comprising: a third device configured separately from the first device and the second device, the third device having the control means. (Method 1) A method for controlling an information processing system that processes information, comprising the steps of: a display step capable of displaying an image of at least one of a real space and a virtual space; a reproduction step of reproducing sounds from a plurality of sound sources to a user viewing the image displayed in the display step; a vibration applying step of applying vibration to the user; a control step of controlling the display step to display a virtual object that can be associated with each of the sound sources together with the image; a selection step of selecting the virtual object displayed in the display step, a vibration applying step of applying vibration to the user in accordance with the strength of a sound from a sound source associated with the virtual object among the plurality of sound sources when the virtual object is selected in the selection step, the vibration applying step being performed in accordance with the strength of a sound from the sound source associated with the virtual object among the plurality of sound sources. (Program 1) A program for causing a computer to execute each of the means of the information processing system according to any one of configurations 1 to 14. [Explanation of symbols]

[0042] 104 Recording Section 105 Audio output section 107 Display section 111 Control section 116 Object selection indicator 117 Vibration section 1000 Information Processing Systems

Claims

1. A display means capable of displaying an image of at least one of a real space and a virtual space; a playback means for playing back sounds from a plurality of sound sources to a user viewing the image displayed on the display means; a vibration applying means for applying vibration to the user; a control means for controlling the display means to display a virtual object that can be linked to each of the sound sources together with the image; a selection means for selecting the virtual object displayed on the display means, The information processing system is characterized in that, when the virtual object is selected by the selection means, the vibration applying means applies vibration to the user in accordance with the strength of a sound from a sound source linked to the virtual object among the multiple sound sources.

2. The information processing system according to claim 1 , wherein the reproduction means reproduces the sound from the sound source associated with the virtual object while imparting vibrations in accordance with the intensity of the sound from the sound source using the vibration imparting means.

3. The information processing system according to claim 2 , wherein the playback means is capable of adjusting a volume of a sound source associated with the virtual object.

4. 2. The information processing system according to claim 1, wherein the selection means is capable of selecting the virtual object based on the user's line of sight.

5. a detection means for detecting a change in information about the virtual object when the virtual object is selected by the selection means, 2. The information processing system according to claim 1, wherein when the change is detected as a result of detection by the detection means, the vibration applying means applies vibration in accordance with the intensity of the sound.

6. the control means is capable of controlling the display means to display a plurality of the virtual objects, 2. The information processing system according to claim 1, wherein the selection means is capable of selecting at least two of the plurality of virtual objects.

7. The information processing system according to claim 6, characterized in that, when the at least two virtual objects are selected by the selection means, a synthetic sound is generated by synthesizing sounds from sound sources linked to the respective virtual objects, and vibration is applied to the user in accordance with the strength of the synthetic sound.

8. 2. The information processing system according to claim 1, further comprising a determining unit for determining whether or not said vibration applying unit should apply vibration in accordance with the intensity of said sound.

9. A switching means is provided for switching whether or not the vibration is applied by the vibration applying means in accordance with the intensity of the sound, 9. The information processing system according to claim 8, wherein the determining means determines whether or not the vibration applying means applies vibration in accordance with the intensity of the sound based on a switching result by the switching means.

10. A sound collecting means for collecting sounds from the plurality of sound sources is provided, The information processing system according to claim 8, characterized in that the determination means determines whether or not to cause the vibration applying means to apply vibration in accordance with the intensity of the sound, based on the volume of the sound collected by the sound collecting means.

11. The information processing system according to claim 10, characterized in that the judgment means determines to stop the vibration imparting means from imparting vibrations in accordance with the volume of the sound when the volume of the sound collected by the sound collecting means is equal to or greater than a threshold value while the vibration imparting means is imparting vibrations in accordance with the volume of the sound.

12. The information processing system according to claim 11, characterized in that the judgment means judges to resume applying vibrations in accordance with the volume of the sound by the vibration applying means when the volume of the sound collected by the sound collecting means is less than a threshold value while the vibration applying means has stopped applying vibrations in accordance with the volume of the sound.

13. 2. The information processing system according to claim 1, wherein the vibration applying means applies vibration to the user's finger while in contact with the finger.

14. a first device having the display means and the playback means; A second device configured separately from the first device and having the vibration applying means and the selection means; 2. The information processing system according to claim 1, further comprising: a third device configured separately from said first device and said second device, said third device having said control means.

15. 1. A method for controlling an information processing system for processing information, comprising: a display step capable of displaying an image of at least one of a real space and a virtual space; a reproduction step of reproducing sounds from a plurality of sound sources to a user viewing the image displayed in the display step; a vibration applying step of applying vibration to the user; a control step of controlling the display step to display a virtual object that can be associated with each of the sound sources together with the image; a selection step of selecting the virtual object displayed in the display step, a vibration applying step of applying vibration to the user in accordance with the strength of a sound from a sound source associated with the virtual object among the plurality of sound sources when the virtual object is selected in the selection step, the vibration applying step being performed in accordance with the strength of a sound from the sound source associated with the virtual object among the plurality of sound sources.

16. 2. A program for causing a computer to execute each of the means of the information processing system according to claim 1.

Images