Information processing apparatus, information processing system, and information processing method

Abstract

An information processing apparatus includes a control unit that generates a vibration stimulation signal to be provided to a user based on sound signals in content, wherein the control unit is configured to: acquire data of the content, the content including sound signals; perform analysis processing on the sound signals; and generate a vibration stimulation signal to be provided to the user in a conversion processing of the sound signals based on the results of the analysis processing.

Description

Technical Field

[0001] The embodiments discussed herein relate to information processing apparatus, information processing system, and information processing method. Background Technology

[0002] Traditionally, a technology is known to provide cross-reality (XR) content to remote users using head-mounted displays (HMDs), etc., which is a live experience and includes images and sounds recorded live at locations such as event venues.

[0003] It is worth noting that XR is a comprehensive expression representing multiple virtual space technologies, including Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Alternate Reality (SR), Audio / Visual (AV), etc.

[0004] In addition, a technique is known for driving a vibration application unit such as an exciter installed in a chair during the reproduction of the aforementioned XR content, thereby allowing the user to virtually experience the reproduced images and vibrations and / or impacts corresponding to the sounds (see, for example, Japanese Patent Publication No. 2007-324829).

[0005] However, there is room for improvement in the aforementioned traditional techniques for enhancing the sense of presence through vibration stimulation during content reproduction.

[0006] For example, when recording content from the external environment, a high-pass filter (HPF) is typically used to cut off the low-frequency range to remove noise such as footsteps and wind noise. Therefore, the low-frequency range of sounds recorded from the external environment is insufficient, making it difficult for users to experience a sense of presence even when vibrations are generated based on these sounds.

[0007] The objects providing vibrations are, for example, chairs and users. In the case of chairs, they are made of different materials or have different types; in the case of users, they have different body shapes. Therefore, for the same vibrational stimulus, the characteristics are often different. Consequently, there is a problem that the expected vibration is not delivered, thus preventing the user from experiencing a sense of presence.

[0008] In view of the above, one aspect of the embodiments is made, and the purpose of the embodiments is to provide information processing apparatus, information processing system and information processing method that can further enhance the sense of presence during content reproduction through vibration stimulation.

[0009] According to one aspect of the embodiment, the sense of presence can be further enhanced by vibration stimulation during content reproduction. Summary of the Invention

[0010] An information processing apparatus according to one aspect of an embodiment includes a control unit that generates a vibration stimulation signal to be provided to a user based on an audio signal in content, wherein the control unit is configured to: acquire data of the content, the content including the audio signal; perform analysis processing on the audio signal; and generate the vibration stimulation signal to be provided to the user in a conversion processing of the audio signal based on the result of the analysis processing. Attached Figure Description

[0011] This disclosure and its many accompanying advantages will become more readily understood by referring to the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 This is a schematic diagram illustrating an information processing method according to an embodiment;

[0013] Figure 2 This is a diagram illustrating an example of the structure of an information processing system according to the first embodiment;

[0014] Figure 3 This is a diagram illustrating an example of the structure of a field device according to the first embodiment;

[0015] Figure 4 This is a diagram illustrating an example of the structure of a remote device according to the first embodiment;

[0016] Figure 5 This is a diagram showing an example of the structure of a vibration output unit;

[0017] Figure 6 This is a block diagram illustrating a remote device according to a first embodiment;

[0018] Figure 7 This is a block diagram showing the sound / vibration conversion processing unit;

[0019] Figures 8 to 10 This is a supplementary diagram illustrating the sound signal conversion processing according to the first embodiment;

[0020] Figure 11 and Figure 12 This is a flowchart illustrating the processing steps performed by a remote device according to the first embodiment;

[0021] Figure 13 This is a block diagram illustrating a remote device according to a second embodiment;

[0022] Figures 14 to 16 This is a supplementary diagram illustrating the sound signal conversion processing according to the second embodiment;

[0023] Figure 17 This is a block diagram illustrating a remote device according to a third embodiment; and

[0024] Figure 18 This is a flowchart illustrating the processing steps performed by a remote device according to a third embodiment. Specific Implementation

[0025] In the following, embodiments of the information processing apparatus, information processing system, and information processing method will be described in detail with reference to the accompanying drawings. Furthermore, the disclosed technology is not limited to the embodiments described below.

[0026] In the following text, multiple structural elements having substantially the same functional structure may be distinguished from each other by providing different numbers with hyphens after the same reference numerals. For example, multiple structures having substantially the same functional structure may be distinguished as needed by referring to, for example, remote device 100-1 and remote device 100-2. When it is not necessary to distinguish multiple structural elements having substantially the same functional structure, only the same reference numerals are provided. For example, when it is not necessary to distinguish remote device 100-1 and remote device 100-2, they may be simply referred to as remote device 100.

[0027] Reference Figure 1 This section provides an overview of the information processing method according to an embodiment. Figure 1 This is a diagram schematically illustrating an information processing method according to an embodiment.

[0028] The information processing system 1 according to the embodiment is a system that provides live experiential XR content, including live images and sound, to a remote location other than the event venue, such as an exhibition venue, concert venue, fireworks display venue, or e-sports competition venue. Note that XR content corresponds to an example of "content".

[0029] like Figure 1 As shown, the information processing system 1 includes a field device 10 and at least one remote device 100. The field device 10 and the at least one remote device 100 are configured to communicate with each other via a network N (e.g., the Internet).

[0030] Figure 1 The example shown illustrates the following scenario: Field device 10 broadcasts XR content live to at least one remote device 100 located at various locations, where the XR content includes images and sound of an event being held at event site 1000 in the Kansai region.

[0031] Figure 1 The example shown illustrates a scenario where remote device 100-1 presents XR content transmitted from field device 10 to user U1 in the Kanto region via HMD.

[0032] HMD is an information processing terminal used to present XR content to user U1 and enable user U1 to enjoy the XR experience. HMD is a wearable computer worn on the head of user U1. Figure 1 The example shown is a goggles type. Note that HMDs can be either glasses type or hat type.

[0033] The HMD includes an image output unit 110 and an audio output unit 120. The image output unit 110 displays images included in XR content provided from the field device 10. Figure 1 In the example shown, the HMD's image output unit 110 is configured to be positioned in front of the user U1.

[0034] The sound output unit 120 outputs sound included in the XR content provided from the field device 10. Figure 1 In the example shown, the HMD's sound output unit 120 is configured, for example, as an earphone and attached to the ear of the user U1.

[0035] Figure 1 The example shown illustrates a scenario where remote device 100-2 presents XR content transmitted from field device 10 to user U2 in the Kyushu region via satellite dome D.

[0036] The satellite dome D is an audiovisual device for XR content, shaped like a dome, and includes an image output unit 110 and an audio output unit 120. Figure 1 In the example shown, the image output unit 110 of the satellite dome D is arranged on the wall. The image output unit 110 is implemented, for example, by arranging a thin liquid crystal display or an organic electroluminescent (EL) display on the wall, or by using a projector to project images onto the wall. The audio output unit 120 of the satellite dome D is arranged near the head position of the seated user U2.

[0037] Despite Figure 1 Its illustration is omitted, but the vibration output unit 130 is arranged near each of users U1 and U2 (see Figure 130). Figure 4 (or later). The vibration output unit 130 outputs vibrations corresponding to the sounds included in the XR content, thereby providing vibrational stimulation to each of the users U1 and U2. The vibration output unit 130 is implemented by a vibration application unit such as an exciter and is arranged in the chair where each of the users U1 and U2 is sitting, or attached to each of the users U1 and U2.

[0038] Incidentally, when recording sounds in the external environment, the low-frequency range is usually truncated by the HPF to remove noise such as footsteps and wind noise. However, if the sound signal that has been passed through the aforementioned HPF is input to the vibration output unit 130 to provide vibration stimulation to each of users U1 and U2, the low-frequency range is insufficient, making it difficult for users U1 and U2 to obtain a sense of presence.

[0039] Therefore, for example, an audio signal can be input to the vibration output unit 130 via an equalizer configured to increase the low-frequency range while employing known techniques. On the other hand, when increasing the low-frequency range by using an equalizer, there is a problem that untruncated residual low-frequency noise also increases, and using an equalizer alone cannot increase the already truncated frequency range sufficiently to help improve the sense of presence.

[0040] The objects providing vibrations are, for example, chairs and users. In the case of chairs, they are made of different materials or have different types; in the case of users, they have different body shapes. Therefore, for the same vibrational stimulus, the characteristics are often different. Consequently, there is a problem that the expected vibration is not delivered, thus preventing the user from experiencing a sense of presence.

[0041] In this respect, when using known techniques, adjustments can be made, for example, based on the user's sensation or by referencing actual measurements from an accelerometer, to approximate the vibration desired by the stimulus provider. However, when adjustments are made based on the user's sensation, it is difficult to reproduce the vibration desired by the stimulus provider, and when adjustments are made by the stimulus provider referencing actual measurements, a stimulus provider with specialized skills is always required.

[0042] Therefore, the information processing method according to the embodiment includes: acquiring XR content including sound signals; performing analysis processing on the sound signals for vibration conversion; and generating vibration stimuli to be provided to the user based on the analysis processing results.

[0043] Specifically, such as Figure 1 As shown, in the information processing method according to the embodiment, firstly, the field device 10 provides XR content from the field (step S1). Next, the remote device 100 performs analysis processing for vibration conversion on the sound signals included in the provided XR content, and generates vibration stimuli to be provided to the user based on the analysis processing results (step S2).

[0044] For example, in the information processing method according to the embodiment, (1) frequency analysis is performed on the sound signal by a method such as Fast Fourier Transform (FFT). Thus, if the level of a predetermined low-frequency range is less than a pre-set threshold, the frequency is divided by N (1 / N) by pitch shift; if the level of the predetermined low-frequency range is not less than the pre-set threshold, the output is unchanged.

[0045] For example, in the information processing method according to the embodiment, (2) sound source estimation is performed on the sound signal by using an artificial intelligence (AI) inference model that estimates the sound source based on the sound signal. As a result, by setting, when the sound source is a frequency division object, the frequency is divided by N by pitch shift, and when the sound source is not a frequency division object, the output is unchanged.

[0046] For example, in the information processing method according to the embodiment, (3) as a method other than pitch shift, a threshold is set for frequency A, which is the lowest frequency of the uncut frequency range, and when a sound greater than the threshold is input, a signal consisting of frequencies equal to or less than frequency A is input to increase the low frequency range.

[0047] The above (1) to (3) will be referred to later. Figures 2 to 12 The first embodiment will be described.

[0048] For example, in the information processing method according to the embodiment, (4) calibration of the vibration characteristics is performed based on the differences between the objects providing the vibration and the state of the objects. (4) will be referred to later. Figures 13 to 16 The following description is provided as a second embodiment.

[0049] For example, in the information processing method according to the embodiment, (5) a specific scene is detected from the input image signal and the input sound signal. Based on the detected scene, the frequency is divided by N by the pitch displacement based on the initially set vibration parameters. (5) will be referred to later. Figure 17 and Figure 18 The following description is provided as a third embodiment.

[0050] In other words, in the information processing method according to the embodiment, as described in (1) to (5) above, a vibration pattern to be provided to the user is generated based on the analysis processing results. The remote device 100 drives the vibration output unit 130 based on the generated vibration pattern to provide, for example, vibration stimulation with an increased low-frequency range, or vibration stimulation based on the object.

[0051] Therefore, the sense of presence can be further enhanced through vibration stimulation during content reproduction.

[0052] As described above, the information processing method according to the embodiment includes: acquiring XR content including sound signals; performing analysis processing on the sound signals for vibration conversion; and generating vibration stimuli to be provided to the user based on the analysis processing results.

[0053] Therefore, by employing the information processing method according to the embodiments, the sense of presence can be further enhanced through vibration stimulation during content reproduction. Hereinafter, embodiments of the information processing system 1 applying the information processing method according to the embodiments will be described in detail.

[0054] First Embodiment

[0055] Figure 2 This is a diagram illustrating an example of the structure of the information processing system 1 according to the first embodiment. Figure 3 This is a diagram illustrating a structural example of the field device 10 according to the first embodiment. Figure 4 This is a diagram illustrating a structural example of a remote device 100 according to a first embodiment. Figure 5 This is a diagram showing a structural example of the vibration output unit 130.

[0056] like Figure 2 As shown, the information processing system 1 includes a field device 10 and at least one remote device 100. Each of the field device 10 and the at least one remote device 100 is an example of an "information processing apparatus" and is implemented by a computer. The field device 10 and the at least one remote device 100 are connected to enable them to communicate with each other via a network N (e.g., the Internet, a private network, and a mobile phone network).

[0057] like Figure 3 As shown, the field device 10 includes at least one camera 11 and at least one microphone 12. The at least one camera 11 records images of the external environment. The at least one microphone 12 records sounds of the external environment. The field device 10 generates XR content, which includes images recorded by the at least one camera 11 and sounds recorded by the at least one microphone 12, and the field device 10 further provides the generated XR content to at least one remote device 100.

[0058] like Figure 4 As shown, the remote device 100 includes an image output unit 110, an audio output unit 120, and a vibration output unit 130. The image output unit 110 displays images included in XR content provided from the field device 10. The audio output unit 120 outputs audio included in the XR content.

[0059] The vibration output unit 130 outputs vibrations based on the sound included in the XR content. As described above, such as Figure 5As shown, the vibration output unit 130 is arranged, for example, in the chair S in which the user U sits. The vibration output unit 130 can be configured to be embedded in clothing, seat belts, etc., for attachment to the user U. Note that the vibration output unit 130 contains a known vibration transducer (e.g., an electro-vibration transducer consisting of a magnet (magnetic circuit) and a coil through which a drive current flows), and an electric power amplifier made of piezoelectric elements to amplify the signal to the level required for driving.

[0060] Figure 6 This is a block diagram illustrating a remote device 100 according to a first embodiment. Figure 7 This is a block diagram showing the sound / vibration conversion processing unit 103b. Figure 6 and Figure 7 as well as Figure 13 and Figure 17 In this document, only the structural elements necessary for illustrating the features of the embodiments are described, and the description of general structural elements is omitted.

[0061] In other words, Figure 6 and Figure 7 as well as Figure 13 and Figure 17 The specific forms of distribution and integration of the structural elements shown are not limited to those shown in the accompanying drawings. Depending on various load types, usage conditions, etc., the device can be functionally or physically separated or integrated in any unit to constitute all or part of the equipment.

[0062] exist Figure 6 and Figure 7 as well as Figure 13 and Figure 17 In the description, the description of the structural elements that have already been described can be simplified or omitted.

[0063] like Figure 6 As shown, the remote device 100 according to an embodiment includes a communication unit 101, a storage device 102, and a control unit 103. A vibration output unit 130 is connected to at least one remote device 100. Illustrations of the image output unit 110 and the sound output unit 120 are deliberately omitted to further illustrate the features of the embodiment.

[0064] The communication unit 101 is implemented, for example, by a network interface card (NIC). The communication unit 101 and the network N are connected to each other in a wired / wireless manner to send information to and receive information from the field device 10 via the network N.

[0065] Storage device 102 is implemented using semiconductor storage elements such as random access memory (RAM) and flash memory, or storage devices such as hard disks and optical disks. Figure 6In the example shown, the storage device 102 stores vibration parameter information 102a and a sound source estimation model 102b.

[0066] Vibration parameter information 102a includes, for example, various parameters related to the vibration to be output to vibration output unit 130, and various thresholds to be used in the determination described later. Sound source estimation model 102b is an AI inference model for estimating sound sources based on the aforementioned sound signal.

[0067] The sound source estimation model 102b uses a sound signal as input and outputs the sound source category with the highest probability distribution in the final layer via a learned neural network. Learning is performed using the sound signal and information about the sound source category provided to that sound signal as correct answer data, thereby reducing the loss between the classifier's output and the correct answer data. For example, the correct answer data is collected through manual annotation.

[0068] The control unit 103 is a controller, such as a central processing unit (CPU), a microprocessor unit (MPU), etc., that executes various programs (not shown) stored in the storage device 102 while using RAM as its working area, thereby implementing the control unit 103. The control unit 103 can be implemented by integrated circuits such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs).

[0069] The control unit 103 includes an acquisition unit 103a and a sound / vibration conversion processing unit 103b to implement and perform the following information processing functions and behaviors.

[0070] The acquisition unit 103a acquires XR content provided by the field device 10 via the communication unit 101.

[0071] The sound / vibration conversion processing unit 103b receives the sound signal included in the XR content acquired by the acquisition unit 103a, and further performs analysis processing for vibration conversion. Based on the analysis processing results, the sound / vibration conversion processing unit 103b generates a vibration mode to be provided to the user.

[0072] like Figure 7 As shown, the sound / vibration conversion processing unit 103b includes a high-frequency range cutoff unit 103ba, a determination unit 103bb, a pitch shift unit 103bc, and an amplifier 103bd.

[0073] The high-frequency truncation unit 103ba uses a low-pass filter (LPF) to truncate the high-frequency range that is not needed in vibration conversion as a preprocessing step on the sound signal whose low-frequency range has already been truncated using an HPF in the recording. This is because humans primarily and strongly perceive the low-frequency components of vibration as vibrations. The determination unit 103bb receives and analyzes the sound signal whose high-frequency range has been truncated in order to determine the necessity / unnecessity of pitch shift.

[0074] The determination unit 103bb determines the necessity / unnecessity of pitch displacement through frequency analysis such as FFT. For example, the determination unit 103bb inputs an audio signal into the sound source estimation model 102b and determines the necessity / unnecessity of pitch displacement based on the output of the sound source estimation model 102b in response to the input.

[0075] When the determining unit 103bb determines that pitch shift is required, the pitch shifting unit 103bc performs pitch shifting on the sound signal. The amplifier 103bd amplifies the sound signal and outputs it as a vibration signal to the vibration output unit 130.

[0076] Here, for reference Figures 8 to 10 Supplementary explanations are provided regarding the audio signal conversion and processing. Figures 8 to 10 This is a supplementary diagram illustrating the sound signal conversion processing according to the first embodiment.

[0077] like Figure 8 As shown in the upper and middle sections, the low-frequency range of the actual ambient sound recorded in the field is truncated by the HPF. As a result, when the signal level in a predetermined frequency range is less than a set threshold, for example, as... Figure 8 As shown in the lower part, when the average level at 20Hz is equal to or less than -20dB, in the audio signal conversion process, the frequency is divided by N (here N=2) by pitch shift. Note that... Figure 8 The example shown corresponds to (1) above. For example, for the predetermined frequency range and threshold mentioned above, appropriate values ​​are determined based on sensory analysis results, etc.

[0078] like Figure 9 As shown, a threshold is set for the cutoff frequency, which is the lowest frequency in the uncut remaining frequency range. If the sound level of the cutoff frequency does not exceed the threshold, the low-frequency range is not increased in the sound signal conversion process.

[0079] On the other hand, such as Figure 10 As shown, when the cutoff frequency exceeds a threshold, during the sound signal conversion process, a signal consisting of frequencies equal to or less than the cutoff frequency (e.g., a signal generated based on sensory analysis results to provide an appropriate sensation) is simultaneously input to increase the low-frequency range. Note that... Figure 9and Figure 10 The example shown corresponds to (3) above. The above method should be used when increasing the low-frequency range helps improve the sense of presence but frequency division is ineffective.

[0080] Next, we will refer to Figure 11 and Figure 12 This describes the processing steps that remote device 100 needs to perform. Figure 11 This is a flowchart illustrating the processing steps performed by the remote device 100 according to the first embodiment. Figure 12 This is a flowchart illustrating the processing steps performed by the remote device 100 according to the first embodiment.

[0081] The main processing steps of audio signal conversion are as follows: Figure 11 and Figure 12 As shown. Figure 11 This corresponds to (1) above. Figure 12 This corresponds to (2) above.

[0082] In the case of (1) above, such as Figure 11 As shown, the sound / vibration conversion processing unit 103b first receives the sound signal (step S101), and then performs frequency analysis on the sound signal (step S102).

[0083] Determine whether the signal level in the predetermined low-frequency range is less than a set threshold (step S103). If the signal level is less than the threshold (step S103: Yes), perform frequency division (step S104), and output the divided sound signal as a vibration signal to the vibration output unit 130 (step S105). Next, end the process.

[0084] On the other hand, if the signal level exceeds the threshold (step S103: No), the sound signal is output to the vibration output unit 130 as a vibration signal without modification (step S105). Next, the process ends. In the case of (3) above, the process in step S104 is to add a signal consisting of a frequency equal to or less than the cutoff frequency.

[0085] In the case of (2) above, such as Figure 12 As shown, the sound / vibration conversion processing unit 103b first receives the sound signal (step S201), and then uses the sound source estimation model 102b to perform inference on the sound signal (step S202).

[0086] As a result of the inference, it is determined whether the sound source is the target of the frequency division (step S203). If it is the target of the frequency division (step S203: yes), frequency division is performed (step S204), and the frequency-divided sound signal is output as a vibration signal to the vibration output unit 130 (step S205). Next, the processing ends.

[0087] On the other hand, if the sound source is not the target of frequency division (step S203: No), the sound signal is output to the vibration output unit 130 as a vibration signal without modification (step S205). Next, the processing ends.

[0088] Second Embodiment

[0089] Next, a second embodiment corresponding to (4) above will be described. Figure 13 This is a block diagram illustrating a remote device 100A according to a second embodiment. Note that... Figure 13 Corresponding to Figure 6 Therefore, the main explanation will be related to Figure 6 Different parts. Figures 14 to 16 This is a supplementary diagram illustrating the sound signal conversion processing according to the second embodiment.

[0090] like Figure 13 As shown, the remote device 100A differs from the first embodiment in that it further includes an acceleration sensor 140 and a calibration unit 103c. The calibration unit 103c calibrates the vibration characteristics based on the differences between the objects to be vibrated and the state of the objects.

[0091] First, the case of differences between objects to be vibrated will be explained. In this case, before presenting actual vibration, the calibration unit 103c acquires the vibration characteristics when a predetermined reference signal is provided to a reference object. For example, an accelerometer 140 is arranged on the seat surface of a reference chair α to acquire the actual vibration characteristics of the chair α when a reference signal is provided. The vibration signal is configured to generate a vibration signal based on the aforementioned characteristics of the reference chair α to achieve object vibration. Figure 14 In the example shown, if the vibration characteristics of the chair being used are similar to those of the reference chair α described above, then the desired vibration can be provided to the user.

[0092] On the other hand, calibration unit 103c acquires vibration characteristics when the same reference signal is provided to a vibration device used by a user who wants to receive actual vibrations. In this case, for example, accelerometer 140 is arranged on the seat surface of wheelchair β to acquire actual vibration characteristics when a reference signal is input to wheelchair β. Assuming... Figure 15 This indicates the vibration characteristics of the aforementioned wheelchair β.

[0093] The calibration unit 103c adjusts the output level of each frequency of the vibration signal to be output to the wheelchair β in order to reduce the difference between the vibration characteristics of the chair α and the vibration characteristics of the wheelchair β.

[0094] For example, such as Figure 14 and Figure 15 As shown, it is assumed that the wheelchair β has a vibrational characteristic with significantly reduced vibration at 40 Hz compared to the chair α. In this case, as... Figure 16 As shown, the calibration unit 103c adjusts the vibration signal to be output to the wheelchair β using an equalizer to increase the 40Hz level by +2dB or more. The calibration unit 103c stores this adjustment feature in the vibration parameter information 102a, so that the sound / vibration conversion processing unit 103b performs the adjustment when actually providing vibration to the wheelchair β.

[0095] Next, we will explain calibration based on object state. It is difficult to measure a person's sensation of vibrations received through the skin; however, it is well known that the intensity of vibrational stimuli perceived by a person is generally related to the amount of fat stored in him / her.

[0096] Therefore, the calibration unit 103c stores parameters in advance for vibration adjustment, for example, based on body weight in 10kg increments. The calibration unit 103c measures the weight of the person to be vibrated. For example, assume the weight of person C is 80kg.

[0097] Next, calibration unit 103c adjusts the vibration characteristics for subject C (i.e., a person weighing 80 kg) so that subject C experiences vibrations similar to those experienced by person B of appropriate weight. For example, a person weighing 80 kg is estimated to have difficulty sensing vibrations compared to a person weighing 60 kg. In this case, for example, calibration unit 103c adjusts the vibration output level of subject C to be +2 dB higher than the vibration output level of a person weighing 60 kg. Note that in this example, the vibration level (amplitude) is adjusted based on weight; however, various parameters used for vibration adjustment, such as characteristics of the vibration frequency level, can be adjusted based on weight.

[0098] As described above, vibration characteristics are calibrated based on the differences between the objects to be vibrated and the state of the objects, thereby enhancing the sense of presence provided by the vibration stimulus independently of the objects. Note that calibration that actually provides vibration (signals) to the object to be vibrated, measures its response, and performs calibration based on the results can be called actual measurement type, while calibration that detects the state of the object (weight, etc.) and performs calibration based on the detection results can be called inferential type.

[0099] In the inferential example above, body weight is used as an example of the state of the object to be vibrated; however, it is not limited to this, and can also be used for example, bone density, age, gender, etc.

[0100] Third Embodiment

[0101] The third embodiment corresponding to (5) above will be described. Figure 17 This is a block diagram illustrating a remote device 100B according to a third embodiment. Figure 17 and Figure 13 Similarly, corresponding to Figure 6 Therefore, the main explanation will be related to Figure 6 Different parts.

[0102] like Figure 17 As shown, the remote device 100B differs from the first embodiment in that the remote device 100B also includes a scene detection unit 103d and an extraction unit 103e.

[0103] The scene detection unit 103d detects a specific scene from the image and sound signals of the XR content acquired by the acquisition unit 103a. For example, the scene detection unit 103d detects a scene based on the arrival of a pre-set time point. In this case, the occurrence time point of the specific scene (the playback position time point of the XR content) is predetermined manually. As for determining the occurrence time point as described above, the following methods are considered: a method of directly determining the time point; and a method of determining the object scene category by performing a process of matching the scene data, image / sound data, etc., estimated based on the scene data included in the XR content data and the playback position time point data.

[0104] Scene detection unit 103d detects scenes based on their positional relationships relative to objects within the XR content. For example, consider a situation within a predetermined distance of fireworks. For instance, proximity within the predetermined distance is determined by the objects (categories) included in the XR content data and their position data. Scene detection unit 103d detects scenes based on changes in conditions within the XR content. For example, consider a user entering a concert hall within the virtual space of the XR content. Scene detection unit 103d detects scenes based on contact relationships with objects within the XR content. For example, consider a user colliding with something in the virtual space of the XR content. For instance, collision detection is also determined by the objects (categories) included in the XR content data and their position data.

[0105] Vibration parameters in each scene are initially set in vibration parameter information 102a, and extraction unit 103e extracts vibration parameters based on the scene detected by scene detection unit 103d.

[0106] The sound / vibration conversion processing unit 103b performs sound signal conversion processing based on the vibration parameters extracted by the extraction unit 103e.

[0107] Next, we will refer to Figure 18This describes the processing steps to be performed by the remote device 100B. Figure 18 This is a flowchart illustrating the processing steps performed by the remote device 100B according to the third embodiment.

[0108] like Figure 18 As shown, in the third embodiment, the scene detection unit 103d detects the scene based on image signals and sound signals of the XR content (step S301). The sound / vibration conversion processing unit 103b receives the sound signal (step S302).

[0109] Determine whether the scene detected by the scene detection unit 103d is a scene for frequency division (whether the scene is an object of vibration enhancement processing) (step S303). Here, if the scene is a scene for frequency division (step S303: yes), frequency division is performed (step S304), and the frequency-divided sound signal is output as a vibration signal to the vibration output unit 130 (step S305). Next, the processing ends.

[0110] On the other hand, if the scene is not a frequency division target (step S303: No), the sound signal is output to the vibration output unit 130 as a vibration signal without modification (step S305). Next, the processing ends.

[0111] As described above, at least one remote device 100, 100A, and 100B includes a control unit 103 that generates a vibration stimulation signal to be provided to a user based on an audio signal in the content. The control unit 103 is configured to: acquire data of XR content (corresponding to an example of “content”) including the audio signal; perform analysis processing on the audio signal; and generate a vibration stimulation signal to be provided to the user in a conversion process of the audio signal based on the result of the analysis processing.

[0112] Therefore, based on at least one remote device 100, 100A and 100B, the sense of presence can be further enhanced by appropriate vibration stimulation based on the analysis results during the reproduction of XR content.

[0113] The conversion process includes enhancement of the low-frequency range in the vibration stimulus signal based on the results of the analysis.

[0114] Therefore, according to at least one remote device 100, 100A and 100B, the low-frequency range of vibration stimulation during XR content reproduction is enhanced to change to an appropriate state, thereby further improving the sense of presence.

[0115] Enhancement processing includes frequency division processing of the audio signals used in the conversion process.

[0116] Therefore, according to at least one remote device 100, 100A and 100B, the low-frequency range of vibration stimulation during the reproduction of XR content is enhanced by frequency division of the sound signal to change to an appropriate state, thereby further improving the sense of presence.

[0117] Frequency division processing includes: dividing the sound signal into frequencies by pitch shifting based on the analysis and processing results.

[0118] Therefore, by utilizing pitch displacement to achieve frequency division of sound signals according to at least one remote device 100, 100A and 100B, the low-frequency range of vibrational stimulation during the reproduction of XR content is enhanced to change to an appropriate state, thereby further improving the sense of presence.

[0119] The conversion process includes synthesizing a vibration signal composed of signals within a predetermined low-frequency range.

[0120] Therefore, based on at least one remote device 100, 100A and 100B, it is possible to generate vibrations in the low-frequency range that are enhanced by methods other than pitch shift, thereby further improving the sense of presence caused by vibration stimulation during XR content reproduction.

[0121] The control unit 103 is also configured to perform enhancement processing when the level of a predetermined low-frequency range in the sound signal is less than a pre-set threshold.

[0122] Therefore, based on at least one remote device 100, 100A, 100B, and according to the level of a predetermined low-frequency range, it is determined whether the enhancement processing for the low-frequency range of the vibration to be provided, performed in the sound / vibration conversion processing unit 103b, is necessary to generate a vibration signal so as to provide appropriate vibration without being over-amplified.

[0123] The control unit 103 is also configured to: estimate the sound source by means of an artificial intelligence (AI) inference model that estimates the sound source of the sound signal; and perform conversion processing corresponding to the estimated sound source.

[0124] Therefore, based on at least one remote device 100, 100A and 100B, vibrations can be generated according to an inferred sound source, thereby further enhancing the sense of presence caused by vibration stimulation during XR content reproduction.

[0125] The control unit 103 of the remote device 100B is also configured to: detect a specific scene from the XR content; and perform a conversion process corresponding to the detected scene.

[0126] Therefore, according to the remote device 100B, it is possible to generate enhanced low-frequency vibrations based on the detected scene, thereby further improving the sense of presence caused by vibration stimulation during XR content reproduction.

[0127] The control unit 103 of the remote device 100A is also configured to perform calibration on the conversion process based on the vibration environment.

[0128] Therefore, according to the remote device 100A, it is possible to generate vibrations that are adjusted according to the condition of the object, thereby enhancing the sense of presence caused by vibration stimulation without depending on the object.

[0129] In the above embodiments, the sound / vibration conversion processing was described as being performed by a remote device; however, the sound / vibration conversion processing can also be performed by a field device. In this case, the XR content to be provided includes a vibration signal for providing vibrational stimulation. Furthermore, data required for calibration, etc., is transmitted between the remote device and the field device.

Claims

1. An information processing device, comprising a control unit, the control unit generating a vibration stimulation signal to be provided to a user based on an audio signal in content, wherein, The low-frequency range of the audio signal in the content is truncated by a high-pass filter. The control unit is configured to: Extract the audio signal from the content; The high-frequency range of the acquired audio signal is truncated using a low-pass filter; and If the level of the sound signal at the cutoff frequency exceeds a predetermined threshold, a vibration signal is generated by amplifying the signal obtained by using a low-pass filter to cut off the high-frequency range of the sound signal by using a pre-created signal consisting of frequencies equal to or less than the cutoff frequency, wherein the cutoff frequency is the lowest frequency in the frequency range not cut off by the high-pass filter. If the level of the sound signal at the cutoff frequency of the high-pass filter does not exceed a predetermined threshold, the signal obtained by truncating the high-frequency range of the sound signal using a low-pass filter will be used as the vibration signal.

2. An information processing method for generating vibration stimulation signals to be provided to a user based on sound signals in content, wherein the low-frequency range of the sound signals in the content is truncated by a high-pass filter, the method comprising: Extract the audio signal from the content; Use a low-pass filter to truncate the high-frequency range in the acquired audio signal; If the level of the sound signal at the cutoff frequency exceeds a predetermined threshold, a vibration signal is generated by amplifying the signal obtained by using a low-pass filter to cut off the high-frequency range of the sound signal by using a pre-created signal consisting of frequencies equal to or less than the cutoff frequency, wherein the cutoff frequency is the lowest frequency in the frequency range not cut off by the high-pass filter. If the level of the sound signal at the cutoff frequency of the high-pass filter does not exceed a predetermined threshold, the signal obtained by truncating the high-frequency range of the sound signal using a low-pass filter will be used as the vibration signal.

Images