INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING SYSTEM AND INFORMATION PROCESSING PROCESS

Abstract

An information processing device according to embodiments controls a vibration device comprising multiple vibration generators and configured to exert vibrations on a user in response to an input signal. The information processing device identifies a directional component of an object representing a vibration source with respect to the vibration device in an input signal and controls the amplitude and delay of each of the vibration generators based on the directional component.

Description

Field of invention

[0001] The present invention relates to an information processing device, an information processing system and an information processing method. General state of the art

[0002] Technologies for providing users with digital content referred to as Cross Reality (XR), which encompasses experiences in virtual space such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), for example using a head-mounted display (HMD), are conventionally known. XR is a collective term for all virtual space technologies, including VR, AR, MR, as well as Substitutional Reality (SR) and Audiovisual (AV).

[0003] For example, a technology has been proposed to enhance a realistic feeling for a video by conveying vibrations to the user that are associated with the video being viewed by the user (see, for example, patent literature example 1).

[0004] Another technology has been proposed that controls vibrations for each of the cells arranged on a seating surface and provides a signal to the user (see, for example, patent literature example 2). List of literature on patent literature Patent literature example 1: Japanese patent disclosure no. 2004-081357 Patent literature example 2: Japanese patent disclosure no. 2021-158392 Brief description of the invention: Technical problem

[0005] However, with conventional technologies it is difficult to give users a natural feeling of locating vibrations.

[0006] Fig. Figure 3 shows a conventional method for providing vibrations. Fig. Figure 3 is a view of the seat surface of a seat on which the user is sitting, viewed vertically downwards from above the head of a user.

[0007] As in Fig. As shown in Figure 3, the seat surface has vibration elements 51a_FL (front left), 51a_RL (rear left), 51a_FR (front right) and 51a_RR (rear right). The output vibrations of each of the vibration elements are controlled in such a way that the user experiences a sensation of localizing the vibrations corresponding to the vibration content, that is, at the position of a vibration source within the content (at the position relative to the user's position).

[0008] If, for example, it is assumed that the vibration source (the object that generates vibrations) is located in the content to the right of the user, the vibration intensity of the vibration generators 51 at the individual positions will be, for example, as shown in Fig. 23 shown front left (FL) set to 1, rear left (RL) set to 0, front right (FR) set to 8 and rear right (RR) set to 1 to provide a sense of localization of the vibration source.

[0009] In this case, the user feels a strong vibration sensation on the right front of the seat and can therefore recognize that the source of the vibration is located at the front right.

[0010] However, with such a method of delivering vibrations, while the user may experience a certain sense of localizing the vibration source due to the difference in vibration intensity between the positions of the seat, there are few factors that convey a sensation of vibration propagation. Therefore, there is a need for a method of delivering vibrations that provides a vivid, realistic sensation, allowing the user to experience a rich sense of vibration localization. Solution to the problem

[0011] To solve this problem and fulfill the task, an information processing device is provided for exerting vibrations on a user according to a content by controlling a vibration device having multiple vibration generators, wherein the information processing device has a controller, the controller identifying a directional component of the object representing a vibration source in an input content and controlling an amplitude and delay of output vibrations of each of the vibration generators based on the directional component. Advantageous effects of the invention

[0012] The present invention makes it possible to generate vibrations that convey a feeling of vibration propagation and to provide users with vibrations with a more realistic sensation. Brief description of the drawings Fig. Figure 1 is a diagram that shows an overview of an information processing system. Fig. Figure 2 is a diagram showing a data flow in the information processing system. Fig. Figure 3 is a diagram showing an example of the construction of a vibration device. Fig. Figure 4 is a diagram that shows an overview of an information processing procedure. Fig. Figure 5 is a block diagram of an information processing device. Fig. Figure 6 is a diagram showing an example of a scene information database. Fig. Figure 7 is a diagram showing an example of the scene information database. Fig. Figure 8 is a diagram showing an example of the scene information database. Fig. Table 9 shows an example of a priority information database. Fig. Figure 10 is a diagram showing an example of a parameter information database. Fig. Figure 11 is a diagram showing an example of a vibration generator information database. Fig. Figure 12 is a block diagram of a scene detection unit. Fig. Figure 13 is a block diagram of a priority setting unit. Fig. Figure 14 is a diagram showing an example of a procedure for determining a higher priority objective. Fig. Figure 15 is a block diagram of a parameter extraction unit. Fig. Figure 16 is a block diagram of an output unit. Fig. Figure 17 is a diagram showing an example of a method for processing vibration localization. Fig. Figure 18 is a diagram showing an example of a signal processing procedure. Fig. Figure 19 is a flowchart showing a process flow that is executed by the information processing device. Fig. Figure 20 is a flowchart showing the sequence of a vibration localization process. Fig. Figure 21 is a diagram showing an example of the vibration generator information database. Fig. Figure 22 is a diagram showing an example of a method for controlling vibrations. Fig. Figure 23 is a diagram showing a conventional method for providing vibrations. Description of embodiments

[0013] With reference to the accompanying drawings, embodiments of an information processing device, an information processing system, and an information processing method disclosed in this application are described in detail below. The present invention is not limited to the following embodiments. First embodiment

[0014] First, using Fig. 1, Fig. 2, Fig. 3 and Fig. 4. An overview of the information processing system and the information processing procedure according to its embodiments will be given. Fig. Figure 1 is a diagram that shows an overview of the information processing system. Fig. Figure 2 is a diagram showing the flow of data in the information processing system. Fig. Figure 3 is a diagram showing an example of the construction of a vibration device. Fig. Figure 4 is a diagram that provides an overview of the information processing procedure. The following section will explain a case where an XR space (a virtual space) is a VR space.

[0015] As in Fig. As shown in Figure 1, an information processing system 1 comprises a display device 3, a loudspeaker 4 and a vibration device 5.

[0016] As in Fig. As shown in Figure 2, an information processing device 10 provides video data to the display device 3. The information processing device 10 also provides audio data to the loudspeaker 4. Furthermore, the information processing device 10 provides vibration data to the vibration device 5.

[0017] As in Fig. As shown in Figure 1, the display device 3, for example, is a head-mounted display. The display device is an information processing terminal used to show the user video data related to XR content provided by the information processing device 100 and to allow the user to enjoy VR experiences.

[0018] The display device 3 can be of a non-transparent type, in which the field of view is completely covered, or of a video-transparent or optically transparent type. The display device 3 includes a device that detects changes in the user's internal and external environment by means of a sensor such as a camera and a motion sensor.

[0019] The loudspeaker 4 is an audio output device that emits sound and is provided, for example, in the form of headphones worn on the user's ears. The loudspeaker 4 generates audio data provided by the information processing device 10 as sound. The loudspeaker 4 is not limited to the headphone type but can be a box (placed on the floor or the like). The loudspeaker can be a stereo or a multi-channel loudspeaker.

[0020] The vibration device 5 has several vibration generators. Each vibration generator has a current-to-vibration transducer and a piezoelectric element, is installed, for example, in a seat on which the user sits, and vibrates according to vibration data provided by the information processing device 10. The information processing device 10 controls each vibration generator of the vibration device 5 individually.

[0021] Fig. Figure 3 is a view of the seat surface of a seat on which the user is sitting, viewed vertically downwards from above the user's head. As in Fig. As shown in Figure 3, the vibration generators of the vibration device 5, that is, vibration generators 51_FL, 51_RL, 51_FR and 51_RR, are installed at the left front, left rear, right front and right rear positions on the seat surface of the seat.

[0022] When the user sits on the seat, the vibration generators come into contact with different parts of the body and transmit vibrations. For example, the vibration generators 51_FL, 51_RL, 51_FR and 51_RR transmit vibrations to the left thigh, left buttock, right thigh and right buttock of the user sitting on the seat, respectively.

[0023] Sounds from the speaker 4 and vibrations from the vibration device 5, in other words, waves from wave devices, are adapted to the video being played and exerted on the user of the content, thereby increasing the realistic sensation during video playback.

[0024] The information processing device 10 is a computer and is connected to the display device 3 via a wired or wireless connection to provide the display device 3 with videos containing XR content. For example, the information processing device 10 may, if necessary, detect changes in a situation as detected by the sensor in the display device 3 and reflect these changes in the XR content.

[0025] For example, the information processing device 10 can change the orientation of the field of view in the virtual space of the XR content in response to changes in the user's head and gaze direction detected by the sensor.

[0026] When providing XR content, the realistic perception of the XR content can be enhanced by emphasizing sounds produced by the speaker 4 according to the scene or by allowing the vibration device 5 to vibrate according to the scene.

[0027] However, the parameters used to control a realistic sensation in such a way as to enhance that realistic sensation (hereinafter referred to as "realism parameters") must be set manually after the XR content has been created, resulting in an enormous amount of work for setting the realism parameters.

[0028] The information processing method therefore aims to automate the determination of these realism parameters. For example, as in Fig. Figure 4 shows that in the information processing method, according to embodiments, scenes that meet a predetermined condition are first detected from the video and audio data associated with the XR content (step S1).

[0029] Here, the predetermined condition is, for example, a condition such that the corresponding video or audio data is a scene that requires the setting of realism parameters, and is defined, for example, by a conditional expression in connection with a situation in which the XR content is located.

[0030] In other words, the information processing method detects a scene as fulfilling the predetermined condition when the situation in the XR content meets the condition defined by a conditional expression. This process eliminates the need for further processing, such as detailed analysis of the video data, thereby reducing the processing load for scene detection.

[0031] Subsequently, in the information processing procedure for the scenes detected by the scene detection, priorities are set (step S2). Here, the priorities indicate a ranking, such as which scene should be given priority with regard to the realism parameters. In other words, in the information processing procedure, if several scenes overlap over time, it is predefined which scene should be given priority with regard to the realism parameters.

[0032] This implementation can provide the user with a suitably realistic sensation even when multiple scenes overlap. As described below, the priorities for sound and vibration are set separately.

[0033] Subsequently, in the information processing procedure, realism parameters are extracted for each scene (step S3). For example, in the information processing procedure using parameter information in which the relationship between scenes and realism parameters has been predefined, realism parameters are extracted for each scene.

[0034] In this case, the relevant realism parameters are extracted according to priority during the information processing procedure. Specifically, for example, if a low-priority scene overlaps a high-priority scene, the realism parameters of the high-priority scene are extracted.

[0035] In the information processing procedure, an audio amplification process is performed using an audio amplification parameter among the extracted realism parameters to amplify audio data (step S4), and the processed data is output to the loudspeaker 4. In the information processing procedure, a vibration conversion process is performed to convert audio data into vibration data, and the vibration data is then amplified using a vibration parameter among the extracted realism parameters (step S5), and the processed data is then output to the vibration device 5.

[0036] In this information processing procedure, this implementation can provide the user with sounds that have been amplified according to the scene viewed by the user, and vibrations that are appropriate for the scene.

[0037] In this way, according to embodiments of the information processing method, a scene is detected from XR content, and after setting a priority for this scene, realism parameters related to wave control, which includes audio processing and vibration processing, are extracted. The information processing method according to embodiments can therefore automate the setting of realism parameters related to improving the realistic perception of content.

[0038] Furthermore, at step S5, the information processing device 10 identifies a directional component of a noise source relative to the vibration device 5 in the input content. The information processing device 10 then controls the output vibrations of several vibration generators based on the identified directional component. The information processing device 10 can thus provide the user with a sense of vibration localization.

[0039] Next, with reference to Fig. 5 An example of the structure of the information processing device 10 will be described according to embodiments. Fig. Figure 5 is a block diagram of the information processing device 10. As in Fig. As shown in Figure 5, the information processing device 10 has a control unit 120 and a storage unit 130.

[0040] The storage unit 130 is implemented, for example, by a semiconductor storage device such as random access memory (RAM) or flash memory, or other storage devices such as a hard drive or an optical disk. In the example that is in Fig. As shown in Figure 5, the storage unit 130 contains an XR content database (DB) 131, a scene information DB 132, a priority information DB 133, a parameter information DB 134 and a vibration generator information DB 135.

[0041] The XR content database 131 is a database that stores a group of XR content to be displayed on display device 3. The scene information database 132 is a database that stores various information regarding scenes to be detected.

[0042] Fig. Figures 6 to 8 are diagrams showing examples of the scene information database 132. As in Fig. As shown in Figure 6, the scene information DB 132 stores, for example, information on points such as "detection scene", "condition category", "target", "condition parameter", "threshold" and "condition expression", which are linked together.

[0043] The "detection scene" specifies the name of a scene to be detected. The "detection scene" acts as an identifier and usually uses a code such as a numerical value; however, for the sake of clarity, a descriptive term is used in this example (repetition is prohibited). The "condition category" specifies a category, such as the information on which the scene detection is based. In the example shown in the figure, the categories primarily include a positional relationship between a user and a target, the user's action, spatial information about the user's location, temporal information about the user's location, and the generation of a sound by the target. Here, "user" refers to the operator themselves within the XR space.

[0044] The "Target" specifies a goal for scene detection. In the example shown in the figure, the target corresponds to information such as OBJECT 1, OBJECT 2, User, ROOM 1, ROOM 1 + OBJECT 3, CONTENT 1, OBJECT 4, OBJECT 5, or OBJECT 6. Here, OBJECT 1, OBJECT 2, OBJECT 3, OBJECT 4, OBJECT 5, and OBJECT 6 represent different objects in the XR space. For example, ROOM 1 represents a room in the XR space where the user is present. CONTENT 1, for example, represents a predetermined event in the VR space.

[0045] The "condition parameter" specifies a condition regarding a parameter, such as which parameter is used for scene detection. As shown in the figure, information such as distance, angle, speed, acceleration, rotational speed, interior space, presence of an object, quantity, start and end times, and audio patterns are linked.

[0046] The "threshold" specifies a threshold value that corresponds to a condition parameter. The "condition expression" specifies a condition expression for the detection of a detection scene. For example, the relationship between the condition parameter and the threshold is defined as a condition expression.

[0047] In Fig. For the sake of clarity, symbols such as "W", "4", and "w" are used to denote point values ​​such as "SCENE W", "OBJECT 4", and "SAMPLE w". In fact, each data value is stored as data in such a way that its specific meaning is understandable.

[0048] For example, “SCENE W”, “SCENE X”, “SCENE Y” and “SCENE Z” are actually data such as “Scene of a walking elephant”, “Scene of a walking horse”, “Scene of a moving motor vehicle” and “Scene of a motor vehicle making a sharp turn”.

[0049] In this case, “OBJECT 4”, “OBJECT 5” and “OBJECT 6” are actually data such as “horse”, “elephant” or “motor vehicle”.

[0050] Moreover, “PATTERN w”, “PATTERN x”, “PATTERN y” and “PATTERN z” are actually data such as “pattern of the sound of a walking horse”, “pattern of the sound of a walking elephant”, “pattern of the sound of a moving motor vehicle” and “pattern of the sound of squealing tires”.

[0051] A noise pattern is represented, for example, by a feature vector with audio features as elements. These features can be obtained, for example, by performing a spectral decomposition on an audio signal (e.g., Mel filter bank or Cepstrum).

[0052] Then, if the similarity (for example, the cosine similarity or the Euclidean distance) between the feature vectors corresponding to two audio patterns is equal to or greater than a threshold, it can be said that the two audio patterns are similar.

[0053] For example, “audio pattern similar to PATTERN w” means that the similarity between a feature vector calculated from the noise produced in a scene and a feature vector of a noise that matches PATTERN w is equal to or greater than a threshold.

[0054] The threshold for the similarity between audio patterns may also be contained in the “threshold” in the scene information DB 132.

[0055] The information processing device 10 can, for example, be a combination of condition categories or condition parameters that are in Fig. The methods shown in 6 are used to detect a scene. For example, a detection scene like the one shown in Fig. 7 shows that the condition categories of several scenes can be determined by combining them, or a detection scene can be defined as in Fig. 8 shows that the condition parameters can be set by combining the condition parameters of several categories.

[0056] For example, defining a new detection scene can be simplified by combining the condition categories or condition parameters in this way.

[0057] Returning to Fig. Section 5 will describe the priority information database 133. For example, according to embodiments, the information processing device 10 sets a priority for each scene on a rule basis. The priority information database 133 stores various pieces of information regarding the priorities of the realism parameters. Fig. Table 9 shows an example of priority information DB 133.

[0058] As in Fig. As shown in Figure 9, the priority information database 133, for example, stores information regarding items such as a "rule number" and a "priority rule" that are linked together. The "rule number" specifies a number for identifying a priority rule, and the "priority rule" specifies a rule regarding priorities.

[0059] The terms "Priority for earlier detected scene" and "Priority for later detected scene (switch to later scene)" mentioned in the character indicate that the realism parameter of a scene that occurs earlier or later in time is given priority. This simplifies the rules, for example, when determining scene priorities.

[0060] “Priority for a specific parameter with greater weight” indicates that the realism parameter of a scene with the larger value from the audio gain parameter or the vibration parameter is given greater priority among the realism parameters.

[0061] In other words, the realism parameter in this case can be provided in conjunction with audio data that is to be significantly amplified, or larger vibration data, since the realism parameter extracted for a scene with a larger audio amplification parameter or vibration parameter is set-

[0062] "Priority for parameters with greater weight" indicates that the realism parameter of a scene with a higher value than either the audio gain parameter or the vibration parameter is given priority among the realism parameters. In this case, parameters from different scenes can be used for the audio gain parameter and the vibration parameter.

[0063] In other words, the realistic perception of each element can be enhanced from the vibration and audio data, as each can be amplified by increasing the realism parameter. The weight, for example, determines the magnitude of this parameter value.

[0064] "Priority for parameters of shorter scene" indicates that the realism parameter of a scene with a shorter duration is given priority. If a short scene interrupts the playback of a long scene, the realism parameter of the shorter scene is preferably set during the shorter scene.

[0065] This implementation allows, for example, the appropriate amplification of a short scene. A rule can be defined that prioritizes the parameter of the longer scene.

[0066] "Priority for larger amplitude in the low frequency band" indicates that if scenes in which a target produces a noise occur simultaneously, priority will be given to the scene corresponding to a target that produces a noise with a larger amplitude in the low frequency band (e.g., below 500 Hz).

[0067] Generally, the amplitude in the low-frequency band of a living being's walking sound is greater the larger the living being is. Therefore, for example, when detecting a scene of a walking elephant and a scene of a walking horse, the scene in which the elephant is walking is given priority according to the rule "priority for greater amplitude in the low-frequency band".

[0068] “Priority for scene with large temporal changes in audio or video” indicates that a scene with large changes in the volume produced by a target or the position of a target in the video over a unit time will be given priority.

[0069] “Priority for scene with target near center of field of view” indicates that a scene in the content video that corresponds to a target located near the center of the screen will be given priority. This rule is later used in conjunction with… Fig. 14 will be explained.

[0070] “Priority for SCENE X over SCENE W” indicates that SCENE X will be given priority if both SCENE W and SCENE X are detected. This allows a person (a designer or developer) to manually define a priority rule for two or more specific scenes in advance.

[0071] Returning to Fig. Section 5 will describe the parameter information database 134. Parameter information database 134 is a database that stores information regarding the realism parameters for each scene. Fig. Figure 10 is a diagram showing an example of the parameter information database 134.

[0072] As in Fig. As shown in Figure 10, the parameter information DB stores 134 pieces of information on points such as "scene name", "audio gain parameters" and "vibration parameters", which are interconnected.

[0073] The "Scene Title" indicates the name of a detection scene described above and corresponds to the "Detection Scene" that is in Fig. 6 and other figures are shown. For the sake of clarity, the "scene description" will be given as explosion scene, concert hall scene, scene of a walking elephant, scene of a walking horse, scene of a moving motor vehicle, and scene of a motor vehicle making a sharp turn.

[0074] The "Audio Gain Parameter" specifies an audio gain parameter to be set in the corresponding scene. For example, audio gain parameters such as... Fig. Depending on the number of speakers, 10 parameters are shown for each speaker, such as "for speaker 1" and "for speaker 2".

[0075] For each speaker (4), for example, the values ​​of parameters related to audio processing, such as "delay" or "band gain / attenuation," are stored. For instance, "delay" specifies a parameter related to the delay time, and "band gain / attenuation" specifies a parameter such as which noise band is amplified or attenuated and to what extent.

[0076] The "vibration parameter" specifies a parameter related to the vibration to be defined in the corresponding scene. For example, parameters of points such as "low-pass filter (LPF)", "amplitude gain factor (ω)" and "delay gain factor (γ)" are stored as "vibration parameters".

[0077] “LPF” provides a parameter (in the example that is in Fig. Figure 10 shows a cutoff frequency) in connection with a low-pass filter used for vibration generation. The "amplitude gain factor (ω)" specifies a parameter related to the amplification and attenuation of the amplitude of the vibration used for vibration generation. The "delay gain factor (γ)" specifies a parameter related to the delay of the vibration generation time used for vibration generation.

[0078] Returning to the description of Fig. Section 5 describes the vibration generator information database 135. The vibration generator information database 135 is a database that stores information regarding the vibration generators present in the vibration device 5. Fig. Figure 11 is a diagram showing an example of the vibration generator information DB 135.

[0079] As in Fig. As shown in Figure 11, the vibration generator information DB 135 stores information about points such as the “vibration generator” and “position coordinates” that are linked together.

[0080] The "vibration generator" provides information for identifying a vibration generator present in the vibration device 5. The "position coordinates" specify the position of the vibration generator using coordinates.

[0081] Here, “FL”, “FR”, “RL” and “RR”, which are listed under “Vibration generator”, correspond to the vibration generators 51_FL, 51_RL, 51_FR and 51_RR in Fig. 3. The “position coordinates” can be determined by the fitter when the individual vibration generators are installed in the vibration device 5.

[0082] For example, the positional relationship between vibration generators can be recorded by referring to the vibration generator information DB 135.

[0083] Returning to the description of Fig. Section 5 will describe the control unit 120. The control unit 120 is a controller implemented, for example, when a central processing unit (CPU) or a microprocessing unit (MPU) executes various programs (not shown in the figure) stored in a memory unit 130, using RAM as the working area. The control unit 120 can be implemented by an integrated circuit such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FGPA).

[0084] The control unit 120 includes a content generation unit 121, a rendering process unit 122, a scene detection unit 123, a priority setting unit 124, a parameter extraction unit 125 and an output unit 126 to implement or execute the functions and actions of the information processing described below.

[0085] Content generation unit 121 creates a 3D model for content within an XR model. For example, content generation unit 121 references XR content database 131 to create a 3D model for a space within the XR content that corresponds to the user's current field of view within the XR content. Content generation unit 121 then passes the generated 3D model to rendering unit 122.

[0086] The rendering processing unit 122 performs a rendering process to convert the 3D model it received from the content generation unit 121 into video and audio data. For example, the rendering processing unit 122 outputs the converted video data to the display device 3 (see Fig. 2) and passes it on to the scene detection unit 123. The rendering process unit 122 passes the converted audio data to the output unit 126 and the scene detection unit 123. The content generation unit 121 and the rendering process unit 122 serve as a calculation unit that calculates condition data for the points in the condition expressions from the content.

[0087] The scene detection unit 123 detects a scene from the input content that meets a predetermined condition. For example, using the video data input from the rendering process unit 122 and a condition expression stored in the scene information database 132, the scene detection unit 123 detects a scene for which the realism parameters are to be set.

[0088] In this case, for example, the scene detection unit 123 receives coordinate information of an object in the XR space and information regarding the object type from the rendering process unit 122 and detects a scene for which the realism parameters are to be set using the condition expression.

[0089] For example, if the XR content is MR content, the scene detection unit 123 can detect objects in the MR space or calculate the coordinates of objects by performing an image analysis on an image taken in the RM space.

[0090] Fig. Figure 12 is a block diagram of the scene detection unit 123. As in Fig. As shown in Figure 12, the scene detection unit 123, for example, has a scene determination unit 123a and a condition definition unit 123b. Using the respective condition data (a condition expression) for scene determination, which is stored in the scene information DB 132, the scene determination unit 123a determines whether the situation in the video data fulfills a detection condition for the respective scene.

[0091] More precisely, the scene determination unit 123a determines, for example, how in Fig. 6 shown on the basis of data (calculated from the content by the content generation unit 121 or the rendering process unit 122) regarding the points in the conditional expressions such as the positional relationship between the user and the target (the object in the XR space), the action of the user and related spatial information about where the user is located, whether the current situation in the XR space corresponds to one of the predefined detection scenes.

[0092] Here, the scene determination unit 123 performs the scene detection process using data in the form of text information that has already been calculated by the content generation unit 121 or the rendering process unit 122, such as user movement in the XR space, object coordinate information, and information regarding object type and spatial information.

[0093] This allows the process from scene detection to the extraction of realism parameters to be carried out in parallel with a process with a relatively high processing load, such as the rendering process by the rendering processing unit 122, even if the CPU performance is relatively low.

[0094] In this case, the scene identification unit 123a can, for example, be based on scene identification information such as in Fig. Figure 7 shows a combination of condition categories or as in Fig. Figure 8 shows a combination of condition parameters that determine whether the current situation in the XR space corresponds to a respective detection scene.

[0095] If it is determined that the situation corresponds to a detection scene, the scene determination unit 123a provides the detection scene information for the video data to the priority setting unit 124 (see Fig. 5) further. If the scene determination unit 123a determines that the situation does not correspond to any of the detection scenes, the realism parameters are reset to their initial state (the realism parameters for no applicable detection scene), since no detection scene is applicable. If it is determined that the current situation in the XR space corresponds to multiple detection scenes, the scene determination unit 123a passes the determined multiple detection scenes to the priority determination unit 124.

[0096] Although a case has been described here in which the scene determination unit 123a determines whether any applicable detection scene is present based on video data, the scene determination unit 123a can determine whether any applicable detection scene is present based on audio data.

[0097] Scene identification unit 123a identifies scenes from the input content in which a sound is generated by a target. The scenes detected in this case correspond to a SCENE W, a SCENE X, a SCENE Y, and a SCENE Z (a scene of a walking elephant, a scene of a walking horse, a scene of a moving motor vehicle, and a scene of a motor vehicle making a sharp turn) in Fig. 6.

[0098] For example, the scene determination unit 123a calculates the similarity between a feature vector obtained from an audio signal of the content and a predetermined feature vector (for example, PATTERN w) and determines whether the similarity is equal to or greater than a threshold value.

[0099] Condition definition unit 123b defines various conditional expressions for scene detection. Condition definition unit 123b defines these conditional expressions based on information entered, for example, by the XR content creator or the user.

[0100] For example, the condition setting unit 123b receives input from the creator or user, such as which type of realism parameters are to be set and for which type of scene, and translates the situation of such a scene into a condition expression. Each time a condition expression is set, the condition setting unit 123a writes information about the condition expression to the scene information database 132 and writes the corresponding realism parameters to the parameter information database 134.

[0101] The condition setting unit 123b can predefine the scene information DB 132 and the parameter information DB 134 based on the content being viewed by the user.

[0102] The condition setting unit 123b can define a condition for detecting a scene in which a target produces a sound in the specified low-frequency domain. For example, the condition setting unit 123b adds a recording of a scene containing the sound of a walking elephant, which includes a sound in the low-frequency domain, as a detection scene to the scene information DB 132 (corresponding to a recording of SCENE W in Fig. 6).

[0103] By recognizing images and sounds contained in the content, the condition setting unit 123b can identify that there is a target (for example, an elephant) in the scene and that a sound is being generated in the low-frequency domain.

[0104] The condition setting unit 123b determines the value of the “vibration parameter” in the parameter information DB 134 according to the size of the target and the amplitude of each frequency band in the low frequency domain.

[0105] This implementation allows the scene information DB 132 and the parameter information DB 134 to be set automatically.

[0106] The condition setting unit 123b sets a condition based on a scene among the scenes in the content in which the amplitude of the noise in the frequency domain generated by the target exceeds a threshold.

[0107] For example, the threshold here can be the same as a threshold used in noise reduction to cut off the low-frequency domain.

[0108] The threshold can be determined by the type (the specifics) of the content. Types of content include music videos, to allow users primarily to listen to music, wildlife documentaries, to explain animal biology, and the like.

[0109] In music videos where an elephant is walking, it may be better to avoid excessive vibrations to prevent disrupting the music. Conversely, in wildlife documentaries where an elephant is walking, it may be better to create vibrations to enhance the realistic effect.

[0110] If the threshold for music videos is set lower than the threshold for wildlife documentaries, it is less likely that the condition setting unit 123b will view a scene in which an elephant walks in music videos as a target scene for which vibrations are generated.

[0111] This design allows vibrations to be generated that are appropriate for the content.

[0112] The above setting process for the scene information DB 132 and the parameter information DB 134 can be carried out by a person who actually views the content and operates an input device, instead of by the condition setting unit 132b.

[0113] This design enables the information processing device 10 to detect scenes desired by the producer or the user and to set the realism parameters desired by the producer or the user for the detected scenes.

[0114] Returning to Fig. Section 5 will describe the priority setting unit 124. The priority setting unit 124 sets priorities for the scenes detected by the scene detection unit 123.

[0115] For example, the priority setting unit 124 references the priority information database 133 to determine which scene should be given priority when the scene detection unit 123 simultaneously detects and identifies multiple types of scenes. If the scene detection unit 123 detects and identifies only one scene, that single scene is given the highest priority.

[0116] Fig. Figure 13 is a block diagram of the priority setting unit 124. For example, the priority setting unit 124, as shown in Fig. Figure 13 shows a time detection unit 124a and a rule setting unit 124b.

[0117] The time-detection unit 124a detects the time at which a scene detected by the scene detection unit 123 occurs and the time at which it ends. For example, based on the scene information from the scene detection unit 123 at each time point, the time-detection unit 124a detects scenes that are present at that time point (and also records the overlap state), the time of occurrence of the present scene, and the time at which the present scene disappears. In other words, the time-detection unit 124a records the states of all scenes that are present at each time point, including the order in which they occur.

[0118] The rule-setting unit 124b establishes the priorities of scenes to be used for determining the realism parameters, based on the scenes detected by the scene detection unit 123. In other words, based on the states of all existing scenes captured by the time-detection unit 124a, the priorities for detection scenes are set to determine the scene whose parameters are preferred for the realism parameters to be used at that time. This implementation allows the information processing device 10 to set the realism parameters according to the priorities.

[0119] In other words, by pre-setting a priority condition for each scene, the information processing device 10 can appropriately determine the scene whose realism parameters should be used first when SCENE A and SCENE B overlap in time.

[0120] For example, the rule setting unit 124b references the priority information DB 133 and, for each of the audio gain parameter and the vibration gain parameter, defines priorities for scenes for which parameters are to be used. In this case, the rule setting unit 124b can define a scene to be used for parameter selection, for example, based on an independent priority rule for each loudspeaker 4.

[0121] As a result, the realism parameters for each loudspeaker 4 are set according to their own rules, thereby further improving the realistic perception compared to a uniform setting of the realism parameters.

[0122] Furthermore, the rule setting unit 124b provides the information regarding the set rule in conjunction with video and audio data to the parameter extraction unit 125 (see Fig. 5) continue.

[0123] Returning to the description of Fig. Section 5 will describe the parameter extraction unit 125. The parameter extraction unit 125 extracts the realism parameters for the scene detected by the scene detection unit 123.

[0124] Fig. Figure 15 is a block diagram of the parameter extraction unit 125. As in Fig. As shown in Figure 15, the parameter extraction unit 125 includes a vibration parameter extraction unit 125a, an audio amplification parameter extraction unit 125b and a learning unit 125c.

[0125] The vibration parameter extraction unit 125a references the parameter information database 134 and extracts a vibration parameter corresponding to the scene assigned the highest priority by the priority setting unit 124. For example, the vibration parameter extraction unit 125a extracts a vibration parameter corresponding to the scene by extracting from parameter information database 134 the vibration parameter corresponding to the "detection scene" with the highest priority obtained from the priority setting unit 124.

[0126] In other words, if the scene detection unit 123 detects several temporally overlapping scenes in which the targets generating noise differ, the parameter extraction unit 125 can select a scene with a higher priority—that is, a scene that is likely to provide the user with a more realistic sensation through vibrations—and extract a vibration generation parameter corresponding to the selected scene. As a result, even during a period of content playback in which several scenes overlap, vibrations can be generated that convey a realistic sensation by using a suitable parameter.

[0127] Specifically, the scene detection unit 123 can perform such a scene selection process based on the priority rules in the priority information database, which is located in Fig. 9 is shown, and the specifications of the priority condition for each scene (which are in the scene information DB, which is in Fig. 4 is shown, defined and saved) to implement.

[0128] For example, if the scene detection unit 123 detects a scene in which an elephant makes a walking sound (a scene of a walking elephant) and a scene in which a horse makes a walking sound (a scene of a walking horse), the parameter extraction unit 125 prioritizes the scene of the walking elephant according to the rule "priority for greater amplitude in the low-frequency band". As a result, during content playback (for example, in the virtual space), vibrations that reproduce vibrations caused by a walking elephant and are also primarily perceived in the real world are applied to the user, allowing the user to experience a vibration sensation that evokes a vivid, realistic feeling, i.e., one that is close to reality.

[0129] If the scene detection unit 123 detects several scenes that are different from each other and overlap in time, and in which targets that produce noise are present, the parameter extraction unit 125 can apply a procedure whereby a parameter is extracted that corresponds to a scene which is selected from the multiple scenes based on the type and location of a target corresponding to each of the multiple scenes in the images contained in the content.

[0130] Specifically, the scene detection unit 123 can implement such a scene selection process by applying the priority rule specifications in the priority information database, which is located in Fig. 9 is shown, and the priority condition for each scene (which is in the scene information DB, which is in Fig. 4 is shown, is set and stored) (in the present example, a function value F(M,d) of the type (m) of the target and the distance (d) to the target is added to the priority conditions and the condition by the function value F(M,d) (for example, a larger function value F(M,d) is given priority) is added to the priority rules).

[0131] Using a concrete example found in Fig. Figure 14 shows a procedure for determining a scene with a higher priority based on the position of the target. Fig. Figure 14 is a diagram showing an example of a procedure for determining a higher priority objective.

[0132] As in Fig. As shown in Figure 14, the display device 3 is assumed to show an image 31 of the reproduced content. The image 31 contains an object 311 (a horse) and an object 312 (an elephant). In this case, it is assumed that the scene detection unit 123 has detected both a scene of a walking horse and a scene of a walking elephant, which meet the conditions as target scenes for vibration control.

[0133] It is assumed that the distance from the reference position (the user position for the content image, for example, the position of the avatar corresponding to the user in the XR content) to object 311 is L1. Conversely, it is assumed that the distance from the reference position to object 312 is L2. The reference vibration amplitudes of objects 311 and 312 (the amplitudes of the low-frequency components of the objects' audio signals in the content) are V1 and V2, respectively. Furthermore, the priority condition is set, for example, to "priority for the larger value of the function F(Ln,Vn) = Vn / (Ln, Ln)".

[0134] The distance from the reference position to the object is calculated, for example, based on information added to the content (e.g., based on the positional information of each object used for video generation in the XR content). The reference vibration amplitudes of the objects can be determined by reading from a data table containing a predefined reference vibration amplitude for each object type, according to the type of target object, or by adding them as content information. Since audio data for audio playback is often added to the content, the reference vibration amplitude can be determined based on the low-frequency characteristics (audio level, low-frequency signal level, etc.).) of the audio data are calculated (vibration patterns show a high correlation with the low-frequency components of a sound, and vibrations are often generated based on the low-frequency components of the sound).

[0135] In this way, the information processing device 10 can determine the low-frequency characteristics of a noise generated by a vibration-generating target within the content. In this case, the information processing unit 10 selects a vibration-generating target based on the determined low-frequency characteristics. This design enables the appropriate selection of a vibration-generating target.

[0136] An example of the low-frequency characteristics of a noise is a low-frequency signal level. In this case, the information processing device 10 selects a vibration-generating target whose determined low-frequency signal level exceeds a threshold. The information processing device 10 can extract the low-frequency signal level from the audio data. This design enables the simple selection of a vibration-generating target using the low-frequency signal level contained in the audio data.

[0137] The threshold for the low-frequency signal level is determined according to the type of content. As described above, it is often more effective to generate vibrations in music videos compared to wildlife documentaries, even with the same objective. This allows for the selection of a vibration target suitable for the type of content (music video, wildlife documentary, etc.).

[0138] In this case, if the relationship between the function values ​​of object 311 (horse) and object 312 (elephant) is F(L1,V1) > F(L2,V2), the scene in which object 311 produces a sound (vibrations), i.e., the scene of the walking horse, is preferably selected, and the parameter extraction unit 125 extracts a vibration parameter corresponding to the scene of the walking horse. Vibrations corresponding to the scene of the walking horse are then applied to the user. Subsequently, if, for example, object 312 (elephant) reaches the reference position and the relationship changes to F(L1,V1) < F(L2,V2), the scene in which object 312 produces a sound (vibrations), i.e., the scene of the walking elephant, is preferably selected, and the parameter extraction unit 125 extracts a vibration parameter corresponding to the scene of the walking elephant.Then, vibrations are exerted on the user that correspond to the scene of the walking elephant.

[0139] If the function F(Ln,Vn) is smaller than a predetermined threshold, that is, if the vibrations caused by the target at the user's position in the content (the virtual space of a game, etc.) are small (the user barely perceives any vibrations, meaning there is hardly any need to generate vibrations), then it is also effective not to choose any target for which vibrations are generated. In other words, it is also effective to choose only the object in the content for which vibrations caused by the object at the user's position in the content (the virtual space of a game, etc.) are of a certain magnitude (to such an extent that the user can feel an improvement in the realistic sensation when vibrations are reproduced), as the target for which vibrations are generated.In other words, from among the candidate targets that represent candidates for the vibration-generating target, the target that has a large impact on a vibration signal to be generated (that is, the vibration target whose vibrations are strongly felt by the user) is chosen.

[0140] This embodiment allows the information processing device 10 to identify, from among candidate targets representing potential vibration-generating targets, a candidate target that has a significant impact on the vibration signal to be generated, and to select this candidate target as the vibration-generating target. As a result, vibrations are exerted on the user that correspond to the user's sensations in real space, thus enabling a reproduction of the content with a vivid, realistic sensation.

[0141] In this case, it is advantageous to modify the threshold for selecting a target for generating vibrations based on the nature of the content. This is because, depending on the specifics of the content, it may be advantageous to refrain from reproducing vibrations altogether or to amplify vibrations caused by an object present in the content. Furthermore, it is advantageous to adjust a selection content (a selection level) for a target for which vibrations are generated.

[0142] More precisely, the principle for vibration generation is as follows. Based on the details of the content, a target for which vibrations are generated is determined within the content (in each scene thereof). Then, based on an acoustic signal (from audio data of the target contained in the content, or audio data of the target generated from audio data in the scene (for example, extracted by filtering the low-frequency domain)), a vibration signal (vibration data) is generated (for example, by extracting and appropriately amplifying the low-frequency component of the target's audio signal) that corresponds to the specified target.

[0143] A method for determining a target for which vibrations are generated includes determining the low-frequency characteristics (for example, the loudness level) of the generated noise at noise-generating objects in the content (in the case of the example above, determining based on the reference vibration intensity based on the type of objects and the distance between the reference position (such as the position where the user is located in the virtual space of the content) and the object); and determining a target (determining a noise-generating object with the higher low-frequency loudness level of the generated noise as the target for which vibrations are generated).

[0144] In this way, a scene with a higher priority is determined based on the position of the target, thereby exerting vibrations on the user that are better attuned to the user's visual intuition, that is, vibrations that correspond to the user's sensation in the real space, in order to enable a reproduction of the content with a vivid, realistic sensation.

[0145] In this case, the vibration parameter extraction unit 125a extracts a corresponding vibration parameter for each vibration device 5. This process can further improve the realistic sensation compared to a uniform extraction of vibration parameters.

[0146] The audio gain parameter extraction unit 125b references the parameter information database 134 and extracts an audio gain parameter corresponding to the scene designated with the highest priority by the priority setting unit 124. The audio gain parameter extraction unit 125b extracts the audio gain parameter for each loudspeaker 4 individually, determining the audio gain parameter to be extracted in the same way as the vibration parameter extraction unit 125a, based on the priorities designated by the priority setting unit 124 (based on the scene with the highest priority).

[0147] Learning Unit 125c learns the relationship between scenes and realism parameters stored in Parameter Information DB 134. For example, Learning Unit 125c learns the relationship between scenes and realism parameters through machine learning, using each scene stored in Parameter Information DB 134, its corresponding realism parameters, and the user's response to controlling the realistic sensation through these parameters as training data.

[0148] In this case, learning unit 125c can, for example, use user ratings of realism parameters (user adjustments after controlling the realistic sensation or user input such as questionnaires) as training data. In other words, learning unit 125c can learn the relationship between scenes and realism parameters from the perspective of what kind of realism parameters should be set and for which scene to achieve a high user rating (that is, whether a high level of realistic sensation was achieved).

[0149] Furthermore, learning unit 125c can determine, based on training results, which type of realism parameters should be set when a new scene is entered. As a concrete example, the realism parameters for a fireworks scene can be determined using the training results from controlling realistic sensation for similar situations, such as an explosion scene. Additionally, rules regarding priorities can be learned based on the presence or absence of factors that can change priorities, during user customization activities following a control of realistic sensation, or during user input such as question catalogs (for example, when the user customization activity assigns parameters to those of other scenes present at the same time, or when the question catalogs contain an answer indicating that other scenes should be given priority).

[0150] This design allows the information processing device 10, for example, to automatically optimize the rules for priorities and the realism parameters.

[0151] Returning to the description of Fig. The output unit 126 will be described in section 5. The output unit 156 outputs the realism parameters extracted by the parameter extraction unit 125 to the loudspeaker 4 and the vibration device 5.

[0152] Fig. Figure 16 is a block diagram of output unit 126. As in Fig. As shown in Figure 16, the output unit includes an audio amplification process unit 126a, an audio / vibration conversion process unit 126b and a vibration localization process unit 126c.

[0153] The audio amplification processing unit 126 performs an amplification process using the audio amplification parameters extracted by the parameter extraction unit 125 from the audio data obtained by the rendering processing unit 122. For example, the audio amplification processing unit 126a performs an amplification process on the audio data by applying delay or band gain / attenuation processing based on the audio amplification parameters.

[0154] In this case, the audio amplification processing unit 126a performs an audio amplification process for each loudspeaker 4 and outputs the audio data that has undergone the audio amplification process to each corresponding loudspeaker.

[0155] The audio / vibration conversion processing unit 126b converts the audio data received from the rendering processing unit 122 into vibration data by performing LPF or other bandwidth-limiting processing suitable for vibrations.

[0156] The vibration localization process unit 126c performs a vibration localization process on the vibration data obtained through conversion by the audio / vibration conversion process unit 126b. The vibration localization process unit 126c then outputs the vibration data for each vibration source, which has undergone amplitude and delay processing. The vibration device 5 vibrates each vibration source according to the vibration data output by the vibration localization process unit 126c.

[0157] With reference to Fig. 17 The vibration localization process method will be described by the vibration localization process unit 126c. Fig. Figure 17 is a diagram showing an example of the vibration localization process method.

[0158] As in Fig. As shown in Figure 17, the vibration localization process unit 126c first identifies the directional component of the sound to be presented to the user (the viewer of the content) (step S11). Since the sensation of vibration localization is based on the position of an object representing a vibration source, specifically, the position of presence (the direction of presence) of the object (the vibration source) is determined from the directional component of the sound originating from the same object, and from the determined position and the user's position (the user's position in the content space), the directional component of the vibrations is determined (identified).

[0159] Although the content may contain multiple vibration sources (objects) that can be offered to the user, for the sake of clarity, a description of a process will be given for a case in which a primary vibration source (an object expected to have the greatest effect in enhancing a realistic sensation) is selected using the above procedure. Similar processes can be performed in parallel for multiple vibration sources to effectively offer the user vibrations based on several sources and to play the content with a vivid, realistic sensation.

[0160] Therefore, the direction 52 of a virtual vibration source with respect to the user (in which a state of vibrations generated by this virtual vibration source is represented) is a direction that indicates the direction from the user to the target, which represents a noise source in the XR space (the virtual space), that is, the directional component of the noise.

[0161] The vibration localization process unit 126c can identify the directional component of the noise (the vibrations), for example, in the same way as in the noise localization process based on the position data of the noise source (the position of the target object) obtained from the rendering process unit 122.

[0162] For example, the vibration localization process unit 126c can identify the position of the target (the noise source) based on the spectrum of an audio signal from each of several channels contained in the audio data, and, based on the identified position, identify the directional component of the noise (the vibrations).

[0163] The vibration localization process unit 126c can identify the directional component of the noise (the vibrations) based on the content metadata (metadata that includes data specifying the position of the target object).

[0164] In other words, the content developed using a 3D machine contains, in addition to data on the sound produced by an object at any given time, information indicating the object's position in the virtual space.

[0165] For example, in the case of a scene of a walking horse, the scene data contains data regarding the horse's steps and data regarding the horse's position (as metadata). The vibration localization process unit 126a uses the horse's position data to identify the position of the sound source (the horse's position as the sound source) for the horse's steps.

[0166] Then the vibration localization process unit 126c identifies the direction from the user position in the virtual space to the position of the sound source for the horse's steps (the position of the horse) as the directional component of the sound and determines the identified directional component as the directional component of the vibrations (the localization sense direction).

[0167] Alternatively, the directional component of the noise can be identified by subjecting an image in the content to image recognition, thereby identifying a target object for the noise source and its position, and the identified directional component of the noise can be determined as the directional component of the vibrations (the localization sense direction).

[0168] Then the vibration localization process unit 126c determines various data for processing, such as coefficient values ​​and correction values, which are used for the vibration control (the vibration data generation process (vibration signal generation process)) of each vibration generator 51 of the vibration device 5.

[0169] For example, each vibration generator 51 exhibits individual differences with regard to its properties (the relationship between the input signal and the vibration output, for example, the ratio between the input signal level and the vibration output level), and correction data are determined to correct for these differences in properties. Since the output vibration level has a significant influence, the vibration generator property data in the present embodiment are specifically determined based on the ratio (in amplitude) between the input signal level and the vibration output level (hereinafter referred to as vibration generator sensitivity).

[0170] The vibration generator sensitivity data can be calculated by measuring a vibration amplitude when a test vibration signal is applied to the vibration generator and using the test vibration signal amplitude and the vibration amplitude, whereby the calculated data are stored and used in the storage unit 130 (the vibration generator information DB 135).

[0171] The vibration localization process unit 126c also determines sensitivity property correction data to correct differences in sensitivity properties, which represent properties of the sensation experienced by the user in response to the vibrations, and sensitivity property correction data to correct differences in the properties of vibration transmission to the user due to the contact state between the user and each vibration generator.

[0172] One of the sensitivity property correction data refers to data used to correct for differences in how vibrations are perceived due to individual user variations or differences in body parts. The vibration localization process unit 126c determines the sensitivity properties, which are the characteristics of the intensity of a vibration sensation, as sensitivity property correction data.

[0173] The sensitivity characteristics can be determined, for example, through user input before viewing the content. Specifically, the sensitivity characteristics can be measured, for instance, by providing the user with vibrations of a predetermined intensity from each vibration generator and allowing the user to input their perception of the vibrations.

[0174] Other sensitivity property correction data are data for correcting the difference in how vibrations are perceived due to the contact state between the user and each vibration generator 51. In the present embodiment, this is the pressure experienced by each vibration generator when the user is seated, which has a significant influence on the intensity characteristics of a vibration sensation; that is, pressure distribution data on the seat surface when the user is seated. The vibration localization process unit 126c determines the pressure value in a section where each vibration generator 51 is installed in the seat surface as sensitivity property correction data.

[0175] The pressure value can be measured by installing a pressure sensor on the seat surface of the seat on which the user sits while viewing the content.

[0176] Alternatively, the sensitivity property correction data can be determined as a combination of the sensitivity properties and the pressure value, for example by providing the user with vibrations of a predetermined strength from each vibration generator and allowing the user to input their sensation towards the vibrations.

[0177] Since the sensitivity property correction data are correction data according to the user's characteristics and state (the state in which the user is seated), the relationship between the vibration level (the level of vibrations when a vibration signal is input into a vibration generator determined to have standard characteristics) and the user's perception (the vibration level) is hereinafter referred to as user sensitivity. This user sensitivity is then stored in memory unit 130 (the vibration generator information DB 135) and used during content playback.

[0178] The vibration localization process unit 126c then calculates an output level correction value for each vibration source using the vibration source sensitivity and the user sensitivity, and stores the calculated output level correction value in the memory unit 130 (the vibration source information DB 135). Specifically, the vibration localization process unit 126c stores the reciprocal of the sum of the vibration source sensitivity and the user sensitivity as the output level correction value in the memory unit 130 (the vibration source information DB 135).In other words, each vibration generator oscillates based on vibration data, and the vibration perception property, which indicates how the user perceives the vibrations (what vibration level he perceives) (the relationship between the vibration data and the user's perception of the vibration intensity, taking into account the characteristic factors (properties) of the vibration generator), is stored as an output level correction value in memory unit 130.Since vibration generator sensitivity is the ratio between the vibration signal level and the vibration level (amplitude), and user sensitivity is the ratio between the vibration level (amplitude) and user perception, the user will perceive the same degree of vibration from any vibration generator after a correction processing (division) with the output level correction value corresponding to the respective vibration generator. In the example shown on page 12... Fig. As shown in Figure 17, an output level correction value 61A is calculated for each vibration generator as “2” for the vibration generator 51FL, as “4” for the vibration generator 51FR, as “1” for the vibration generator 51RL and as “3” for the vibration generator 51RR.

[0179] The vibration localization process unit 126c then performs signal processing (step S13). The signal processing is carried out with reference to Fig. 18 will be described. Fig. Figure 18 is a diagram illustrating the basic principle of the signal processing method in the present embodiment. This processing method is implemented by the control unit 120 (the CPU) executing calculation formulas and processing programs based on this principle. For clarity, this is an example of processing in a two-dimensional horizontal space within a content playback space (where the processing ignores the vertical direction). Since most of the content is spread largely on the plane of a vibration target (the ground), and the direction of motion is also mostly on the plane (the ground), processing in a two-dimensional space is sufficient for an approximation.

[0180] As in Fig. As shown in Figure 18, the vibration localization process unit 126c first plots the position of each vibration source in the coordinate space for the calculation (step S21). In other words, the vibration localization process unit 126c plots points that correspond to the position coordinate data of the vibration sources obtained from the vibration source information DB 135 (point 53_FL, point 53_RL, point 53_FR, point 53_RR).

[0181] The vibration localization process unit 126c then calculates the center point of the plotted position coordinate points of the vibration generators (intermediate coordinates of the X and Y coordinate values ​​of the points). The vibration localization process unit 126c draws lines 535a, 535b, 535c, and 535d (non-diagonal lines) that connect the position coordinate points of the vibration generators and form the perimeter of a polygon (a quadrilateral). The vibration localization process unit 126c also draws a line 525 that passes through the center point and extends in the direction 52, which was defined in step S11. Fig. 17 was determined, extends.

[0182] Then the vibration localization process unit 126c plots the intersection points (points 531 and 532) of line 525 and lines 535a and 535c (step S22).

[0183] Of the line segments whose intersections were plotted at step S22, the line segment (line 535a) that passes through point 531 on the side of direction 52 is designated as the non-deceleration-side line segment (535a). The endpoints of the non-deceleration-side line segment 535a (points 53_FL and 53_FR) are designated as non-deceleration-side points (53_FL and 53_FR).

[0184] Of the line segments whose intersection points are drawn at step S22, the line segment (line 535c) that passes through point 532 on the side opposite direction 52 is designated as the delay-side line segment (535c). The endpoints of the delay-side line segment 535c (points 53_RL and 53_RR) are designated as delay-side points (53_RL and 53_RR).

[0185] The figure, consisting of lines connecting the position coordinate points of the vibration generators (lines forming the outer perimeter), is not necessarily a quadrilateral, but can be a polygon other than a quadrilateral or a polyhedron. In other words, the figure is a polygon corresponding to the number of vibration generators to be controlled (for example, a pentagon if five vibration generators are to be controlled). From the sides of the polygon, the line 525, which passes through the center of the polygon and extends in the direction 52, as described in step S11 of Fig. As determined in section 17, extending and intersecting, the side of direction 52 is the non-deceleration-side line segment, and the side of the direction opposite to direction 52 is the deceleration-side line segment. The endpoints of the non-deceleration-side line segment are the non-deceleration-side points, and the endpoints of the deceleration-side line segment are the deceleration-side points.

[0186] If from the sides of the polygon, the line 525, which runs through the center of the polygon and points in the direction 52, which was at step S11 of Fig. Since the line segments were determined to be extended, intersecting, and selected from the non-delay-side and delay-side line segments, the selected line segments are not adjacent to each other. As described below, this process prevents a vibration source from being subjected to both non-delay-side vibration control and delay-side vibration control, thus simplifying the control calculation. Furthermore, this process also prevents a vibration source from performing the functions of two vibration sources, thereby increasing control accuracy. Because the center of the surface has the same relationship with each side of the polygon, the same processing is applied to vibration sources in any direction. As a result, the creation of a processing program can be expected to be easy.

[0187] Subsequently, the vibration localization process unit 126c performs a process related to the control of vibration perception positions, based on the technical concept of phantom sensation (Phs). Phantom sensation is defined as follows: “When the same stimulus (for example, vibrations) is applied simultaneously to two points, a sensation of stimulation is felt at the midpoint between the two points. If the magnitudes (for example, the amplitudes in the case of vibrations) of the stimulus are different, the point at which the sensation of stimulation is felt (hereinafter referred to as the stimulus perception point) moves to the point to which the greater stimulus is applied.” The position of the stimulus perception point is determined by an approximation such that the distance ratio from each stimulus point is inversely proportional to the ratio of the stimulus intensity (in the case of vibrations, the amplitude) at each stimulus point.In the present embodiment, the vibration stimulus for the user is controlled on the basis of this concept.

[0188] Furthermore, the vibration localization process unit 126c performs a process related to controlling a sense of vibration direction, based on the technical concept of perceptible apparent motion. Perceptible apparent motion is "the creation of a sense of motion in vibrations by providing a time difference between vibrations at two points."

[0189] In other words, the vibration localization process unit 126c, based on the concepts of phantom sensation and perceptible apparent movement, performs a process to generate vibrations that provide the user with a sense of localization with a vivid, realistic sensation.

[0190] Specifically, points 531 and 532, which are found in step S22 in Fig. The points shown in Figure 18 are used as vibration perception positions, and a time difference in the vibration time between points 531 and 532 is provided so that the user gets a sense of the movement of the vibrations between points 531 and 532. In other words, the user perceives localized vibrations moving in the direction of the line connecting points 531 and 532, that is, towards the position of the vibration source (the vibration-generating object in the content).

[0191] First, the process will be described in detail based on the technical concept of phantom sensation, using a concrete example.

[0192] The vibration localization process unit 126c performs the process based on points 531 and 532 as stimulus perception points. Based on the technical concept of phantom sensation, the stimulus perception point is point 531 if the ratio between the distance from point 531 to the vibration source position 53_FL (L1) and the distance from point 531 to the vibration source position 53_FR ​​(L2) is the inverse of the ratio between the amplitude generated by vibration source 51_FL and the amplitude generated by vibration source 51_FR. Therefore, the vibration localization process unit 126c calculates L2 / (L1 + L2) as the correction value 601AFL (correction value added to the vibration signal) for vibration source 51_FL. The vibration localization process unit 126c also calculates L1 / (L+1 + L2) as a correction value 60AFR for the vibration generator 51_FR.

[0193] The vibration localization process unit 126c then performs the same processing for the process with point 532 as the stimulus perception point in order to calculate L4 / (L3 + L4) as the correction value 60ARL for the vibration source 51_L. The vibration localization process unit also performs the same processing for stimulus perception point 532 in order to calculate L3 / (L3 + L4) as the correction value 60ARR for the vibration source 51_RR. L3 is the distance from point 532 to the vibration source position 53_RL, and L4 is the distance from point 532 to the vibration source position 53_RR.

[0194] By adding the correction value 60AFL to vibration data VD generated by the procedure described above, a vibration signal is generated and then output to the vibration generator 51_FL. Similarly, by adding the correction value 60AFR to vibration data BD, a vibration signal is generated and then output to the vibration generator 51_FR, shifting the user's stimulus perception point to point 531. Likewise, by adding the correction value 60ARL to the vibration data VD, a vibration signal is generated and then output to the vibration generator 51_RL. Similarly, by adding the correction value 60ARR to the vibration data BD, a vibration signal is generated and then output to the vibration generator 51_RR, shifting the user's stimulus perception point to point 532.

[0195] If the distances L1, L2, L3 and L4 are 3k, 2k, 2k and 3k respectively, the correction value for the vibration generator 51_FL is 3k / (3k + 2k) = 0.6 and the correction value for the vibration generator 51_FR is 2k / (3k + 2k) = 0.4.

[0196] The correction value 60AFL for the vibration generator 51_RL is 2k / (3k + 2k) = 0.4, and the correction value 60AFR for the vibration generator 51_RR is 3k / (3k + 2k) - 0.6.

[0197] In fact, however, individual differences in the vibration generators, user sensitivity, and the condition in which the user sits on the vibrating seat cause an error in the strength of the vibrations felt by the user, resulting in user stimulus perception points at points 541 and 542.

[0198] The vibration localization process unit 126c then corrects the vibration signal for each vibration source using the output level correction value (61AFL, 61AFR, 61ARL, 61ARR) for each vibration source (51_FL, 51_FR, 51_RL, 51_RR) previously determined by the process, which refers to Fig. As described in section 17, it was calculated.

[0199] Specifically, the vibration localization process unit 126c defines the sum of the vibration data VD, the correction value 60ARL, and the correction value 61AFL as vibration data for the vibration generator 51_FL and outputs a vibration signal to the vibration generator 51_FL. Similarly, the vibration localization process unit 126c defines the sum of the vibration data VD, the correction value 60AFR, and the correction value 61AFR as vibration data for the vibration generator 51_FR and outputs a vibration signal to the vibration generator 51_FR. The vibration localization process unit 126 defines the sum of the vibration data VD, the correction value 60ARL, and the correction value 61ARL as vibration data for the vibration generator 51_RL and outputs a vibration signal to the vibration generator 51_RL.The vibration localization process unit 126 defines the sum of the vibration data VD, the correction value 60ARR and the correction value 61ARR as vibration data for the vibration generator 51_RR and outputs a vibration signal to the vibration generator 51_RR.

[0200] If the correction values ​​61AFR, 61AFL, 61ARL and 61ARL are as in step S12 in Fig. Figure 17 shows 2, 4, 1 and 3 respectively, and the distances L1, L2, L3 and L4 are 3k, 2k, 2k and 3k respectively. The vibration data 53DFL, 53DFR, 53DRL and 53DRR of the vibration signals output to the vibration generators 51_FL, 51FR, 51_RL and 51_RR are as follows, where the original vibration data is VD and vibration signals based on this vibration data 53D are output to the corresponding vibration generators 51. 53DFL=VD⋅3k / (3k+2k) / 2=0.3⋅VD 53DFR=VD⋅2k / (3k+2k) / 4=0.1⋅VD 53DRL=VD⋅2k / (2k+3k) / 1=0.4⋅VD 53DRR=VD⋅3k / (2k+3k) / 3=0.2⋅VD

[0201] As a result, the error caused by individual differences in the vibration generators and the user's sensitivity is corrected, and the user's stimulus perception point moves from the positions at points 541 and 542 to the target positions at points 531 and 532, as in step S24 in Fig. 18 is shown.

[0202] In this way, the vibration localization process unit 126c controls the amplitude and delay of the output vibrations of the individual vibration generators based on the arrangement of the vibration generators.

[0203] This design allows the amplitude to be controlled according to the actual arrangement of the vibration generators, enabling the user to experience a more natural feeling of vibration localization.

[0204] The vibration localization process unit 126c controls the amplitude and delay of the output vibrations of each vibration generator based on the user's vibration perception characteristic relative to the output vibrations of each vibration generator.

[0205] For example, the information processing device 10 stores the sensitivity characteristics for each part of the user's body in advance. The vibration localization processing unit 126c then uses different sensitivity characteristics depending on whether the point in close contact with each vibration source is the left or right side of the user's body, or the thigh or buttocks.

[0206] In this way, the amplitude can be controlled according to the user's actual vibration perception; that is, the amplitude can be controlled taking into account the relationship between the vibration signal and the vibration sensation felt by the user. Accordingly, the way the user perceives the vibrations can be brought closer to what is intended by the design.

[0207] The property of vibration perception is the property with regard to individual differences between individual vibration generators and individual differences between individual users.

[0208] For example, the vibration perception property is determined based on the relationship between the input signal and the output vibration level of each vibration generator, the user's weight, physical condition, or posture, and the like. The vibration localization process unit 126c uses the vibration perception property to control the amplitude of each vibration generator.

[0209] This design can generate vibrations that take into account individual differences between the individual vibration generators and individual differences between individual users.

[0210] It can be difficult to determine the user's posture while viewing content through user input or similar means. The vibration localization process unit 126c, for calibration purposes, causes the vibration generators to oscillate while the user is in posture during content viewing and measures the user's vibration perception.

[0211] For example, the information processing device 10 instructs the user to assume a posture while viewing the content. The device then displays a sample content (for calibrating the vibration perception property for the user's posture) and generates vibrations for calibration. Based on user feedback or biometric information, the information processing device 10 determines a correction value for the vibration perception property related to the user's posture while viewing. The information processing device 10 then stores this determined correction value and uses it in the subsequent process of calculating the vibration perception property.

[0212] This design can provide vibrations that are precisely adapted to the posture of each user while viewing content.

[0213] Furthermore, the vibration localization process unit 126c corrects the amplitudes of all vibration generators corresponding to the scene, based on the amplitude gain factor obtained from the parameter information DB 134.

[0214] The process will now be described in detail based on the technical concept of perceptible apparent movement, using a concrete example.

[0215] The vibration localization process unit 126c calculates the delay time of the vibration generators corresponding to the delay-side points, in this example, vibration generators 51_RL and 51_RR (Δt: delay time from the vibration time of vibration generators 51_FL and 51_FR, corresponding to the non-delay-side points) (step S25). The delay time of vibration generators 51_FL and 51_FR, corresponding to the non-delay-side points, from the time of vibration generation (noise generation from the same target object) is set to 0. However, in some scenarios, it may be effective to delay or speed up the vibration generation time of vibration generators 51_FL and 51_FR, corresponding to the non-delay-side points.

[0216] Although realistic perception can also be improved by a delay time of a predetermined fixed duration, it may be more effective to calculate the delay time using, for example, the following formula. Delay time Δti=ai⋅yi⋅Y(i is a respective point in time)

[0217] Here, `ai` is a value representing whether a delay should be applied, set to 1 if delay processing is required and to 0 if it is not. Additionally, `yi` is a vibration gain factor, a value used to appropriately amplify the vibrations generated by a vibration-producing target object in a target scene of the content. For example, a large gain factor of `y1` indicates that vibrations should be greatly amplified, and the delay time is also increased according to the degree of amplification to make the difference more perceptible. Essentially, `y` is a value representing the time vibrations take to travel the distance between points 531 and 532, which represent stimulus perception points. Using a suitable constant is advantageous to reduce the processing load.The constant for Y can be an intermediate value of the distance between points 531 and 532 (the average of the distance between vibration generator positions 53_FL and 53_RL and the distance between vibration generator positions 53_FR ​​and 53_RL) or a value determined to be suitable by sensitivity tests or the like.

[0218] The value ai, which indicates whether a delay should be applied, and the gain factor yi are values ​​that are determined according to the state of the vibration-generating target object in the target scene of the content, and are determined, for example, based on the results of an image analysis of the content images, the analysis results of the noises of the content (especially the noise generated by the vibration-generating target object), or additional information regarding the content (which was added to the content beforehand as control data).

[0219] The vibration localization process unit 126c calculates the delay time Δti for each scene using this formula and outputs a corresponding vibration signal, based on the calculated delay time Δti, to the individual vibration generators 51 at each time point in order to make the vibration generators 51 vibrate (step S25).

[0220] If the calculated delay time Δti in the vibration data example above is, for example, one second, the respective vibration generators 51 oscillate as follows: Vibration generator 51_FL: Vibration data 0.3·VD, vibration time 0 seconds (delay time from the playback time of the target scene in the content; the same applies to the following vibration generators) Vibration generator 51_FR: Vibration data 0.1·VD, vibration time 0 seconds Vibration generator 51_RL: Vibration data 0.4·VD, vibration time 1 second Vibration generator 51_RR: Vibration data 0.2·VD, vibration time 1 second

[0221] As a result, the user perceives the vibrations at point 531 at the playback time of the target scene in the content, while they feel the vibrations at point 532 one second later, thus perceiving the vibrations along the direction from the vibration source. This allows the user to experience a sense of localization of the vibration source (the vibration-generating object) and enjoy the content playback with a vivid, realistic sensation.

[0222] A modification for the delayed drive of the vibration generators 51 will now be described. In the process example above, the vibration localization process unit 126c carried out a process in which the vibrations were moved discontinuously from a vibration generator corresponding to a non-delay-side point to a vibration generator corresponding to a delay-side point, whereas the present example creates the sensation that the vibration position is moving gradually. This process is also based on the technical concept of phantom sensation.

[0223] Specifically, the vibration localization process unit 126c dampens the amplitudes of the vibration generators 51_FL and 51_FR, which correspond to the non-delayed-side points, over the delay time Δt. Conversely, the vibration localization process unit 126c increases the amplitudes of the vibration generators 51_RL and 51_RR, which correspond to the delayed-side points, over the delay time Δt.

[0224] If the delay time Δti in the vibration data example above is, for example, one second, each vibration generator will oscillate after T seconds (T being the delay time Δti (one second) or less) based on the following vibration data. Vibration generator 51_FL: Vibration data 0.3·VD·((1- T / 1) Vibration generator 51_FR: Vibration data 0.1·VD·((1- T / 1) Vibration generator 51_RL: Vibration data 0.4·VD·(T / 1) Vibration generator 51_RR: Vibration data 0.2·VD·(T / 1)

[0225] It is also effective to set the final attenuation values ​​for vibration generators 51_FL and 51_FR, corresponding to the non-delay points, to a moderately low noise level instead of a noiseless level, or to set the final attenuation values ​​before the delay time Δt has elapsed from the playback time of the target scene in the content. It is also effective to start the increase in amplitudes for vibration generators 51_RL and 51_RR, corresponding to the delay points, from a moderately low level instead of a silent level, or to set attenuation start values ​​after a predetermined time (the delay time Δt or less) has elapsed from the playback time of the target scene in the content.

[0226] In this way, the present embodiment allows the user viewing the content to experience, based on the technical concepts of phantom sensation and perceptible apparent movement, a sense of vibration source position and a sense of vibration movement appropriately based on the details of the content, so that the user can enjoy the reproduction of the content with a vivid, realistic sensation.

[0227] These procedures can be described as follows: “The amplitude of the output vibrations of the individual vibration generators is controlled such that the positional relationship (the line that defines points 551 and 552) between a first composite vibration position (point 551), which is determined on the basis of the vibration level of each vibration generator in a first group of several vibration generators (the vibration generators 51_FL and 51_FR), and a second composite vibration position (point 552), which is determined on the basis of the vibration level of each vibration generator in a second group of several vibration generators (the vibration generators 51_FR and 51_RR), corresponds to a directional component (the direction 52) of the identified vibration source;and a delay of the output oscillations of the individual vibration generators (the oscillation time of the vibration generators 51_FL and 51_FR (a delay 0 from the occurrence of the vibrations in the content) and the oscillation time of the vibration generators 51_RL and 51_RR (for example a delay Δt from the occurrence of the vibrations in the content)) is controlled according to the directional component of the vibration source".;

[0228] With reference to Fig. Section 19 will now describe a process flow that is carried out by the information processing device 10 according to the embodiment. Fig. Figure 19 is a flowchart showing the process sequence executed by the information processing device 10. The process sequence described below is executed repeatedly by the control unit 10.

[0229] The process in the flowchart, which is in Fig. The process shown in Figure 19 is executed repeatedly when the information processing device 10 is in a powered-on state. At the beginning of the process, it is determined whether an actuation to start the playback of XR content is present, and if the start actuation is detected (Yes at step S101), the process proceeds to step S102, while otherwise (No at step S101), the process terminates.

[0230] First, an XR content setting process is executed (step S102). Here, the XR setting process includes, for example, various processes related to initial settings of devices for playing back the XR content and the user's selection of the XR content.

[0231] The information processing device 10 then begins playing back the XR content (step S103) and performs a scene detection process on the played-back XR content (step S104). Next, the information processing device 10 performs a priority setting process on the result of the scene detection process (step S105) and executes a realism parameter extraction process (step S106).

[0232] The information processing device 10 then performs an output process for various vibration data or audio data reflecting the process result of the realism parameter extraction process (step S107). The information processing device 10 then determines whether the XR content has ended (step S108), and if it is determined that the XR content has ended (YES in step S108), the process ends.

[0233] If the information processing device 10 determines in step S108 that the XR process has not finished (No in step S108), the process proceeds again to step S104.

[0234] With reference to Fig. Section 20 will describe the process of vibration localization. Fig. Figure 20 is a flowchart showing the sequence of the vibration localization process. The vibration localization process corresponds to the process executed by the vibration localization process unit 126c (of the control unit 120). This process is also part of the process at steps S107 and S108 in the process described in Fig. Figure 19 illustrates the process details of each step. The specific process details are as described above.

[0235] First, the vibration localization process unit 126c identifies as in Fig. 20 shows the directional component of a noise (vibrations) (step S201).

[0236] The vibration localization process unit 126c then determines a correction value CI to correct any vibration level difference perceived by the user due to individual differences in the individual vibration sources, individual differences in the user, and the state in which the user views the content (step S202). The correction value CI is determined and stored before the content is played (for example, when the user has sat down), and the stored correction value CI is read at step S202.

[0237] Subsequently, based on the technical concept of phantom sensation, using the directional component of the noise determined in step S201 and the setup position information for each vibration generator, the vibration localization process unit 126c calculates a correction value FS to correct the vibration level of each vibration generator (step S203).

[0238] The vibration localization process unit 126c then determines an (added) correction value CV for correcting the vibration data for each vibration generator from the correction value CI for correcting the effects of individual differences between vibration generators and the like, which was determined in step S22, and the correction value FS based on the technical concept of phantom sensation, which was determined in step S203. The vibration data, which was determined separately based on the content, is then subjected to a correction (an addition) with the determined correction value CV for the vibrations of the respective vibration generator in order to determine output data for each vibration generator (step S204).

[0239] Then, based on the technical concept of perceptible apparent motion, the vibration localization process unit 126c calculates the vibration time of each vibration generator (the time of output of the vibration signal to each vibration generator). In other words, the delay time from the time of vibration generation (in the present embodiment, the time of sound generation, since the vibration signal is generated based on an audio signal) of the vibration-generating object in the content scene is calculated (step S205).

[0240] Then the vibration localization process unit 126 provides the vibration data and the vibration timing data for the individual vibration generators as output data, and the output unit 126 outputs a signal to each vibration generator (step S107 in Fig. 19).

[0241] As described above, the vibration localization process unit 126c in the information processing device 10 according to the embodiment has several vibration generators, identifies the directional component of a vibration source in the input content and controls the amplitude and delay of the output vibrations of each vibration generator on the basis of the directional component.

[0242] Through such control, the information processing device 10 can provide the user with a sense of localizing a sound source (a sense of position) and a sensation of vibration transmission (a sense of vibration movement) by regulating the amplitude and delay of the output oscillations of the multiple vibration generators. In other words, the information processing device 10 conveys to the user a sense of localizing a sound source based on the relationship between the amplitudes of the output oscillations of the vibration generators and a sense of vibration movement based on the difference in the timing of the output oscillations of the vibration generators. As a result, the information processing device 10 can provide the user with a vivid and realistic sensation of vibrations within the content. Second embodiment

[0243] A second embodiment simplifies the process of calculating the correction value based on the technical concept of phantom sensation so that, for example, even a relatively slow computing unit (such as a CPU) can support the process.

[0244] In short, the directional component of the noise (the vibrations) is identified in a suitable number of area units (in this example, eight angular ranges, i.e., eight levels), and the subsequent calculation of the correction value is performed by model processing for each angular range, for example, using a data table in which control values ​​for each angular range are stored, or using a calculation program designed for each angular range. This implementation simplifies the process and reduces the processing load for the computing unit.

[0245] With reference to Fig. Section 22 will describe a procedure to determine the amplitude and delay time in advance. Fig. Figure 22 is a diagram showing a method for determining the directional component of the vibrations.

[0246] The vibration localization process unit 126c (the control unit 120) determines to which angular range r1 to r8 the directional component of the vibrations in the content (determined in the present embodiment from a noise in the content) belongs, wherein the angular ranges r1 to r8 are eight subdivided parts of the user environment (step S31). Data defining the angular ranges were pre-stored (for example, at the time of design) in the memory unit 130, and the angular range for the directional component of the vibrations is determined using the stored data. In the present embodiment, the angular ranges r1 to r8 are each set to 45 degrees, as shown in step S31, with angular range r1 at the front 45 degrees serving as the reference.

[0247] The vibration localization process unit 126c then determines the direction (d1 to d8: referred to as the representative direction) that represents the midpoint of the angular range for which the directional component of the vibrations has been determined to belong, as vibration direction data to be used for calculating the correction value based on the technical concept of phantom sensation. This process can be implemented by pre-storing a data table in the memory unit 130, which represents the relationship between the angular ranges r1 and r8 and the representative directions d1 to d8, and then performing the comparison using the data in the data table.

[0248] For example, if the user's viewing angle is 0° and the angle is denoted as positive (clockwise), then the range r1 is from -30° to -30° and its representative direction d1 is 0°. The range r2 is from 30° to 60°, and its direction d1 is 45°. If the directional component of the vibrations is, for example, 45°, the vibration localization process unit 126c determines that the representative direction is d2, and the subsequent process uses d2. The information processing device 10 then performs signal processing using the data of the representative direction of the directional component of the vibrations, which was determined by the above method. This processing is similar to the method shown in FIGURE 17. In this case, there are only eight representative directions d, and in the present embodiment, correction values ​​based on the technical concept of phantom sensation are pre-calculated for these eight representative directions d1 to d8 and stored in the storage unit 130 in the form of a data table.

[0249] Furthermore, correction values ​​based on the technical concept of perceptible apparent motion are also pre-calculated and are aggregated correction values ​​(for example, the sum of the correction values ​​for a vibration level) based on the calculated correction values ​​and the correction values ​​determined as above based on the technical concept of phantom sensation. These are calculated and stored in the form of a data table in memory unit 130. In this case, a separate data table for correction values ​​based on the technical concept of phantom sensation is not required.

[0250] Instead of calculating the aggregated correction values, another possible method is to calculate correction values ​​based on the technical concept of phantom sensation using a table, to calculate correction values ​​based on the technical concept of perceptible apparent motion without using a table, and to correct vibration data using these correction values.

[0251] Fig. Figure 21 is a diagram illustrating the structure of a data table, showing an example of a data table for correction values ​​based on the technical concept of phantom sensation and the technical concept of perceptible apparent movement.

[0252] The data table stores correction values ​​for the vibration amplitude and correction values ​​for the vibration timing (the delay) for each vibration generator (51FL, 51FR, 51RL, 51RR) and for each representative direction (d1 to d8), which were calculated in advance (for example, at the time of design).

[0253] The vibration localization process unit 126c then extracts the correction values ​​for the amplitude and delay corresponding to the determined representative direction d for each vibration generator 51 from the data table and corrects the vibration data.

[0254] If the representative direction is, for example, direction d2, the amplitude correction value for vibration generator 51FL is -2 dB and the delay time is 0 ms; the amplitude correction value for vibration generator 51FR is +4 dB and the delay time is 0 ms; the amplitude correction value for vibration generator 51RL is +4 dB and the delay time is 50 ms; and the amplitude correction value for vibration generator 51RR is -4 dB and the delay time is 50 ms. The vibration data is then corrected with these correction values, and the corresponding vibration signal is output to the respective vibration generator.

[0255] The data table that is in Fig. As shown in Figure 21, factors that vary depending on the circumstances of viewing the content, such as user sensitivity or the user's position on the seat (pressure distribution on the seat surface), are excluded as factors for correction, as are seat types (for example, the vibration generators themselves or their arrangement). However, these variable factors can be added to the data table as parameters to enable control adapted to these variable factors.

[0256] In this way, the information processing device in the second embodiment determines to which of several predetermined angular ranges the directional component of the vibrations belongs, selecting from the models (of the data table in Fig. 21), which are individually defined for the angular ranges, a model that corresponds to the specified angular range of the direction component (a data record for the corresponding direction in the table in Fig. 21) corresponds, and controls the amplitude and delay of the output oscillations of each oscillator based on the selected model.

[0257] Specifically, the model has a data table (the data table in Fig. 21) in which amplitude correction values ​​for the amplitude and delay correction values ​​for the delay are stored for each of several angular ranges. The information processing device 10 controls the amplitude and the delay of the output oscillations of each vibration generator based on the amplitude correction value and the delay correction value for each vibration generator, which were previously stored in the data table according to the angular range of the directional component.

[0258] The second embodiment can therefore, instead of carrying out a complicated process using the directional component of the vibrations, control be carried out by a process that uses a model (a data table) that has been generated in advance according to the angular range to which the directional component of the vibrations belongs, thereby reducing the processing load, such as the computational effort.

[0259] In the description of the above embodiments, the content is XR content, but the embodiments are not limited to this. Specifically, the content can be 2D video and audio, video only, or audio only.

[0260] Experts can easily deduce additional effects and modifications. Therefore, the more general aspects of the present invention are not limited to the specific details and representative embodiments presented and described above. Thus, various modifications can be made without departing from the spirit or scope of the general inventive concept defined by the accompanying claims and their equivalents. Reference symbol list 1 Information processing system 3 Display device 4 speakers 5 Vibration device 10 Information processing device 31 image 121 Content generation unit 122 Rendering Processing Unit 123 Scene Detection Unit 123a Scene Determination Unit 123 Condition definition unit 124 Priority setting unit 124a Time-detection unit 124 rule-setting unit 125 parameter extraction unit 125a Vibration parameter extraction unit 125b Audio Gain Parameter Extraction Unit 125c Learning Unit 126 output units 126a Audio amplification processing unit 126b Audio / Vibration Conversion Process Unit 126c Vibration Localization Process Unit 131 XR content database 132 Scene Information Database 133 Priority Information Database 134 Parameter Information DB 311, 312 Object QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2004-081357

[0004] JP 2021-158392

[0004]

Claims

[1] An information processing device for applying vibrations corresponding to a content to a user by controlling a vibration device having a plurality of vibration generators, wherein the information processing device has a controller, wherein the controller identifies a directional component of an object representing a vibration source in an input content, and controls an amplitude and a delay of output vibrations of each of the vibration generators based on the directional component. [2] Information processing apparatus according to claim 1, wherein the controller determines to which of several predetermined angular ranges the directional component belongs, selects a model that corresponds to the specific angular range of the directional component from models that are individually specified for the angular ranges, and controls an amplitude and a delay of the output oscillations of each of the oscillators based on the selected model. [3] Information processing apparatus according to claim 2, wherein the model has a data table in which an amplitude correction value for the amplitude and a delay correction value for the delay are stored for each of the angular ranges, and the controller controls an amplitude and a delay of the output vibrations of each of the vibration generators based on the amplitude correction value and delay correction value stored in the data table, which correspond to the angular range of the directional component. [4] The information processing apparatus according to claim 1, wherein the controller controls an amplitude and a delay of the output vibrations of each of the vibration generators based on an arrangement of the vibration generators. [5] The information processing apparatus according to any one of claims 1 to 4, wherein the controller controls an amplitude and a delay of the output vibrations of each of the vibration generators based on the vibration perception characteristic of the user to the output vibrations of each of the vibration generators. [6] The information processing apparatus according to claim 5, wherein the controller calculates the user's vibration perception characteristic based on the characteristics of the vibration generator and a state of the user. [7] The information processing apparatus according to claim 5, wherein the controller measures the user's vibration perception property by vibrating the vibrator for calibration in a state of the user's posture when viewing a content. [8] Information processing apparatus according to claim 1, wherein the controller controls an amplitude of the output vibrations of each of the vibration generators such that a relationship between a first composite vibration position determined on the basis of a vibration level of each vibration generator in a first vibration generator group comprising a plurality of vibration generators and a second composite vibration position determined on the basis of a vibration level of each vibration generator in a second vibration generator group comprising a plurality of vibration generators matches the identified directional component, and controls a delay of the output vibrations of each of the vibration generators according to the directional component. [9] Information processing system, comprising an information processing device configured to play an XR content; a display device configured to display a video in response to a video signal output from the information processing device; an audio output device configured to generate sounds in response to an audio signal output from the information processing device; and a vibration device having a plurality of vibration generators and configured to apply vibrations to a user in response to a vibration signal output from the information processing device, wherein the information processing device comprises a controller configured to detects a scene in which a sound is generated by a target from the XR content, extracts a vibration parameter corresponding to the scene, the vibration parameter controlling the vibration device, a signal obtained by processing the signal of the sound generated by the target and amplified using the vibration parameter, and outputting a signal for controlling an amplitude and a delay of each of the vibration generators based on an identified directional component of the target with respect to the vibration device in the XR content as a vibration signal to the vibration device, extracts an audio parameter related to audio processing corresponding to the scene, and outputs the audio signal that has undergone an amplification process using the audio parameter to the audio output device. [10] Information processing method for controlling a vibration device having a plurality of vibration generators and configured to apply vibrations corresponding to a content to a user, whereby the information processing procedure identifying a directional component of an object representing a vibration source in an input content, and outputting a signal for controlling an amplitude and a delay of output vibrations of each of the vibration generators based on the directional component to the vibration device includes.

Images