Content reproduction device, vibration control signal generation device, server device, vibration control signal generation method, content reproduction system, and design support device

Abstract

A content reproduction device (10) is provided with a vibration output mechanism (P3) for generating vibrations and a controller (13), the vibration output mechanism (P3) giving vibrations to a user in accordance with content to be reproduced. The controller (13) detects the vibration output device (30) of the vibration output mechanism (P3), and controls the vibration generated by the vibration output mechanism (P3) on the basis of the detected vibration output device (30).

Description

Technical Field

[0001] The present invention relates to a content reproduction device, a vibration control signal generating device, a server device, a vibration control signal generating method, a content reproduction system and a design assisting device. Background Art

[0002] In the past, a technique has been proposed to improve the sense of presence of content by giving a user vibrations corresponding to the content the user is viewing or listening to. For example, a technique is known to improve the sense of presence by vibrating the air around the user so that the user feels the air vibrations throughout the body (see, for example, Patent Document 1).

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 11-46391 Summary of the invention

[0006] -Problems to be solved by the invention-

[0007] In a content reproduction device that imparts vibration to a user in order to enhance the sense of presence of the content, the structure that imparts (outputs) vibration to the user (hereinafter referred to as a vibration output mechanism) is not limited to the same structure. For example, in chair-type vibration output mechanisms of different models, the types of vibration output devices used, the material and shape of the chair on which the user sits, the assembly position of the vibration output device, etc. are different. Therefore, depending on the type of vibration output mechanism, the optimal vibration generation is different, and the vibration output value for vibration control used for vibration signal generation is different. Therefore, it is necessary to design and adjust a dedicated content reproduction device according to the vibration output mechanism used. Alternatively, it is necessary to design and adjust a dedicated vibration output mechanism corresponding to a vibration control signal generating device that generates a vibration control signal. In addition, in the past, since these responses were performed manually, there were problems in the operation that required a huge amount of man-hours.

[0008] In view of the above problems, an object of the present invention is to provide a technology capable of efficiently designing and adjusting a vibration output mechanism.

[0009] -Methods for solving problems-

[0010] The exemplary present invention is a content reproduction device that imparts vibrations corresponding to the content to be reproduced to a user, and the content reproduction device comprises: a vibration output mechanism that generates vibrations; and a controller. The controller detects a vibration output device of the vibration output mechanism and controls the vibrations generated by the vibration output mechanism according to the detected vibration output device.

[0011] -Effects of the Invention-

[0012] According to the present invention, it is possible to efficiently select a vibration output device and adjust a vibration control parameter value in order to improve the sense of presence of content. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is an explanatory diagram showing an example of a content reproduction system according to an embodiment.

[0014] Figure 2 Yes means Figure 1 An illustrative diagram of an overview of a vibration control signal generation process performed by a content reproduction device.

[0015] Figure 3 Yes means Figure 1 A structural diagram of an example of a content reproduction device.

[0016] Figure 4 This is a diagram showing an example of a scene information DB.

[0017] Figure 5 This is a diagram showing an example of a parameter information DB.

[0018] Figure 6 This is an explanatory diagram showing an example of arrangement of vibration output devices in a vibration seat of a content reproduction system.

[0019] Figure 7 It means in Figure 3 An explanatory diagram of operations such as setting vibration control parameter values ​​performed in a content reproduction device and an overview of the operation of the content reproduction device.

[0020] Figure 8 It means by Figure 3 A flowchart of scene recognition processing for content being reproduced, which is executed by a controller of a content reproduction device.

[0021] Fig. 9 This is a diagram showing an example of a scene recognition DB.

[0022] Fig. 10A Yes means Figure 6 An illustrative diagram of a first example of the responsibilities (structural vibration, air vibration) of each vibration output device in a content reproduction device.

[0023] Fig. 10B Yes means Figure 6 An illustration of a second example of the responsibilities (structural vibration, air vibration) of each vibration output device in a content reproduction device.

[0024] Fig. 10C Yes means Figure 6An illustration of a third example of the responsibilities (structural vibration, air vibration) of each vibration output device in a content reproduction device.

[0025] Fig.11 It is an explanatory diagram showing an example of a content reproduction system according to a modified example.

[0026] Fig.12 Yes means Fig.11 A structural diagram of an example of a server device.

[0027] Fig.13 Yes means Fig.12 A flowchart of a vibration seat design process performed by a controller of a server device. DETAILED DESCRIPTION

[0028] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In addition, the present invention is not limited to the contents of the embodiments described below.

[0029] <1. Overview of vibration control signal generation method>

[0030] Figure 1 1 is an explanatory diagram showing an example of a content reproduction system PS according to an embodiment. Figure 1 As shown, the content reproduction system PS includes a content reproduction device 10 , a display device (video display) P1 , a speaker (sound output device) P2 , and a vibration seat (vibration output mechanism) P3 .

[0031] The content reproduction device 10 is a device that generates control signals for each actuator for image reproduction, sound reproduction, and vibration reproduction according to the content to be reproduced. The display device P1 is a device that provides the user U1 with images corresponding to the content to be reproduced (images based on the image (control) signal of the content reproduction device 10). The speaker P2 is a device that provides the user U1 with sounds corresponding to the content to be reproduced (sounds based on the sound (control) signal of the content reproduction device 10).

[0032] The display device P1 is, for example, a head-mounted display. The display device P1 outputs images corresponding to the reproduced content, allowing the user U1 to enjoy, for example, an XR (Cross Reality) experience. The display device P1 includes a device that detects changes in the internal and external conditions of the user U1 through a sensor unit, such as a camera, a microphone, a motion sensor, etc.

[0033] In addition, the content provided to the user U1 is not limited to XR content, but may also be content displayed on a normal display, such as movies, concert images, games, etc. In this case, the display device P1 may be, for example, a display such as a television set set on the floor, on a table, or hung on a wall.

[0034] The speaker P2 is, for example, an earphone type and is mounted on the ear of the user U1. The speaker P2 outputs sound corresponding to the content to be reproduced, thereby providing the sound of the content to the user U1. The speaker P2 is not limited to an earphone type, and may be, for example, a box-type speaker installed on the floor, on a table, or hung on a wall, a so-called box speaker.

[0035] The vibration chair P3 is, for example, a chair-type vibration output mechanism, which includes a seat (seat (chair) body) 20 on which the user U1 sits and a plurality of vibration output devices 30. The vibration output device 30 is provided inside or outside the seat 20. The vibration output device 30 is composed of, for example, an electric vibration converter including an electromagnetic circuit, a piezoelectric element, and an electric cylinder. The vibration output device 30 generates vibration (vibration based on a vibration (control) signal of the content reproduction device 10) according to the content to be reproduced, and imparts (outputs) the vibration to the user U1.

[0036] Figure 2 Yes means Figure 1 FIG. 1 is an explanatory diagram of an overview of a vibration control signal generation process performed by the content reproduction device 10 . First, the content reproduction device 10 detects a scene satisfying a predetermined condition from video data and audio data related to content (step S1 ).

[0037] The given conditions of a scene refer to the conditions for determining that a scene should generate vibration (should be given vibration to the user). Specifically, the given conditions of a scene refer to conditions related to the characteristics of an object in the content (weight, moving speed, etc.), the distance between an avatar that matches the user and the object in the content space (hereinafter referred to as the object distance), etc. For example, when the conditions of "object weight (kg) / object distance (m) squared is greater than 10" and "the object is moving" are satisfied in the content space, the content reproduction device 10 determines that the scene should generate vibration.

[0038] Next, the content reproduction device 10 sets the priority of the event that is the object of vibration for the scene detected by the scene detection (step S2). In other words, in the case of a scene determined to be one in which vibration should be generated, sometimes multiple events satisfy the vibration generation condition. For example, in the case of an elephant approaching while driving on a bad road in the content space, the vibration for driving on the bad road and the vibration for the elephant approaching become the vibrations that should be generated. However, in the case of vibrations corresponding to both parties, the cause of the vibration (which object is the vibration for) cannot be felt based on the human perception characteristics of vibration. Therefore, the content reproduction device 10 performs the following control: the event of the vibration that the user should feel more is set as an event with a high priority, and vibrations for this event are preferentially generated.

[0039] Next, the content reproduction device 10 extracts the vibration control parameter value corresponding to the scene with the highest priority (the object event for vibration generation) (step S3). The vibration control parameter is data used when generating vibration, and includes parameters such as low-pass filter characteristics (cut-off frequency, etc.), delay characteristics, and amplification characteristics. These parameter values ​​are stored in a data table or the like in association with the scene. In addition, in the case of generating vibrations corresponding to multiple scenes according to the upper priority, the same vibration generation process is performed on each of the multiple upper priority scenes.

[0040] Next, the content reproduction device 10 generates a vibration control signal for the vibration output device 30 of the vibration seat P3 based on the extracted vibration control parameter value (step S4), and outputs the vibration control signal to the vibration output device 30 of the vibration seat P3. Specifically, in the information contained in most of the content, the correlation between sound and vibration is relatively high. Therefore, the content reproduction device 10 processes the sound data contained in the content based on the content data (object distance, object weight, object category, etc.) and the vibration control parameter value corresponding to the scene, and generates a vibration control signal. Then, the generated vibration control signal is output to the vibration output device 30 of the vibration seat P3. In this way, the content reproduction device 10 can give vibration to the user according to the content to be reproduced.

[0041] <2. Content playback device>

[0042] Figure 3 Yes means Figure 1 A structural diagram of an example of a content reproduction device 10. Figure 3 In the figure, the structural elements necessary for explaining the features of the present embodiment are shown, and description of general structural elements is omitted.

[0043] like Figure 3 As shown, the content reproduction device 10 includes a storage unit 12 and a controller 13. In addition, in this description, the content input into the content reproduction device 10 is described as "XR content". That is, the "user" refers to the operator himself in the XR space (the virtual character, avatar for the content viewing user). The content viewing user (the operator himself) hears the sound heard by the virtual character (avatar) (surrounding sounds (sounds of other characters, etc.), the sound of the virtual character). Therefore, the user in the XR content has a function (microphone function) to convert the user's own voice and the surrounding sounds into sound (electrical) signals.

[0044] The storage unit 12 is configured to include a volatile memory and a nonvolatile memory. The volatile memory is, for example, a RAM (Random Access Memory). The nonvolatile memory is, for example, a ROM (Read Only Memory), a flash memory, and a hard disk drive. The nonvolatile memory stores programs and data that can be read by the controller 13. At least a portion of the programs and data stored in the nonvolatile memory can be obtained from another computer device (server device) or a portable recording medium connected by wire or wireless.

[0045] A plurality of databases (hereinafter sometimes referred to as “DB (Database)”) for various processes are provided in the storage unit 12 . As the databases, a content DB 121 , a scene information DB 122 , a priority information DB 123 , a parameter information DB 124 , and a scene recognition DB 125 are provided.

[0046] The content DB 121 is a database storing data of a content group reproduced by the content reproduction device 10. Based on the data of the content group, the image, sound, vibration, etc. of each content are reproduced. In addition, the data of these contents can also be obtained from an external server (the external server is treated as the content DB 121), and the content DB 121 and the external server can also be used in combination.

[0047] The scene information DB 122 is a database that stores various information related to scenes that are targets for vibration generation. Figure 4 This is a diagram showing an example of the scene information DB 122 .

[0048] like Figure 4 As shown, the scene information DB 122 includes data on items such as "detection scene", "condition type", "object", "condition parameter", "threshold value" and "conditional expression", and each information is stored in correspondence with the information of the "detection scene".

[0049] The item "detection scene" of the scene information DB122 is the name of the scene constituting the identification information of the scene. "Detection scene" also becomes identification information for identifying data records in the scene information DB122. In other words, a data record of the scene information DB122 is generated for each "detection scene" data, and data of the items of "condition type", "object", "condition parameter", "threshold value" and "conditional expression" corresponding to the data record are stored. "Detection scene" usually uses a scene identification code such as a numerical value, but in this embodiment, an identifiable name is used for easy understanding of the description.

[0050] The item "condition type" of the scene information DB 122 indicates the type of scene detection information, that is, what kind of information is used as a basis for detecting the scene. Figure 4In the example shown, data of “condition type” is roughly classified into such types as the positional relationship between the user and the object in the XR space, the user’s action, spatial information of the user’s presence, time information of the user’s presence, or sound generated by the object.

[0051] The item "object" of the scene information DB 122 indicates the type of the object used for scene detection. Figure 4 In the example shown, "object" corresponds to information such as object 1, object 2, user, space 1, space 1 + object 3, content 1, object 4, object 5, object 6, etc. Here, object 1, object 2, object 3, object 4, object 5, object 6 respectively represent different targets in the XR space. In addition, space 1 represents, for example, a space in the XR space where the user exists, and content 1 represents the content itself.

[0052] The item "condition parameter" of the scene information DB 122 indicates a condition related to a parameter, such as which parameter is used for an object (data of "object") when performing scene detection. Figure 4 In the example shown, the “condition parameter” corresponds to parameter type information such as distance, angle, speed, acceleration, rotation speed, in space, existence of an object, number, start time to end time, and sound pattern.

[0053] The item "threshold" of the scene information DB 122 indicates a threshold corresponding to a condition parameter for determining a detection scene. The item "conditional formula" of the scene information DB 122 indicates a conditional formula for detecting a detection scene, and for example, the relationship between the condition parameter and the threshold is defined as a conditional formula and stored.

[0054] In addition, for the sake of convenience, Figure 4 In the example of "scene W", "object 4", and "mode w", symbols such as "W", "4", and "w" are used to mark each item value, but in fact, each item value stores data in a manner that allows the specific meaning to be understood.

[0055] Specifically, for example, the detection scenes "scene W", "scene X", "scene Y", and "scene Z" actually become data such as "elephant walking scene", "horse walking scene", "car driving scene", and "car sharp turn scene". In this case, "object 4", "object 5", and "object 6" as the target objects actually become data such as "elephant", "horse", and "car". Furthermore, the conditional expressions "pattern w", "pattern x", "pattern y", and "pattern z" actually become data such as "horse walking sound pattern", "elephant walking sound pattern", "car driving sound pattern", and "tires skiing sound pattern".

[0056] The sound pattern is represented by, for example, a feature vector whose elements are the feature quantities of the sound. Then, when the similarity (e.g., cosine similarity, Euclidean distance) between the feature vectors corresponding to the two sound patterns is greater than a threshold value, it can be determined that the two sound patterns are similar. For example, the conditional expression "the sound pattern is similar to pattern w" means that the similarity between the feature vector calculated from the sound generated in the scene and the feature vector of the sound corresponding to pattern w is greater than a threshold value.

[0057] In addition, the content reproduction device 10 can be combined Figure 4 For example, the condition type may be the positional relationship between the user and the object, and the condition parameters may be the position and angle of the scene α, that is, the scene α may satisfy the conditions of scene A and scene B.

[0058] The priority information DB 123 is a database that stores various information related to the priority of vibration generation. The content reproduction device 10 sets the priority of vibration generation for each scene in which vibration should be generated based on a predetermined given rule. The rules related to the priority of vibration generation are stored in the priority information DB 123. Although detailed description is omitted here, as priority rules, for example, "priority for scenes detected first (or later)", "priority for scenes with short duration", "priority for scenes with large amplitude in the low frequency band", "priority for scenes that end first", etc. are set.

[0059] The parameter information DB 124 is a database that stores information on vibration control parameters for each scene. Figure 5 1 is a diagram showing an example of parameter information DB 124. Figure 5 As shown, the parameter information DB 124 includes items of “scene type” and “vibration control parameter”, for example, and stores each information in association with the information of “scene type”.

[0060] The item “Scene Type” of the parameter information DB 124 indicates the type of the scene. Figure 4 The data of the "detection scene" shown is associated with the data of the "scene category" by a predetermined method (for example, a data table indicating a correspondence relationship). In other words, the data of the item "scene category" of the parameter information DB 124 and the data of the item "detection scene" of the scene information DB 122 are associated by a predetermined method, and as a result, the data records of the scene information DB 122 and the parameter information DB 124 are associated (linked).

[0061] The item "vibration control parameter" of the parameter information DB 124 indicates the vibration control parameter set in the corresponding scene, and stores the data (value) of each parameter separately for each vibration output device 30 of the vibration seat P3. As "vibration control parameter", for example, data of items such as "LPF (Low Pass Filter, low frequency characteristics)", "delay (delay characteristics)" and "amplification (amplification rate)" are stored separately. Figure 5 The “vibration control parameters” for two types of vibration outputters are shown in FIG. 1 , but the “vibration control parameters” are stored for the vibration outputters that are controlled individually.

[0062] Figure 5 The data examples shown are parameter values ​​for vibration generation processing based on content sound. "LPF" indicates the cutoff frequency of the low-pass filter that extracts low-frequency components from the sound. "Delay" indicates the time to delay the vibration relative to the sound. "Amplification" indicates the amplification rate such as the degree to which the original vibration generated from the sound is amplified or attenuated to perform vibration control.

[0063] return Figure 3 The description continues. The controller 13 implements various functions of the content reproduction device 10, including a processor that performs calculation processing, etc. The processor is configured to include a CPU (Central Processing Unit), for example. The controller 13 can be composed of one processor or multiple processors. In the case of being composed of multiple processors, these processors are connected to each other so as to be communicable and perform processing in a collaborative manner.

[0064] The controller 13 includes a scene detection unit 131, a priority setting unit 132, a parameter extraction unit 133, and an output unit 134. In the present embodiment, the functions of the controller 13 are realized by the processor executing a calculation process according to a program stored in the storage unit 12.

[0065] The scene detection unit 131 includes a scene determination unit 131 a for determining whether a scene of the content being reproduced is a scene for which vibration control (generation) should be performed, and a parameter setting unit 131 b for setting parameter values ​​used in a process for vibration generation processing.

[0066] The scene determination unit 131a determines whether the scene satisfies a predetermined given condition based on the content being reproduced. The scene determination unit 131a determines whether it is a scene where vibration control should be performed (vibration should be generated) (detecting a scene where vibration control should be performed), for example, using image data and sound data related to the content and conditional expressions stored in the scene information DB 122. Specifically, the scene determination unit 131a determines whether it is a scene where vibration control should be performed, for example, based on the coordinate information of the target (vibration generating object) in the XR space and information related to the target category, using the conditional expressions of the scene information DB 122. In addition, when the scene of the content being reproduced is a scene where vibration control should be performed, the scene determination unit 131a determines which of the "detected scenes" stored in the scene information DB 122 the scene is based on this information.

[0067] For example, the scene determination unit 131a detects a scene in which a sound is generated from an object in the content being reproduced. Figure 4 The scene W, scene X, scene Y, and scene Z whose condition type is "sound generated from an object" are candidate scenes. The scene determination unit 131a calculates the similarity between the feature vector obtained from the sound signal of the content and the feature vector of the predetermined sound in the candidate scene (the sound pattern of the condition parameter), and determines whether the candidate scene satisfies the sound pattern condition based on the determination result of whether the similarity is above a predetermined similarity threshold. Furthermore, the scene determination unit 131a determines whether the candidate scene satisfies the object distance condition based on the determination result of whether the object distance in the content being reproduced is below a predetermined threshold. Then, the candidate scene that satisfies these sound pattern conditions and the object distance conditions (satisfies the conditional formula) is determined as a detection scene for vibration generation.

[0068] In addition, when the scene determination unit 131 a determines that the scene does not correspond to any detected scene, it is regarded that there is no corresponding detected scene and vibration is not generated (the vibration control parameter is set to a value of no vibration).

[0069] The parameter setting unit 131b sets (initializes, changes) the value of the "vibration control parameter" of the parameter information DB 124. The main methods for setting the parameter value include a setting method based on input information from the developer of the XR content or the user, and an automatic setting method based on the content category or the like.

[0070] Specifically, in the parameter value setting method based on the user's input information, the user selects the scene for setting (adjusting) the parameter value and the parameter category to be set (adjusted), and sets the parameter of the setting object in the scene by operating the up and down operation buttons, etc. When setting, it is preferred to display a test image of the scene of the parameter setting object, and generate vibration based on the parameter being set, so that the setting is performed while actually feeling the vibration.

[0071] In addition, in the automatic setting method of parameter values ​​based on the scene category of the content, first, the scene category of the reproduced content is detected. The detection of the content category is determined by the scene category information assigned to the content information, or estimated by analyzing a part of the content image or sound. Then, the automatic setting method sets each parameter value according to the detected content category.

[0072] The parameter value can be obtained from a server (which collects parameter information from each device, performs statistical processing, etc., and stores appropriate parameter values ​​corresponding to the scene category of the content) based on the scene category information of the content (sending a parameter value request signal including the scene category information). This allows the setting of the parameter information DB 124 to be more appropriately structured.

[0073] In addition, the parameter setting unit 131b sets the vibration control parameter value as a scene for generating vibration in a scene in which the amplitude of the sound in the low-frequency region generated by the object in the content scene exceeds a predetermined threshold value. The correlation between an object that generates large vibrations and an object that generates large sounds in the low-frequency region is large, and the size of the vibration is also correlated with the size of the low-frequency region of the sound. Therefore, in a scene in which the amplitude of the sound in the low-frequency region exceeds a predetermined threshold value, it is estimated that the amplitude of the vibration that should be generated to improve the sense of presence is also large, and it is efficient to set the vibration control parameter value as a scene for generating vibration.

[0074] In addition, regarding such a scenario, a method may be considered in which the scenario is set by a user or content developer, or obtained from a server (which collects scene information and parameter information of various contents from each device, performs statistical processing, etc., and stores appropriate scene information and parameter information).

[0075] In addition, the above-mentioned sound amplitude threshold value can also be determined according to the content category (content). Specifically, a data table of content categories (content) and intensity threshold values ​​is prepared in advance, and when selecting a scene for setting conditions, the intensity threshold value corresponding to the target content is retrieved from the data table, and the scene for setting conditions is selected using the retrieved intensity threshold value.

[0076] For example, the categories of content include music videos that allow users to mainly listen to music, animal documentaries that explain the biological bodies of animals, etc. In the case of a walking scene of an elephant in a music video, it is often better not to generate excessive vibrations so as not to interfere with the music. On the other hand, in the walking scene of an elephant in an animal documentary, it is often better to generate vibrations in order to create a sense of presence.

[0077] Therefore, the parameter setting unit 131b sets the threshold value in the music video to be lower than the threshold value in the animal documentary. As a result, the elephant's walking scene in the music video is less likely to be set as a scene for generating vibrations than the elephant's walking scene in the animal documentary, and the generation of unnecessary vibrations is suppressed in the elephant's walking scene in the music video. Thus, vibrations suitable for the content can be generated.

[0078] In addition, the parameter values ​​of the scene information DB 122 and the parameter information DB 124 can be updated by calculating (correcting) new parameter values ​​(such as the adjustment value itself or the value obtained by adding an offset, etc.) based on the various adjustments (vibration level adjustment, blur adjustment, etc.) actually made by the user while viewing the content.

[0079] return Figure 3 The description continues. The priority setting unit 132 sets priorities for the scenes detected by the scene detection unit 131. The priority setting unit 132, for example, refers to the priority information DB 123 and selects which scene to prioritize when a plurality of scenes are detected simultaneously by the scene detection unit 131. In addition, when only one scene is detected by the scene detection unit 131, the scene becomes the highest priority.

[0080] The parameter extraction unit 133 extracts the vibration control parameter value for the scene whose priority is set by the priority setting unit 132. In detail, the parameter extraction unit 133 refers to the parameter information DB 124 and extracts the vibration control parameter value corresponding to the "detection scene" set as the highest priority by the priority setting unit 132 from the parameter information DB 124. At this time, the parameter extraction unit 133 extracts the vibration control parameter value corresponding to each of the plurality of vibration output devices 30, so that each vibration output device 30 can be controlled with a dedicated vibration control parameter value. As a result, compared with the case where each vibration output device 30 is controlled with the same vibration control parameter value, a further improvement in the sense of presence can be achieved.

[0081] The content reproduction device 10 can infer, from among the candidate objects that are candidates for vibration-generating objects, a candidate object whose vibration generated by the candidate object has a large impact on the user, based on the priority set by the priority setting unit 132, and select it as the vibration-generating object. In this case, the threshold for selecting the object that generates vibration is preferably changed based on the content category. In other words, this is because, depending on the content of the content, it is sometimes preferable to suppress or emphasize the reproduction of the vibration caused by the object appearing in the content, and it is preferable to adjust the content (judgment level) of the determination of the object that generates vibration.

[0082] In other words, the principle of generating vibration is as follows. The object that generates vibration in the content (each scene) is determined based on the content of the content. Then, based on the sound signal corresponding to the determined object, a vibration signal (vibration data) is generated. At this time, the sound signal corresponding to the object is the sound data of the object contained in the content, or the sound data of the object generated based on the sound data in the scene (for example, the low-frequency area is filtered and extracted). In addition, the vibration signal (vibration data) is generated by extracting the low-frequency component of the sound signal of the object and appropriately amplifying it.

[0083] In addition, as a method for determining an object for generating vibration, the object is determined by estimating the low-frequency characteristics (e.g., volume level) of the sound emitted by the sound generating object of the content. In this case, the low-frequency characteristics of the sound emitted by the sound generating object are estimated, for example, based on a reference vibration intensity based on the category of the object (target) in the virtual space, and the distance between the reference position (user position in the virtual space, etc.) and the object (target). In determining the object, the one with a larger low-frequency volume level of the sound emitted by the sound generating object is determined as the object for generating vibration.

[0084] At this time, parameter extraction unit 133 extracts a dedicated vibration control parameter value for each vibration output device 30. This can further improve the sense of presence compared to a case where each vibration output device 30 is controlled with the same vibration control parameter value.

[0085] Furthermore, the parameter extraction unit 133 includes a learning unit 133 a . The learning unit 133 a learns the relationship between the scene stored in the parameter information DB 124 and the vibration control parameter value.

[0086] The learning unit 133a, for example, uses the scenes stored in the parameter information DB 124, the corresponding vibration control parameter values, the reactions of the users (content viewers who are assigned vibrations) to the vibration control of the vibration output device 30 based on the parameter values, etc. as learning data to perform machine learning, thereby updating the vibration control parameter values ​​stored in the parameter information DB 124.

[0087] At this time, the learning unit 133 a may use, for example, user evaluations (user's vibration adjustment operations after vibration control, user survey results, etc.) on vibration control parameter values ​​(vibrations given to users) as learning data.

[0088] In this way, the learning unit 133 a learns (updates) the vibration control parameter value according to the scene from the viewpoint of what kind of vibration control parameter value should be set for each scene in order to obtain a high user evaluation, that is, a high sense of presence.

[0089] Furthermore, when a new scene is reproduced, the learning unit 133a determines what vibration control parameter value should be set based on the learning result. Specifically, for example, when the viewing user performs an adjustment operation for vibration during the reproduction of a fireworks scene that is not registered as a vibration generating scene, the vibration control parameter value is calculated using the new scene and the adjustment content as learning data, and data such as the vibration control parameter value based on the learning result is stored in the parameter information DB 124, etc. In addition, the vibration control parameter values ​​of similar scenes can also be used to learn (generate) the vibration control parameter values ​​of the new scene. For example, when a new fireworks scene is reproduced, the vibration control parameter values ​​of similar conditions such as explosion scenes can be used to perform vibration control, and the vibration control parameter values ​​of the fireworks scene can be learned (generated) using the learning result, that is, using the user's response.

[0090] The output unit 134 generates a vibration control signal for each vibration output device 30 using the vibration control parameter value extracted by the parameter extraction unit 133, and outputs the signal to each vibration output device 30. Specifically, the output unit 134 performs a frequency band limiting process suitable for the vibration of the LPF on the sound data in the vibration generation scene of the content being reproduced, and converts the sound data into the original vibration data. Furthermore, the output unit 134 performs a vibration adjustment process based on the vibration control parameter value extracted by the parameter extraction unit 133 on the original vibration data, and generates a vibration control signal.

[0091] Specifically, the output unit 134 performs, for example, frequency characteristic additional processing such as low-frequency emphasis, delay, and amplification on the original vibration data according to the vibration control parameter value. In this way, the output unit 134 outputs the vibration control signal (vibration control data) adjusted according to the vibration control parameter value, which is a signal suitable for vibration obtained by processing the signal of the sound generated by the object in the XR space of the content, to the plurality of vibration output devices 30. At this time, the output unit 134 outputs the vibration control data that has been individually adjusted for each vibration output device 30 to the corresponding vibration output device 30.

[0092] In addition, if Figure 5As shown in the figure, the vibration control parameter values ​​are set for each scene, but it is also effective to further correct them according to the detailed conditions in the scene (or detailed scene categories). For example, in a scene where there is a vibrating object (such as an elephant) in the XR space, the vibration characteristics can be adjusted by increasing or decreasing the values ​​of the vibration control parameters "LPF", "delay" and "amplification" according to the distance between the user and the object (detailed scenes divided by distance).

[0093] Thus, in a scene where vibration should be generated when reproducing content, the content reproduction device 10 generates an original vibration signal based on the sound data in the scene, and then processes the original vibration signal according to the scene category to generate a vibration control signal. As a result, even general content without dedicated data for vibration control in the content data can generate vibrations that are finely adapted to each scene in the content.

[0094] <3. Setting of vibration control parameters>

[0095] Next, setting of vibration control parameter values ​​for the plurality of vibration output devices 30 constituting the vibration output mechanism (vibration seat P3 ) will be described.

[0096] In the content reproduction device 10, in order to improve the sense of presence of the content, the vibration given to the user (person) is decomposed into air vibration and structural vibration. Air vibration refers to vibration transmitted from the vibration source to the body through the air. Structural vibration refers to vibration transmitted to the body through direct contact with the vibration source or contact via structural components or the ground.

[0097] In the reproduction of content such as XR content, in the case of a scene with strong vibrations accompanied by the user's body movement, the vibration given to the user is suitable for structural vibration. In the case of a scene in which a vibration other than structural vibration is given to the user, the vibration is suitable for air vibration. Scenes with strong vibrations accompanied by the user's body movement refer to scenes in which, for example, a heavy object in the content shakes, the entire image (camera) shakes, the user's movement in the XR content exceeds a given threshold, an object containing the user in the content, the camera contacts other objects, etc. In addition, for the user in the XR content, the user's grounded state (standing on the ground, etc., flying in the air) affects the strength of the structural vibration and the air vibration.

[0098] In the content reproduction device 10, the vibration given to the user is decomposed into air vibration and structural vibration for each scene of the content, so that the multiple vibration outputters 30 are respectively responsible for the air vibration and the structural vibration. That is, the content reproduction device 10 generates a vibration control signal for driving the vibration outputter 30 using the vibration control parameter value corresponding to the air vibration, or generates a vibration control signal for driving the vibration outputter 30 using the vibration control parameter value corresponding to the structural vibration.

[0099] Thus, the air vibration transmitted to the body through the air and the structural vibration transmitted directly to the body through contact with a vibrating object can be appropriately separated and given to the content viewing user. Furthermore, by appropriately setting the vibration mode such as the generation (imparting) position, vibration frequency, and vibration waveform of the air vibration and structural vibration, the user can be given vibration in various ways. Therefore, the sense of presence of the content can be improved.

[0100] Figure 6 1 is an explanatory diagram showing an example of the arrangement of the vibration output device 30 in the vibration chair P3 of the content reproduction system PS. In the content reproduction system PS, the vibration output devices 31 and 32, for example, composed of a vibrator (vibration plate), are provided inside the seat surface 21 and the back surface 22 of the seat 20, respectively. In addition, the vibration output device 33, for example, composed of a six-axis electric cylinder, is provided at the lower part 23 of the seat 20.

[0101] The vibration output devices 31 of the seat surface 21 of the seat 20 are arranged at five locations, namely, the four corners (vibration output devices 31s) and the center (vibration output device 31c) of the seat surface 21. Moreover, the vibration output device 31c in the center of the seat surface 21 has a larger vibration plate than the other four surrounding vibration output devices 31s, and is easy to generate large-amplitude vibrations at low frequencies. The vibration output devices 32 of the back surface 22 of the seat 20 are arranged at four locations, namely, the four corners of the back surface 22. In addition, a vibration output device 33 that generates vibrations (low frequency, large amplitude) that shake the seat 20 as a whole is arranged at the lower portion 23 of the seat 20. In addition, if a structure is adopted in which the seat 20 as a whole is mounted on the vibration output device 33, the seat 20 as a whole can be shaken by the vibration generated by the vibration output device 33.

[0102] Regarding structural vibration, the vibration output device that is conducive to the reproduction of low frequencies is responsible, and in this case, the vibration output device 31c and the vibration output device 33 are responsible. In particular, the strong structural vibration accompanying the user's body movement is responsible for the vibration output device 33, which gives the user a vibration that shakes the user's body as a whole. Regarding air vibration, the vibration output device that is not strongly pressed against the user, that is, has low contact with the user, is responsible, and in this case, the vibration output device 32 that is difficult to apply pressure caused by the user's weight is responsible. The vibration output devices 31s at the four corners of the seat surface 21 are responsible for either structural vibration or air vibration according to the content being reproduced and its scene, taking into account the aspect of not giving vibration that shakes the user's body as a whole and the aspect of relatively high contact with the user.

[0103] in addition, Figure 6 The configuration and function (structural vibration and air vibration) of the vibration output device 30 involved are examples. According to the type, installation location, number, etc. of the vibration output device 30, the contact state between the vibration output device 30 and the viewing user, and the reproduced content (especially when only specific content is reproduced like a dedicated game console), the structural vibration or air vibration is appropriately determined, and the vibration seat P3 is designed and assembled. In addition, the contact state between the vibration output device 30 and the viewing user can be detected based on the user's operation input and the camera's photographic image of the user's viewing state. When the contact state is in close contact, the structural vibration is used, and when the contact state is not in close contact, the air vibration is used.

[0104] Figure 7 It means in Figure 3 FIG. 1 is an explanatory diagram of operations such as setting of vibration control parameter values ​​performed in the content reproduction device 10 and an overview of the operation of the content reproduction device 10. These operations / operations are roughly divided into a preparation phase before content reproduction (step S11) and during content reproduction (step S12).

[0105] In the preparation stage before the content reproduction of step S11, the content viewing and listening user and the like set the hardware conditions (the type, setting position, number, etc. of the vibration output device 30, and the contact state of the vibration output device 30 and the viewing and listening user). The input of the hardware conditions (setting conditions of the vibration output device 30, etc.) is input by the content viewing and listening user and the like operating an input device such as a keyboard, or by searching a database based on the model of the vibration seat P3. In addition, it can also be automatically input by inputting information provided from the connected vibration seat P3 (the structural information of the vibration seat P3 is pre-stored in a storage device, and read out and provided when the content reproduction device 10 is connected). In addition, the detection of the contact state of the vibration output device 30 and the viewing and listening user is performed based on the operation result of the content viewing and listening user on the input device such as a keyboard, or based on the analysis result of the camera image obtained by photographing the state of the content viewing and listening user.

[0106] Moreover, the controller 13 determines the responsibility of each vibration output device 30, that is, the structural vibration generation, the air vibration generation, or the combination of both vibrations, based on the information of the setting conditions and the contact state, and stores it in the storage unit 12. In addition, as the responsibility information of each vibration output device 30, information such as a correction coefficient when each vibration is generated (for example, a coefficient of 1 when the structural vibration is generated, and a coefficient of 0.5 when the air vibration is generated) is added, and this information is used when the vibration signal is generated (for example, the basic structural vibration signal is multiplied by the coefficient to obtain a vibration signal of the vibration output device 30 that meets the requirements, etc.). In addition, these decisions are made based on, for example, a data table (generated by the designer and developer of the content reproduction device 10, etc. through experiments, etc.) that stores data indicating the relationship between the setting conditions and the contact state and the responsibility of each vibration output device 30 that is set.

[0107] Next, the controller 13 analyzes the content of the content, extracts each scene in which vibration should be generated in the content, and detects the conditions related to the vibration in each scene, such as the grounding state of the user in the scene. In addition, the controller 13 determines the vibration control parameter value of each vibration output device 30 based on the conditions related to the vibration in the scene and the responsible information of each vibration output device 30 determined above for each scene in which vibration should be generated. Furthermore, the controller 13 stores the determined parameter information of each scene or each vibration output device 30 in the parameter information DB 124.

[0108] In addition, if Fig. 9 As shown, the scene recognition DB 125 of the content reproduction device 10 stores various acoustic features of the scene, vibration classification, grounding state of the user in the content, and vibration control parameter values ​​in association with the scene category. In addition, the data of the scene recognition DB 125 is generated by, for example, a designer or developer of the content reproduction device 10 through experiments, etc., and is stored in the scene recognition DB 125.

[0109] Furthermore, the content viewing user manually adjusts the position of each vibration output device 30 and the vibration control parameter value as needed (for example, according to the content viewing user's physique and the grounding environment of the content playback device 10 ) (step S13 ).

[0110] When the content is reproduced in step S12, the controller 13 (scene recognition unit 135) of the content reproduction device 10 determines which scene of the content being reproduced is stored in the scene recognition DB 125 based on the characteristics of the sound and the characteristics of the image in the scene of the content being reproduced. In other words, the controller 13 (scene recognition unit 135) compares the characteristic data of the scene recognition DB 125 with the characteristics of the sound and the characteristics of the image in the scene of the content being reproduced, and detects the scene recorded in the data of the scene recognition DB 125 whose characteristics match.

[0111] Then, the controller 13 (output unit 134) reads out the vibration control parameter value corresponding to the determined (detected) scene (the vibration control parameter value of the same data record in the scene recognition DB 125). In addition, when there are multiple scenes in the content being reproduced that should generate vibration, the controller 13 (priority setting unit 132) selects the scene with the highest priority as the target scene for generating vibration. Then, the controller 13 (output unit 134) generates a vibration control signal for each vibration output device 30 based on the read vibration control parameter value and the sound information of the content being reproduced, and outputs it to each vibration output device 30.

[0112] That is, the controller 13 generates a vibration control signal for each vibration output device 30 based on the vibration control parameter value for each vibration output device 30 for each scene recognized from the content, and controls the vibration generated by each vibration output device 30. Moreover, the vibration control parameter value is set based on conditions such as the installation state of each vibration output device 30 in the vibration seat P3 (vibration output mechanism). Therefore, the controller 13 generates a vibration control signal for each vibration output device 30 based on the scene of the content being reproduced and the type, installation position, and number of the vibration output devices 30, and controls the vibration generated by the vibration output devices 30.

[0113] Thus, each vibration output device 30 can be vibrated appropriately with respect to various vibration seats P3 (various vibration output mechanisms) of different types, installation locations, and numbers of the vibration output devices 30. In addition, each vibration output device 30 can be vibrated appropriately with respect to a variety of contents and scenes of different types and contents. In addition, by automating the setting of the vibration control parameter value for each vibration output device 30 using the controller 13, the setting / adjustment of the vibration control parameter value accompanying the change or variation of the vibration seat P3 (various vibration output mechanisms) can be efficiently performed.

[0114] In addition, the controller 13 sets a vibration control parameter value for each scene of the content being reproduced, and controls the vibration generated by the vibration output device 30 based on the vibration control parameter value. For example, in the case where the content is a music video, it is possible to give the user vibrations that are respectively suitable for soothing music such as folk songs and other fast-paced music. In addition, for example, in the case where the content is an animal documentary, in the walking scene of an elephant and the running scene of a horse, vibrations suitable for each scene can be provided to the user. That is, by providing the user with vibrations suitable for the scene of the content being reproduced, the sense of presence of the content can be improved.

[0115] In addition, the controller 13 may set a vibration control parameter value for each content according to the content category (the overall content of the content) instead of each scene of the content being reproduced, and vibrate each vibration output device 30 based on the vibration control parameter value. In this case, for example, when the content is a music video, when the user listens to the music, appropriate vibration can be given to the user. In addition, for example, when the content is an animal documentary, vibration suitable for allowing the user to observe the biological state of the animal and explain the biological state of the animal can be given to the user. That is, by giving the user a vibration suitable for the content of the content, the sense of presence of the content can be improved.

[0116] <4. Scene recognition processing when reproducing content>

[0117] Figure 8 It means by Figure 3 Flowchart of scene recognition processing for content being reproduced executed by the controller 13 of the content reproduction device 10. Specifically, Figure 8 The flowchart shows a scene recognition process for determining which vibration classification is more suitable for the vibration generated for each scene of the content being reproduced by the controller 13 of the content reproduction device 10. The flowchart shows the technical content of a computer program that enables a computer device to implement the scene recognition process. In addition, the computer program is provided (sold, circulated, etc.) in the form of various readable non-volatile recording media storing the computer program or downloaded from a server storing the computer program via a communication line. The computer program may consist of only one program or a plurality of programs that cooperate with each other.

[0118] This scene recognition is started in the content playback preparation stage S11, for example, by the user's content playback preparation start operation.

[0119] Fig. 9 1 is a diagram showing an example of the scene recognition DB 125. In detail, Fig. 9 is used based on Figure 8The database contains information on the values ​​of each item in the scene recognition processing results, and determines the vibration classification (air vibration, structural vibration) and vibration control parameter values ​​in each scene. Fig. 9 The vibration classification data of the data record whose values ​​of each item are consistent with the scene recognition processing result becomes the vibration category of the vibration generated in the scene, and the vibration control parameter value becomes the vibration control parameter value of the vibration generated in the scene. In addition, it is also possible to set two vibration classifications (air vibration and structural vibration) in the same scene. In this case, each vibration output device 30 responsible for air vibration and structural vibration generates vibration according to the corresponding vibration control parameter value.

[0120] In addition, the scene recognition DB 125 in this example is a database (data table) of recognition scenes corresponding to the sitting / standing conditions of the content-viewing user and the voice and virtual character in the content. In addition, in order to easily understand the relationship between the value of each item and the vibration classification, for data records with the same data of each item, the boxes of each item are appropriately merged and displayed. In addition, since the data of the standing condition of the virtual character is recorded in the same data recording method as the sitting condition, the full display is stopped and detailed description is omitted.

[0121] like Fig. 9 As shown, as data items in the scene recognition DB 125, there are "frequency", "amplitude", "stability of sound source", "stability of audible sound", "pitch", "number of simultaneous directions", "size of low frequency (audible sound)", which are characteristics of sound (sound data) in the scene of the content, an item of "grounding" indicating the sitting / standing state of the virtual character relative to the viewing user that can be judged based on the image data in the content, and vibration classification that generates vibration appropriately under the conditions of the content shown in other items, and correspondence is established with the recognition scene (stored in the data record generated for each recognition scene). In addition, as a data item in the scene recognition DB 125, there is also a "vibration control parameter", which stores the vibration control parameter value suitable for the scene that matches the data record. The designer of the content reproduction device 10 and the like screens the appropriate scene based on experiments, etc., generates characteristic data of the sound corresponding to the scene, vibration control parameter values ​​matching the scene, etc., and stores the various data of these scene recognition DBs 125.

[0122] In addition, regarding the "vibration control parameters", it can also be used as Figure 5 The parameter information DB 124 shown in the figure is set as another database, and the data records of the two databases are associated with each other by the scene type (the data of the items "recognition scene" and "scene type").

[0123] In addition, the scene recognition DB 125 also has an item “recognition scene” indicating the scene category, and the scene category recognition data (scene name, etc.) that is consistent with the data of each item related to the characteristics of the sound (sound data) in the scene of the above content is stored as the “recognition scene” data.

[0124] Figure 8 The process shown is a flowchart showing the vibration control process during content reproduction, and is repeatedly executed by the controller 13 during content reproduction (with a period such that the delay caused by the change of the vibration control parameter value accompanying the scene change does not cause discomfort to the content viewing user).

[0125] In step S101, the controller 13 (scene recognition unit 135) inputs and analyzes the sound signal in the scene of the content being reproduced, extracts data related to each item of the feature in the sound, and transfers to step S102. In addition, the sound signal analysis can be realized by digitizing the sound signal and performing various processing such as frequency decomposition through calculation processing. In addition, the image signal in the scene of the content being reproduced can also be input and analyzed for the determination of the feature in the sound described below.

[0126] In step S102, the controller 13 (scene recognition unit 135) determines whether the frequency band (main frequency band) of the sound in the scene is high or low (determined by the upper and lower relationship with respect to the threshold), and transfers to step S103. In other words, for example, the threshold is determined to be a frequency of 20 Hz, and the scene recognition unit 135 determines that the frequency is low when the sound intensity distribution of the sound below 20 Hz is high, and determines that the frequency is high when the sound intensity distribution of more than 20 Hz is low.

[0127] In step S103 , the controller 13 (scene recognition unit 135 ) determines whether the amplitude (average value, maximum value) of the sound is large or small (determined by the upper and lower relationship with respect to the threshold value), and the process proceeds to step S104 .

[0128] In step S104, the controller 13 (scene recognition unit 135) determines whether the sound source is stable or unstable (whether it emits a continuous sound or a sudden sound), and transfers to step S105. The sound source refers to an object that generates sound within the content. A sound source that generates the same sound continuously, such as the sound of a car running or the sound of animal footsteps, is stable, and a sound source that generates abruptly, such as a car horn or an animal's cry, is unstable. As an example of this determination method, the scene recognition unit 135 determines that the sound source is stable when the dynamic range of the sound is lower than a predetermined given threshold, and determines that it is unstable when it exceeds the threshold.

[0129] In step S105, the controller 13 (scene recognition unit 135) determines whether the audible sound is stable or unstable, and proceeds to step S106. The audible sound is the sound heard by the user (the avatar corresponding to the user) in the content (for example, the XR content space), and the sound recorded by the microphone equipped by the user in the content. For example, the scene recognition unit 135 determines that the audible sound is stable when the dynamic range of the audible sound is lower than a predetermined given threshold, and determines that the audible sound is unstable when the dynamic range exceeds the threshold.

[0130] In step S106, the controller 13 (scene recognition unit 135) determines whether the pitch of the sound is strong or weak, and then moves to step S107. For example, the scene recognition unit 135 determines that the pitch is strong when the pitch variation (frequency variation) of the sound is large (such as the cry of an animal with a wide frequency range and large pitch variation), and determines that the pitch is low when the pitch variation of the sound is small (such as the steady sound of a machine with a narrow frequency range).

[0131] In step S107, the controller 13 (scene recognition unit 135) determines whether the sound is generated from a single direction or from multiple directions simultaneously, and the process proceeds to step S108. For example, when there are multiple sound sources at a distance closer than a predetermined threshold to the user, the scene recognition unit 135 determines that the sound is generated from multiple directions simultaneously, and in other cases, determines that the sound is generated from a single direction.

[0132] In step S108, the controller 13 (scene recognition unit 135) determines whether the signal level of the low-frequency component of the sound is large or small, and transfers to step S109. For example, the scene recognition unit 135 determines that the low-frequency component of the sound is large when the sound pressure of the sound component below a predetermined given frequency exceeds a predetermined given threshold, and determines that the low-frequency component of the sound is small when the sound pressure is below the threshold.

[0133] In step S109 , the controller 13 (scene recognition unit 135 ) performs data comparison processing of the scene recognition DB 125 using the determination results from steps S101 to S108 , determines a matching scene, and then moves to step S110 .

[0134] In step S110, the controller 13 (output unit 134) extracts the vibration control parameter value corresponding to each vibration output device 30 corresponding to the determined scene from the scene recognition DB 125, and uses the vibration control parameter value to process the sound signal of the content to generate a vibration signal. Then, the generated vibration signal is output to each corresponding vibration output device 30 to generate the desired vibration, and the processing ends. In addition, the controller 13 (output unit 134) uses the vibration control parameter value before the update to continuously generate the vibration corresponding to the content (sound) of the content being reproduced until the vibration control parameter value is updated.

[0135] according to Figure 8 According to the hardware structure of the content reproduction system PS and the characteristics of the sound in the content reproduction, an appropriate vibration control parameter value is set for each vibration output device 30 in the content reproduction system PS, and each vibration output device 30 vibrates with the vibration generated based on the set vibration control parameter value. Therefore, it is possible to give the content viewing user an appropriate vibration corresponding to the hardware structure of the content reproduction system PS and the content of the content, and the content viewing user can enjoy the content reproduction with a sense of presence.

[0136] In addition, in this processing example, the vibration control parameter value is determined (calculated) when the content is reproduced, but the vibration control parameter value of the content to be reproduced is calculated in advance by the same method, and is associated with the reproduction scene (reproduction time, scene, etc.) of the content and stored in advance. Then, when the content is reproduced, the vibration control parameter value associated with the corresponding scene can be extracted to perform vibration control. In this case, for example, the vibration control parameter value can be recorded as one of the content information together with the content main body information (image / sound information) on a content recording medium such as a content optical disk.

[0137] Fig. 10A , Fig. 10B as well as Fig. 10C Yes means Figure 6 1 and 2 are diagrams for explaining the first, second, and third examples of the functions (structural vibration, air vibration) of each vibration output device 30 in the content reproduction device 10. The functions of each vibration output device 30 differ depending on the hardware structure of the content reproduction system PS and the content (scene) content.

[0138] In this embodiment, Fig. 10A In the embodiment, the vibration output devices 31s at the four corners of the seat surface 21 and the four vibration output devices 32 at the back surface 22 are responsible for air vibration and structural vibration respectively. Fig. 10B In the embodiment, the vibration output devices 31s at the four corners of the seat surface 21 and the four vibration output devices 32 at the back surface 22 are responsible for air vibration, and the vibration output device 31c at the center of the seat surface 21 is responsible for structural vibration. Fig. 10C In the embodiment, the vibration output devices 31s at the four corners of the seat surface 21 and the four vibration output devices 32 at the back surface 22 are responsible for air vibration, the vibration output device 31c at the center of the seat surface 21 is responsible for structural vibration (frequency 20 Hz to 40 Hz), and the vibration output device 33 at the lower part 23 of the seat 20 is responsible for structural vibration (frequency less than 20 Hz). In other words, the content reproduction device 10 generates and outputs vibration signals of each vibration output device 30, 31, 32 according to the vibration control parameter value (set in the scene recognition DB 125) corresponding to the function of each vibration output device 30, 31, 32, thereby generating structural vibration or air vibration.

[0139] <5. Modifications>

[0140] Fig.11 2 is an explanatory diagram showing an example of a content playback system PS according to a modification. In addition, the same reference numerals or the same names are given to the common components in the modification as those in the embodiment described above, and the description thereof may be omitted.

[0141] like Fig.11 As shown, the content reproduction system PS of the modified example is connected to the server device 40 using a communication line.

[0142] The content reproduction device 10 has the same components as those of the above-described embodiment. The content reproduction device 10 outputs a vibration signal corresponding to the content to be reproduced to the vibration output device 30 , and applies vibration to the user U1 via the vibration output device 30 .

[0143] The server device 40 is connected to the content playback device 10 via a network N so as to be able to communicate bidirectionally. The server device 40 may be a physical server or a virtual server. The network N is, for example, a local area network or the Internet.

[0144] Fig.12 Yes means Fig.11 A structural diagram of an example of a server device 40. Fig.12 In the figure, the structural elements necessary for explaining the features of the present embodiment are shown, and description of general structural elements is omitted.

[0145] like Fig.12 As shown, the server device 40 includes a communication unit 41 , a storage unit 42 , and a controller 43 . The communication unit 41 is an interface for performing data communication with other devices via a network N. The communication unit 41 is, for example, a NIC (Network Interface Card).

[0146] In addition, the server device 40 has the same structural elements as the content reproduction device 10 of the embodiment described above. In addition, for the same structural elements (same structure, operation, etc.), the same Figure 3 The same names as the names of the structural elements in are expressed, and for symbols, the symbol of SV is added in front of the symbol and its description is omitted.

[0147] In the case of the content reproduction system PS of this modified example, the hardware configuration information of the content reproduction device 10, the image information indicating the sitting state of the vibration chair P3 of the content viewing user U1, and the information of the reproduction target content are sent from the content reproduction device 10 to the server device 40. In addition, the vibration control signal for each vibration output device 30 of the vibration chair P3 is sent from the server device 40 to the content reproduction device 10. Then, the content reproduction device 10 outputs the image signal and the sound signal of the content and the vibration control signal from the server device 40 to the display device P1, the speaker P2, and each vibration output device 30 synchronously.

[0148] In addition, the role sharing between the content reproduction device 10 and the server device 40 is not limited to this modification, and can be appropriately set. For example, the server can also have a content reproduction function, and the image signal and sound signal of the content and the vibration control signal to each vibration output device 30 are sent from the server device 40 to the content reproduction device 10.

[0149] By using the server device 40, the server device 40 has information and programs that can correspond to various types of content reproduction systems PS with different structures, and can perform processing corresponding to the hardware structure of the content reproduction system PS and the content to be reproduced according to a request from the content reproduction system PS. Therefore, according to this modification, the content reproduction system PS does not need to have a dedicated structure, and the update of various information and programs can also be managed on the server device 40 side.

[0150] <6. Design of the vibration seat (vibration output mechanism) of the content reproduction system>

[0151] Next, the design process of the vibration output mechanism will be described by taking the design process of the vibration chair P3 of the content reproduction system PS as an example.

[0152] In this design processing example, an example using the server device 40 as a design assisting device for the vibration output mechanism is described, but it is also possible to use the content reproduction device 10. In addition, in the following description, the design processing can also be implemented in a design system based on a computer system having the same structural elements as those of the server device 40 used in the processing.

[0153] Fig.13 Yes means Fig.12The flowchart of the design process of the vibration seat P3 executed by the controller 43 of the server device 40 is shown. The flowchart shows the technical content of the computer program that enables the server device 40 to implement the design process of the content reproduction device 10. In addition, the computer program is provided (sold, circulated, etc.) in the form of various readable non-volatile recording media storing the computer program, or downloaded from the server storing the computer program via a communication line. The computer program may be composed of only one program or a plurality of programs that cooperate with each other.

[0154] Fig.13 The processing shown is executed in the server device 40 when the designer of the content playback device 10 performs design processing, for example, when a processing start operation is performed through an operation unit such as a keyboard.

[0155] In step S201, the controller 43 inputs the audio data of the content and transfers to step S202. At this time, the video data of the content may also be input and used for subsequent judgment processing. In addition, if the content used is a content with a high frequency of use in the content reproduction system PS or a content of a similar category, it is possible to design suitable for the content with a high frequency of use. For example, when designing a dedicated content reproduction system PS for a certain game, the game content is used.

[0156] In step S202, the controller 43 (scene detection unit SV131) determines whether the main component of the generated vibration is air vibration or structural vibration based on the sound data of the content and referring to the image data as needed, stores the result in the storage unit 12, and moves to step S203.

[0157] In step S203, the controller 43 (scene detection unit SV131) determines whether the reproduction of the content is completed (the amount required by the preset design), and returns to step S202 if it is not completed, and transfers to step S204 if it is completed. In other words, through the processing of steps S202 and S203, it is possible to grasp the number of conditions in which the main component of the vibration to be generated in the target content is air vibration and the number of conditions in which it is structural vibration.

[0158] In step S204 , the controller 43 (scene detection unit SV131 ) calculates the ratio of the situation where the air vibration is the main component of the vibration in the target content to the situation where the structural vibration is the main component of the vibration, and then moves to step S205 .

[0159] In step S205, the controller 43 inputs data on the state of the seat after all the vibration output devices 30 that can be installed in the vibration seat P3 are installed (the position of each vibration output device 30 and the vibration effect level), and also inputs data such as the component price and installation cost of each vibration output device 30, and the target price of the designed vibration seat P3, and then transfers to step S206. In addition, the information of the vibration seat P3 is input, for example, by the designer of the vibration seat P3, etc., operating a keyboard, etc.

[0160] In step S206, the controller 43 determines the order of reduction of each vibration output device 30 based on the ratio of air vibration to structural vibration in the target content calculated in step S204 and the vibration effect level (contribution to the improvement of telepresence) of each vibration output device 30 input in step S205, and transfers to step S207. In other words, the lower the vibration generation ratio of the responsible vibration category (air vibration, structural vibration) in the target content and the lower the vibration effect on the content viewing user, the earlier the order of deletion (the higher the deletion priority).

[0161] For example, the ratio of each main component of air vibration and structural vibration in the object content, that is, the vibration component ratio, is 8:3, and the vibration output devices responsible for air vibration are set to A1, A2, and A3 in order of vibration effect level from high to low, and the vibration output devices responsible for structural vibration are set to B1, B2, and B3 in order of vibration effect level from high to low. The proportion of air vibration in the object content is relatively large, so the first priority of deletion is the vibration output device B3 with the lowest vibration effect level among the vibration output devices responsible for structural vibration. Moreover, since the deletion priority of a vibration output device responsible for structural vibration is determined, the proportion value of air vibration is reduced (for example, halved) in order to reduce the dominance of air vibration (the vibration component ratio becomes 4:3). If the same process is continued, the proportion of air vibration is large next, so the second priority of deletion is the vibration output device B2 with the lowest vibration effect level among the vibration output devices responsible for structural vibration. Moreover, the new vibration component ratio becomes 2:3. Secondly, since the proportion of structural vibration is large, the third priority of deletion is the vibration output device A3 with the lowest vibration effect level among the vibration output devices responsible for air vibration. Then, this process is continued until all the deletion priorities are determined. In this case, the deletion priorities are, from the highest to the lowest, the vibration output devices B3, B2, A3, B1, A2, and A1.

[0162] The vibration effect level may be obtained, for example, through sensory evaluation by design and development team members, or by statistical processing of the results of the vibration effect level evaluated by each subject (e.g., questionnaire results of each subject) when reproducing a certain content by sequentially setting a plurality of vibration output devices 30 to a vibration generating state.

[0163] In step S207, the controller 43 reduces the vibration output device 30 that is the highest in the reduction order determined in step S206 among the vibration output devices 30 mounted on the vibration seat P3, calculates the manufacturing price of the vibration seat P3 in this case based on the input prices of the vibration output devices 30 of the vibration seat P3, and transfers to step S208. In addition, in this example, the vibration output devices 30 are reduced simply according to the reduction priority of the vibration output devices 30, but the vibration output devices 30 may also be reduced in consideration of factors such as the cost of the vibration output device 30 and the remaining price until the target described later is achieved.

[0164] The reduction of components of the vibration seat P3 is a virtual process for calculating the manufacturing price, and the reduction of components of the vibration seat P3 (physical) is not actually performed. In fact, for example, the designer etc. decides the final specification / design based on the result of the above-mentioned evaluation process.

[0165] In step S208, the controller 43 determines whether the predetermined target cost for the vibration seat P3 and the predetermined target number of vibration output devices 30 (whether it is below the target) are reached. If reached, the controller 43 notifies (displays) the result (the structure of the vibration seat P3 after the vibration output devices 30 are appropriately reduced) and ends the processing. If not reached, the controller returns to step S207 and continues the reduction processing and its evaluation processing.

[0166] Thus, while performing a simulation of reducing the vibration output devices 30 mounted on the vibration seat P3 in an appropriate order, the structure of the vibration seat P3 that meets the target can be confirmed, and the design of the content reproduction system PS (vibration seat P3) can be made more efficient.

[0167] <7. Notes, etc.>

[0168] Various technical features disclosed as embodiments in this specification can be subjected to various changes within the scope of the main purpose of the technical creation. That is, the above-mentioned embodiments are illustrative in all aspects and are not restrictive. The technical scope of the present invention is not represented by the description of the above-mentioned embodiments, but by the claims, including the meaning equivalent to the claims and all changes within the scope. In addition, the multiple embodiments shown in this specification can be implemented in appropriate combinations within the possible scope.

[0169] In addition, in the above-mentioned embodiment, it is described that various functions are realized by software through the operation processing of the CPU according to the program, but at least a part of these functions can also be realized by electrical hardware resources. As hardware resources, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), etc. can be used. In addition, conversely, at least a part of the functions realized by hardware resources can also be realized by software.

[0170] In addition, the scope of this embodiment may include a computer program that causes a processor (computer) to implement at least a part of the functions of the content reproduction device 10. In addition, the scope of this embodiment may include a computer-readable non-volatile recording medium that records such a computer program. The non-volatile recording medium may be, for example, an optical recording medium (e.g., an optical disk), an optical-magnetic recording medium (e.g., an optical-magnetic disk), a USB memory, or an SD card, in addition to the above-mentioned non-volatile memory.

[0171] -Explanation of symbols-

[0172] 10Content reproduction device

[0173] 12 Storage

[0174] 13 Controller

[0175] 20 seats

[0176] 21 seats

[0177] 22 Back

[0178] 23 lower part

[0179] 30, 31, 31c, 31s, 32, 33 vibration output device

[0180] 40 Server device (design support device)

[0181] P1 Display Device

[0182] P2 Speaker

[0183] P3 vibration seat (vibration output mechanism)

[0184] PS content reproduction system

[0185] U1 user.

Claims

1. A content reproduction device that imparts vibration corresponding to the content to be reproduced to a user, The content reproduction device comprises: a vibration output mechanism that generates vibration; and Controller, The controller performs the following processing: detecting a vibration output device of the vibration output mechanism, Based on the detected vibration output device, the vibration generated by the vibration output mechanism is controlled.

2. The content reproduction device according to claim 1, wherein: The controller controls the vibration generated by the vibration output mechanism based on the sound of the content being reproduced and a parameter value corresponding to the vibration output device.

3. The content reproduction device according to claim 2, wherein: The controller controls the vibration generated by the vibration output mechanism based on a parameter value corresponding to a scene of the content being reproduced.

4. The content reproduction device according to claim 2, wherein: The controller performs the following processing: The vibration output device is divided into two types: one for generating air vibration and one for generating structural vibration. The vibration output device is controlled to generate vibrations corresponding to those for generating air vibrations and structural vibrations.

5. A vibration control signal generating device, which outputs a vibration control signal corresponding to a reproduction signal of content to be reproduced to a vibration output mechanism that generates vibration, The vibration control signal generating device comprises a controller, The controller performs the following processing: detecting a vibration output device of the vibration output mechanism, The vibration control signal is generated according to the detected vibration output device.

6. A server device, which provides vibration generation information for generating a vibration control signal corresponding to a reproduction signal of content to a vibration control signal generating device via a communication line, wherein the vibration control signal generating device outputs the vibration control signal to a vibration output mechanism that generates vibrations, The server device includes a controller. The controller performs the following processing: receiving information of the vibration output device of the vibration output mechanism from the vibration control signal generating device, The vibration generation information is generated according to the received vibration output device.

7. A method for generating a vibration control signal, wherein the vibration output mechanism generates vibrations corresponding to the content through the vibration control signal, The vibration output device of the vibration output mechanism is detected, and the vibration control signal is generated according to the detected vibration output device.

8. A content reproduction system, comprising: An image display, which displays an image corresponding to the image signal; A sound output device, which outputs a sound corresponding to the sound signal; a vibration output mechanism that outputs vibration corresponding to the vibration signal; and a content reproducing device that generates the image signal, the sound signal, and the vibration signal corresponding to the content data, and outputs the signals to the image display, the sound output device, and the vibration output mechanism; The content reproduction device performs the following processing: detecting a vibration output device of the vibration output mechanism, The vibration signal is generated according to the detected vibration output device.

9. The content reproduction system according to claim 8, wherein: The content reproducing device generates the vibration signal based on the sound signal and the vibration outputter.

10. A design aid, A design assisting device for a vibration output mechanism having a plurality of vibration output devices and providing a user with vibrations corresponding to the content to be reproduced, The design support device comprises a controller, The controller performs the following processing: detecting the arrangement states of the plurality of vibration output devices relative to the vibration output mechanism, determining a reduction priority of each of the plurality of vibration outputters based on the vibration effect levels of the plurality of vibration outputters, The vibration output device used is set according to the reduction priority, Outputting vibration control signals to the plurality of vibration output devices respectively, wherein the vibration control signals are generated according to matching the vibration action of the vibration output mechanism with the matching object content of each of the plurality of vibration output devices, The cost information when the vibration output mechanism is constituted by the set plurality of vibration output devices is calculated.

11. The design assistance device according to claim 10, wherein: The controller performs the following processing: The vibration output device is divided into one for generating air vibration and one for generating structural vibration, and the reduction priority is determined. The vibration output device to be used is set according to the ratio of scenes suitable for generating air vibrations to scenes suitable for generating structural vibrations in the matching object content suitable for generating air vibrations and structural vibrations.

Images