Realistic sensation improvement device, realistic sensation improvement method, realistic sensation improvement program, and realistic sensation improvement system

Abstract

To flexibly execute update of a model that is used for realistic sensation improvement processing.SOLUTION: A realistic sensation improvement device is configured to perform, on content, realistic sensation improvement processing corresponding to first scene information included in a scene group comprising a plurality of pieces of scene information set in advance for identifying scenes in the content, and the realistic sensation improvement device acquires second scene information corresponding to a scene in the content, selects the first scene information from the scene group according to the similarity to the second scene information, and performs the realistic sensation improvement processing corresponding to the selected first scene information on the content.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a device for enhancing a sense of presence, a method for enhancing a sense of presence, a program for enhancing a sense of presence, and a system for enhancing a sense of presence. [Background technology]

[0002] A control device that produces content such as movie viewing in a vehicle has been proposed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-188390 Summary of the Invention [Problem to be solved by the invention]

[0004] Content includes multiple scenes each including a series of images, sounds, etc. In rendering the content, for example, scene information indicating the type of scene included in the content is detected using a scene detection model, realism enhancement processing corresponding to the detected scene information is acquired from the realism enhancement model, and the acquired realism enhancement processing is executed on the scene. Here, it is conceivable that the scene detection model may be updated to add a scene to be detected. In such a case, there is a risk that the realism enhancement processing for the added scene will not be performed until the realism enhancement model is updated to correspond to the added scene. A similar problem may occur when the realism enhancement model is updated but the scene detection model is not updated. In other words, there is a constraint that both the scene detection model and the realism enhancement model must be updated simultaneously.

[0005] One aspect of the disclosed technology aims to provide a sense of realism enhancement device, a sense of realism enhancement method, a sense of realism enhancement program, and a sense of realism enhancement system that can flexibly update a model used in the sense of realism enhancement process. [Means for solving the problem]

[0006] One aspect of the disclosed technology is exemplified by the following realism enhancement device: The realism enhancement device performs realism enhancement processing on the content corresponding to first scene information, which is scene information identifying a scene in the content and is included in a scene group consisting of a plurality of preset scene information, acquires second scene information corresponding to a scene in the content, selects the first scene information from the scene group according to the similarity with the second scene information, and performs realism enhancement processing on the content corresponding to the selected first scene information. [Effects of the Invention]

[0007] According to the disclosed technology, it is possible to flexibly update the model used in the processing for enhancing realism. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration and operation of a content playback system (entertainment system) according to a basic embodiment. [Figure 2] FIG. 2 is a diagram illustrating a schematic configuration of a content playback device according to the basic embodiment. [Figure 3] FIG. 3 is a diagram illustrating a schematic configuration of a realistic sensation enhancing device according to a basic embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of an entertainment system according to the embodiment. [Figure 5] FIG. 5 is a diagram showing an example of content. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of a control device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a hardware configuration of the VR chair according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a processing block of the control device according to the embodiment. [Figure 9] FIG. 9 illustrates an example of a management table stored in the auxiliary storage unit. [Figure 10] FIG. 10 is a diagram illustrating an example of a processing flow of the control device according to the embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the management table. DETAILED DESCRIPTION OF THE INVENTION

[0009] <Basic embodiment> First, in order to clarify the characteristic configuration and operation, a content playback system that is a basic embodiment in a representative system configuration example will be described. Fig. 1 is a block diagram showing the configuration and operation of a content playback system B (entertainment system) according to the basic embodiment. Fig. 1 is a diagram illustrating an example of an overall schematic diagram of the content playback system according to the basic embodiment.

[0010] The content playback system B includes a content playback device B1, a sense of realism enhancement device B2, a display B3, a speaker B4, and an exciter B5. The content playback device B1 reads content information from a content medium CB1 such as a DVD, and performs decoding processing and the like to play back the content (outputting video and audio signals). The content playback device B1 then outputs the video and audio signals of the content to the sense of realism enhancement device B2.

[0011] FIG. 2 is a diagram illustrating a schematic configuration of a content playback device B1 according to a basic embodiment. The content playback device B1 includes a second scene determination unit B102 that determines a scene type in the content being played back, such as a scene type "vehicle traveling." The content playback device B1 also includes a second scene database (referred to as "DB" in the figure) B103 that associates second scene information (information type) with scene determination information. The scene determination information is video and audio data for identifying a scene, such as data on image elements and audio elements contained in objects and backgrounds appearing in the scene, and identifies the scene by comparing the information with the video and audio being played back in the content. The second scene determination unit B102 determines the scene type of the content scene being played back based on the second scene database B103 through such scene determination processing, and outputs the scene type (second scene information) stored in the second scene database B103 corresponding to the determination result to the realism enhancement device B2. The second scene determination unit B102 may be realized by an AI (artificial intelligence) model trained with a training data set in which the video and audio signals of each scene in many pieces of content are used as input data and the corresponding scene types (scene type information is generated by a training data generator by watching the content, etc.) are used as correct answer data.

[0012] 3 is a diagram illustrating a schematic configuration of a sense of realism enhancing device B2 according to a basic embodiment. The sense of realism enhancing device B2 processes the video signal and audio signal from the content playback device B1 to generate a sense of realism-enhanced video signal, audio signal, and vibration signal, and outputs them to a display B3, a speaker B4, and an exciter B5, respectively. The display B3, the speaker B4, and the exciter B5 respectively receive the video signal, audio signal, and vibration signal processed by the sense of realism enhancing device B2. The device outputs video, audio, and vibrations according to the vibration signal, allowing the user to view the content with a rich sense of realism.

[0013] The sense of realism enhancing device B2 includes a first scene determination unit B201, a first scene database (referred to as "DB" in the drawing) B202, a parameter selection unit B203, a video processing unit B204, an audio processing unit B205, a vibration processing unit B206, and a parameter database (referred to as "DB" in the drawing) B207. The first scene determination unit B201 converts the scene type from the content playback device B1 (the scene type in the second scene group made up of each piece of second scene information registered in the second scene database B103) into a scene type that can be handled by the sense of realism enhancing device B2 (the scene type in the first scene group made up of each piece of first scene information registered in the first scene database B202 (and the parameter database B207)). Specifically, for example, a linguistic analysis (semantic similarity search) is performed on the second scene type data input from the content playback device B1 and each piece of data of the first scene type in the first scene database B202, and the first scene type that is most similar in meaning to the second scene type data is determined as the conversion result for the second scene type data.

[0014] The parameter selection unit B203 selects parameter values ​​to be used for each signal processing from parameter values ​​in a parameter database B207 based on the first scene type determined (converted) by the first scene determination unit B201. The parameter database B207 stores video parameters for video signal processing, audio parameters for audio signal processing, and vibration parameters for vibration signal processing, associated with each first scene type data. Specifically, a data record is formed using each first scene type data as a primary key, and values ​​of the video parameters, audio parameters, and vibration parameters corresponding to the first scene type data are stored in the corresponding data record. The parameter selection unit B203 then searches the parameter database B207 for first scene information using the first scene type determined (converted) by the first scene determination unit B201, selects the video parameter values, audio parameter values, and vibration parameter values ​​stored in the data record of the corresponding first scene information data as parameter values ​​to be used for each processing, and outputs the selected parameter values ​​to the video processing unit B204, audio processing unit B205, and vibration processing unit B206.

[0015] The video processing unit B204 processes the video signal input from the content playback device B1 using the video parameter values ​​selected by the parameter selection unit B203 to generate a video signal with enhanced realism, which is output to the display B3. The video processing unit B204 may be configured with a digital signal processor or the like and may be implemented using various known video processing programs. The audio processing unit B205 processes the audio signal input from the content playback device B1 using the audio parameter values ​​selected by the parameter selection unit B203 to generate an audio signal with enhanced realism, which is output to the speaker B4. The audio processing unit B205 may be configured with a digital signal processor or the like and may be implemented using various known video processing programs. The vibration processing unit B206 processes the audio signal input from the content playback device B1 using the vibration parameter values ​​selected by the parameter selection unit B203 to generate a vibration signal for enhancing realism, which is output to the exciter B5.

[0016] The vibration processing unit B206 can be configured with a digital signal processor or the like, and can be realized using a known signal waveform processing program or the like. Specifically, in real situations, vibrations are correlated with acoustic signals. For example, the sound that is generated and transmitted to the user when a vehicle approaches is a sound that depends on the object characteristics of the vehicle itself, such as the mass and shape of the vehicle, the running state of the vehicle, the surrounding environment, and the positional relationship between the vehicle and the user. Furthermore, the vibration that is generated and transmitted to the user at that time is a vibration that depends on the object characteristics of the vehicle itself, such as the mass and shape of the vehicle, the running state of the vehicle, the surrounding environment, and the positional relationship between the vehicle and the user, just like the sound. For this reason, these sounds and vibrations are correlated. This correlation is used to generate a vibration signal from an acoustic signal. Specifically, the vibration parameters are An acoustic signal is input to a filter whose characteristics are determined according to the input data, and vibration parameters are also input as data for determining the characteristics of the filter.

[0017] Then, when the signals thus processed are input, the display B3 displays video with an enhanced sense of realism, the speaker B4 outputs audio with an enhanced sense of realism, and the exciter applies vibrations to the user to enhance the sense of realism. Furthermore, the first scene determination unit B201 converts second scene type data from the outside (content playback device B1 in this example) into any of the first scene type data (type data belonging to the first scene group) that can be processed using language analysis processing or the like. Therefore, whatever the second scene type data from the outside, it can be converted into appropriate first scene type information, and subsequent signal processing to enhance the sense of realism can be performed appropriately. Next, a more specific embodiment will be described.

[0018] <Embodiment> An embodiment in which the basic embodiment described above is applied to a chair and a display will be described below with reference to the drawings. FIG. 4 is a diagram showing an example of an entertainment system 1 according to the embodiment. The entertainment system 1 includes a control device 2, a display 3, a virtual reality (VR) chair 4, and a network N1. In FIG. 4, speakers 411L and 411R, a backrest vibration unit 412, a seat vibration unit 413, and an air blower 415 provided in the VR chair 4 are illustrated by dotted lines. FIG. 4 also illustrates a chair rocking unit 414 provided on the legs 43. The control device 2 is an example of a realism enhancing device B2 of the basic embodiment. The speakers 411L and 411R are an example of a speaker B4 of the basic embodiment. The display 3 is an example of a display B3 of the basic embodiment. The backrest vibration unit 412, the seat vibration unit 413, and the air blower 415 are an example of an exciter B5 of the basic embodiment.

[0019] The entertainment system 1 provides a user seated in a VR chair 4 with effects corresponding to scenes of content displayed on a display 3. A scene is, for example, a series of images, sounds, and other data included in the content. The content may include multiple scenes. The control device 2, the display 3, and the VR chair 4 are connected to each other via a network N1 so that they can communicate with each other. The entertainment system 1 is an example of a "sense of realism enhancement system."

[0020] The control device 2 is an information processing device. The control device 2 displays content stored in a storage unit on the display 3. The control device 2 also instructs the VR chair 4 to produce effects according to the scene of the content to be displayed on the display 3. The control device 2 is an example of a "sense of realism enhancement device."

[0021] The display 3 is a display device, and may be, for example, a liquid crystal display (LCD), a plasma display panel (PDP), an inorganic electroluminescence (EL) panel, or an organic EL panel.

[0022] The VR chair 4 is a chair in which a backrest vibration unit 412, a seat vibration unit 413, and a chair rocking unit 414 are provided on the seat 41, the backrest 42, and the legs 43, respectively. In addition, in the VR chair 4, for example, speakers 411L and 411R are provided on the backrest 42 at positions close to the user's ears. In the VR chair 4, the backrest vibration unit 412, the seat vibration unit 413, the chair rocking unit 414, and the speakers 411L and 411R are driven in response to instructions from the control device 2. The seat vibration unit 413 is driven to vibrate the seat 41. The backrest 42 is vibrated by driving the backrest vibration unit 412. In addition, the chair rocking unit 414 is driven to move the entire VR chair 4 so that it rocks. In this way, the VR chair 4 is equipped with speakers By driving the vibration units 411L and 411R, the backrest vibration unit 412, the seat vibration unit 413, and the chair rocking unit 414, the user seated in the VR chair 4 can enjoy content with an improved sense of realism. The display 3 and the VR chair 4 are examples of an "output device."

[0023] The network N1 is a computer network that interconnects information processing devices. The network N1 may be a wired network or a wireless network. Examples of wired networks include a local area network (LAN), an intranet, the Internet, and an in-vehicle network. Examples of wireless networks include a wireless LAN, a mobile communication network, and Bluetooth (registered trademark).

[0024] The content displayed on the display 3 in the entertainment system 1 includes multiple scenes. FIG. 5 is a diagram showing an example of content 900. The content 900 includes, for example, a video displayed by the control device 2 and sound output by the speakers 411L and 411R. Examples of the content 900 include movies, dramas, sports programs, concert programs, and talk shows. The content 900 includes multiple scenes, such as scene 1, scene 2, ..., and scene N. In the content 900, the scenes change over time in the order of scene 1, scene 2, ..., and scene N. Different scenes differ, for example, in screen brightness, background music (BGM), the tone of voice and speed of the performers, etc. Therefore, different realism enhancement processes are required to enhance the sense of realism for each scene. Therefore, the entertainment system 1 employs the following configuration to enable the execution of realism enhancement processes appropriate for each scene.

[0025] 6 is a diagram showing an example of the hardware configuration of the control device 2 according to the embodiment. The control device 2 includes a central processing unit (CPU) 201, a main memory unit 202, an auxiliary memory unit 203, a communication unit 204, and a connection bus C1. The CPU 201, the main memory unit 202, the auxiliary memory unit 203, and the communication unit 204 are interconnected by the connection bus C1.

[0026] The CPU 201 is also referred to as a microprocessor unit (MPU) or processor. The CPU 201 is not limited to a single processor and may have a multiprocessor configuration. Furthermore, a single CPU 201 connected via a single socket may have a multi-core configuration. At least a portion of the processing performed by the CPU 201 may be performed by a processor other than the CPU 201, such as a dedicated processor such as a digital signal processor (DSP), a graphics processing unit (GPU), a numerical calculation processor, a vector processor, or an image processing processor. Furthermore, at least a portion of the processing performed by the CPU 201 may be performed by an integrated circuit (IC) or other digital circuit. Furthermore, at least a portion of the CPU 201 may include an analog circuit. Integrated circuits include large-scale integrated circuits (LSIs), application-specific integrated circuits (ASICs), and programmable logic devices (PLDs). PLDs include, for example, field-programmable gate arrays (FPGAs). The CPU 201 may be a combination of a processor and an integrated circuit. The combination is called, for example, a microcontroller unit (MCU), a system-on-a-chip (SoC), a system LSI, a chipset, etc. In the control device 2, the CPU 201 loads the program stored in the auxiliary memory 203 into the work area of ​​the main memory 202. The CPU 201 then controls the peripheral devices by executing the program loaded into the work area of ​​the main memory 202. This allows the control device 2 to execute processing that meets a predetermined purpose. The main memory 202 and the auxiliary memory 203 is a recording medium that can be read by the CPU 201. The CPU 201 is an example of a "processor."

[0027] The main storage unit 202 is exemplified as a storage unit that is directly accessed by the CPU 201. The main storage unit 202 includes a random access memory (RAM) and a read only memory (ROM).

[0028] The auxiliary storage unit 203 stores various programs and various data on a readable and writable recording medium. The auxiliary storage unit 203 is also called an external storage device. The auxiliary storage unit 203 stores an operating system (OS), various programs, various tables, etc. The OS includes a communication interface program that exchanges data with external devices connected via the communication unit 204. The external devices include, for example, other information processing devices and external storage devices connected via a network N1, etc. The auxiliary storage unit 203 may be, for example, part of a cloud system, which is a group of computers on a network.

[0029] The auxiliary storage unit 203 is, for example, an erasable programmable ROM (EPROM), a solid state drive (SSD), a hard disk drive (HDD), etc. The auxiliary storage unit 203 is, for example, a compact disc (CD) drive device, a digital versatile disc (DVD) drive device, a Blu-ray (registered trademark) disc (BD) drive device, etc. The auxiliary storage unit 203 is an example of a "storage unit."

[0030] The communication unit 204 is, for example, an interface with the network N1, and communicates with external devices via the network N1.

[0031] 7 is a diagram showing an example of the hardware configuration of the VR chair 4 according to the embodiment. The VR chair 4 includes a CPU 401, a main memory unit 402, an auxiliary memory unit 403, a communication unit 404, speakers 411L and 411R, a backrest vibration unit 412, a seat vibration unit 413, a chair rocking unit 414, a blower unit 415, and a connection bus C2. In the VR chair 4, the CPU 401, the main memory unit 402, the auxiliary memory unit 403, the communication unit 404, speakers 411L and 411R, the backrest vibration unit 412, the seat vibration unit 413, the chair rocking unit 414, and the blower unit 415 are mutually connected by the connection bus C2. The CPU 401, main memory 402, auxiliary memory 403, communication unit 404 and connection bus C2 are similar to the CPU 201, main memory 202, auxiliary memory 203, communication unit 204 and connection bus C1 of the control device 2, so their description will be omitted.

[0032] The speakers 411L and 411R are devices that output sound. The speakers 411L and 411R are arranged, for example, on the backrest 42 of the VR chair 4 at positions close to the ears of a user seated in the VR chair 4. The speaker 411R is provided, for example, at a position corresponding to the right ear, and the speaker 411L is provided, for example, at a position corresponding to the left ear.

[0033] The backrest vibration unit 412 is provided on the backrest 42 and vibrates the backrest 42. The seat vibration unit 413 is provided on the seat 41 and vibrates the seat 41. The seat vibration unit 413 includes, for example, an actuator. The chair rocking unit 414 is provided between the seat 41 and the legs 43 and rocks the seat 41 back and forth and left and right. Each of the backrest vibration unit 412, the seat vibration unit 413 and the chair rocking unit 414 includes, for example, an actuator. The air blower 415 is a blower. The air blower 415 is placed, for example, on the backrest 42 of the VR chair 4 at a position close to the face of the user seated in the VR chair 4. The air blower 415 is placed with its air blowing direction facing the face of the user seated in the VR chair 4. The backrest vibration unit 412, the seat vibration unit 413 and the chair rocking unit 414 include, for example, an actuator. The unit 413, the chair rocking unit 414, and the air blowing unit 415 are driven in response to instructions from the CPU 401, for example.

[0034] <Processing block of control device 2> 8 is a diagram showing an example of processing blocks of the control device 2 according to the embodiment. The control device 2 includes a first acquisition unit 21, a determination unit 22, a second acquisition unit 23, an execution unit 24, a sense of realism enhancement model 25, and a detection model 26. The control device 2 executes the processes of each unit of the control device 2, such as the first acquisition unit 21, the determination unit 22, the second acquisition unit 23, the execution unit 24, the sense of realism enhancement model 25, and the detection model 26, by the CPU 201 executing a computer program deployed in an executable manner in the main storage unit 202.

[0035] The realism enhancement model 25 is a learning model that, when first scene information indicating the characteristics of a scene included in the content 900 is input, outputs a realism enhancement process that is applied to the scene to improve the realism of the scene. Examples of the first scene information include a name indicating the characteristics of the scene and information indicating frequency fluctuations of the video that constitutes the scene. The realism enhancement model 25 associates a realism enhancement process with each of a plurality of pre-set first scene information. The plurality of pre-set first scene information is an example of a "scene group." The realism enhancement process output by the realism enhancement model 25 includes, for example, specification of parameters such as the volume of the sound output from the speakers 411L and 411R, the amplitude and frequency of the vibration of the backrest vibration unit 412, the amplitude and frequency of the vibration of the seat vibration unit 413, the amplitude and frequency of the vibration of the chair rocking unit 414, the rotation speed of the fan of the air blower unit 415, and the brightness of the video displayed on the display 3. In other words, the sense of realism enhancement process can be regarded as processing performed on a scene to enhance the sense of realism of the scene. The sense of realism enhancement model 25 is constructed, for example, by machine learning using the first scene information as an explanatory variable and the sense of realism enhancement process as an objective variable.

[0036] The detection model 26 outputs second scene information based on, for example, the sound, image, brightness, etc. of the input scene. Examples of the second scene information include a name indicating the characteristics of the scene and information indicating frequency fluctuations of the video or sound constituting the scene. The detection model 26 is constructed, for example, by machine learning using the scene as an explanatory variable and the second scene information as a target variable. For example, an existing model may be used as the detection model 26. An example of an existing model that can be used as the detection model 26 is AudioSet (https: / / research.google.com / audioset / ontology / index.html) provided by Google. The detection model 26 is stored, for example, in the auxiliary storage unit 203. Note that the realism enhancement model 25 and the detection model 26 are constructed separately, and therefore compatibility between the first scene information and the second scene information is often not taken into consideration. In other words, the first scene information and the second scene information for the same scene may differ.

[0037] The first acquisition unit 21 acquires scenes to be displayed on the control device 2 from content 900 including multiple scenes. The first acquisition unit 21 acquires the scenes by, for example, performing image recognition, audio recognition, etc. based on the video and audio of the content 900 to determine scene divisions. The content 900 to be displayed on the control device 2 may be stored in, for example, the auxiliary storage unit 203 of the control device 2. The content 900 may also be acquired from another information processing device via the network N1. The first acquisition unit 21 inputs the acquired scene into the detection model 26 to acquire second scene information indicating the characteristics of the scene.

[0038] The determination unit 22 determines first scene information corresponding to the second scene information acquired by the first acquisition unit 21. For example, the determination unit 22 calculates the similarity between each piece of first scene information stored in the realism enhancement model 25 and the second scene information. 2, the first scene information with the highest calculated similarity may be determined as the first scene information corresponding to the second scene information.

[0039] When the first scene information and the second scene information are names indicating scene characteristics, for example, the determination unit 22 calculates the similarity between the meaning of the name of the first scene information and the meaning of the name of the second scene information. The similarity of meaning can be calculated, for example, by using a dataset that collects similarities between word pairs. As another example, the similarity of meaning can also be calculated using a learning model constructed by machine learning with word pairs as explanatory variables and similarity as a target variable. When the first scene information and the second scene information are information indicating frequency fluctuations, the similarity may be determined according to the difference in frequency fluctuations. The difference in frequency fluctuations may be, for example, the difference between the average value of frequency fluctuations in the first scene information and the average value of frequency fluctuations in the second scene information. In other words, when the first scene information and the second scene information are frequency fluctuations, the closer the frequency fluctuations are, the higher the calculated similarity.

[0040] Here, the determining unit 22 may determine that one piece of first scene information corresponds to one piece of second scene information. That is, the second scene information and the first scene information may be associated one-to-one. Alternatively, the determining unit 22 may determine that one piece of first scene information corresponds to a plurality of pieces of second scene information. That is, the second scene information and the first scene information may be associated many-to-one.

[0041] Also, for example, a management table associating first scene information with second scene information may be stored in the auxiliary storage unit 203. FIG. 9 is a diagram showing an example of the management table 221 stored in the auxiliary storage unit 203. In the management table 221, first scene information is associated with second scene information. Also, in the management table 221, multiple pieces of second scene information may be associated with one piece of first scene information. The determination unit 22 may refer to the management table 221 to determine the first scene information corresponding to the second scene information.

[0042] The second acquisition unit 23 inputs the first scene information determined by the determination unit 22 into the realism enhancement model 25, and acquires the realism enhancement processing to be applied to the scene indicated by the first scene information.

[0043] The execution unit 24 executes the realism enhancement process by driving the display 3, speakers 411L, 411R, backrest vibration unit 412, seat vibration unit 413, chair rocking unit 414, and air blower 415 in accordance with the parameters of the realism enhancement process acquired by the second acquisition unit 23, and instructs the display 3 and speakers 411L, 411R to output the scene acquired by the first acquisition unit 21. For example, the execution unit 24 drives the backrest vibration unit 412 and the seat vibration unit 413 to vibrate the seat 41 and the backrest 42 at a specified amplitude and frequency in accordance with the realism enhancement process acquired by the second acquisition unit 23. In addition, the execution unit 24 rocks the seat 41 back and forth and left and right, for example, by vibrating the chair rocking unit 414 at an amplitude and frequency specified in the realism enhancement process. Furthermore, the execution unit 24 rotates the fan of the air blower 415 at a rotation speed specified in the realism enhancement process, for example, to blow air toward the user seated in the VR chair 4. The execution unit 24 changes the brightness of the scene acquired by the first acquisition unit 21, for example, according to the brightness specified in the realism enhancement process, and outputs the changed brightness to the display 3. The execution unit 24 changes the volume of the sound output in the scene, for example, according to the volume specified in the realism enhancement process, and outputs the changed brightness to the speakers 411L and 411R.

[0044] <Processing flow of the embodiment> FIG. 10 is a diagram showing an example of a processing flow of the control device 2 according to the embodiment. The processing in FIG. 10 is started, for example, when a user issues an instruction to play content. An example of the processing flow of the control device 2 will be described with reference to FIG.

[0045] In step S1, the first acquisition unit 21 acquires a scene included in the content instructed to be played. In step S2, the first acquisition unit 21 inputs the scene acquired in step S1 into the detection model 26, thereby acquiring second scene information.

[0046] In step S3, the determination unit 22 determines first scene information corresponding to the second scene information acquired in step S2. For example, the determination unit 22 calculates the similarity between each piece of first scene information stored in the realism enhancement model 25 and the second scene information. Then, the determination unit 22 determines the first scene information for which the highest similarity is calculated as the first scene information corresponding to the second scene information acquired in step S2.

[0047] In step S4, the second acquisition unit 23 inputs the first scene information determined in step S3 into the realism enhancement model 25 to acquire the realism enhancement processing to be applied to the scene acquired in step S1. In step S5, the execution unit 24 applies the realism enhancement processing acquired in step S4 to the scene acquired in step S1. The execution unit 24 outputs the scene to which the realism enhancement processing has been applied to the display 3 and the speakers 411L and 411R. In addition, the execution unit 24 drives the backrest vibration unit 412, the seat vibration unit 413, the chair rocking unit 414, and the air blower 415 in accordance with the realism enhancement processing acquired in step S4. If the processing for all scenes included in the content 900 has been completed (YES in step S6), the processing is terminated. If an unprocessed scene remains (NO in step S6), the processing from step S1 is executed for the next scene.

[0048] <Effects of the embodiment> If the realism enhancement model 25 and the detection model 26 are designed separately, it may happen that the first scene information does not correspond to the second scene information. As a result, even if the second scene information is output by the detection model 26, it may happen that the realism enhancement processing corresponding to the second scene information cannot be acquired from the realism enhancement model 25. As a result, it may happen that the realism enhancement processing cannot be applied to the scene.

[0049] In this embodiment, the determination unit 22 determines the first scene information corresponding to the second scene information. Therefore, even if the realism enhancement model 25 and the detection model 26 are designed separately, it is possible to identify a realism enhancement process corresponding to the second scene information detected by the detection model 26. Furthermore, even if one of the realism enhancement model 25 and the detection model 26 is updated, the determination unit 22 associates the first scene information with the second scene information, so it is possible to identify a realism enhancement process corresponding to the scene detected by the detection model 26. In other words, even if one of the realism enhancement model 25 and the detection model 26 is updated, it is possible to execute a realism enhancement process for a scene.

[0050] During operation of the entertainment system 1, the realism enhancement model 25 and the detection model 26 may be updated in order to increase the number of scenes to be subjected to the realism enhancement process or to classify scenes more finely and perform the realism enhancement process. For example, if the detection model 26 is updated to add second scene information corresponding to a new scene, but the realism enhancement model 25 is not updated, it is conceivable that the realism enhancement process for the added new scene will not be output by the realism enhancement model 25. According to this embodiment, the determination unit 22 determines first scene information corresponding to the second scene information using the similarity between the first scene information and the second scene information. Therefore, even if second scene information corresponding to a new scene is added to the detection model 26, the determination unit 22 determines first scene information that has a high similarity to the added second scene information. Consequently, according to this embodiment, even if second scene information corresponding to a new scene is added to the detection model 26, the determination unit 22 determines first scene information that has a high similarity to the added second scene information. Consequently, according to this embodiment, even if second scene information corresponding to the new scene is added to the detection model 26, the determination unit 22 determines first scene information that has a high similarity to the added second scene information. Furthermore, according to this embodiment, the sense of realism enhancement model 25 and the detection model 26 can be updated individually, which eliminates the need to synchronize the update timings of the sense of realism enhancement model 25 and the detection model 26, and therefore the sense of realism enhancement model 25 and the detection model 26 can be updated flexibly. Consequently, according to this embodiment, the operational load of the entertainment system 1 can be reduced.

[0051] <Example> The technology described in the embodiment can be applied to various situations. For example, the entertainment system 1 may be applied to a vehicle. That is, the display 3 may be used as a display provided on the console of the vehicle, and the VR chair 4 may be used as a seat for the passenger. The control device 2 may be used in an information processing device such as a navigation system mounted on the vehicle. By applying the entertainment system 1 to the vehicle, it is possible to provide the passengers in the vehicle with content that gives them a sense of realism.

[0052] When the entertainment system 1 is applied to a vehicle, the VR chairs 4 may be used in seats other than the driver's seat. When the VR chairs 4 are used in all seats, the speakers 411L, 411R, the backrest vibration unit 412, the seat vibration unit 413, the chair rocking unit 414, and the air blower unit 415 may be inhibited from being driven while the vehicle is moving. With this configuration, the driver's concentration on driving is inhibited from decreasing while the vehicle is moving.

[0053] <Modification> In the embodiment described above, the VR chair 4 is provided with the speakers 411L, 411R, the backrest vibration unit 412, the seat vibration unit 413, the chair rocking unit 414, and the air blower 415, but some of these components may be omitted. Also, the speakers 411L, 411R may be provided in the display 3.

[0054] In the above-described embodiment, the first scene information and the second scene information are exemplified by name and frequency variations. However, the first scene information and the second scene information may be other information. For example, the first scene information and the second scene information may be signal waveforms representing video and sound in a scene. When the first scene information and the second scene information are signal waveforms, the degree of similarity may be higher as the shapes of the signal waveforms become more similar.

[0055] It is also possible that a plurality of pieces of first scene information having the same similarity to a given piece of second scene information exists. In such a case, it is possible to associate a priority with each piece of first scene information. FIG. 11 is a diagram showing an example of management table 221A. In management table 221A, a priority is added to management table 221. The priority is associated with each piece of first scene information. When first scene information having the same similarity exists, determination unit 22 may determine that the first scene information associated with the highest priority in management table 221A is the first scene information corresponding to the second scene information.

[0056] Furthermore, when there are a plurality of pieces of first scene information with the same similarity, it is possible to execute a mixture of the realism enhancement processes associated with each of the plurality of pieces of first scene information in the realism enhancement model 25. When executing a mixture of the processes, for example, the parameters of the processes executed in the plurality of realism enhancement processes (such as the frequency and magnitude of vibrations by the backrest vibration unit 412, the seat vibration unit 413, and the chair rocking unit 414, the brightness of the video displayed on the display 3, and the rotation speed of the fan of the air blower unit 415) may be averaged and executed.

[0057] In the embodiment described above, one realism enhancement model 25 is provided, but a realism enhancement model 25 may be provided for each type of hardware that executes the realism enhancement process. It is also possible to provide a sense of presence enhancement model 25 for processing, a sense of presence enhancement model 25 for processing to enhance the sense of presence of sounds output from speakers 411L and 411R, a sense of presence enhancement model 25 for processing to enhance the sense of presence of vibrations caused by backrest vibration unit 412, seat vibration unit 413, and chair rocking unit 414, and a sense of presence enhancement model 25 for processing to enhance the sense of presence of air blown by air blower 415. By providing a sense of presence enhancement model 25 for each type of hardware in this way, it becomes easier to execute a sense of presence enhancement process that makes use of the characteristics of each hardware.

[0058] In the embodiment described above, one realism enhancement model 25 is provided, but the realism enhancement model 25 may be provided according to the type of content for which the realism enhancement process is to be executed. That is, a realism enhancement model 25 for the realism enhancement process for video and a realism enhancement model 25 for the realism enhancement process for audio may be provided separately. By providing a realism enhancement model 25 according to the type of content in this way, it becomes easier to execute the realism enhancement process that makes use of the characteristics of each content.

[0059] The embodiments and modifications disclosed above can be combined with each other.

[0060] <Computer-readable recording medium> An information processing program that causes a computer or other machine or device (hereinafter referred to as a computer, etc.) to realize any of the above functions can be recorded on a computer-readable recording medium. Then, by having the computer, etc. read and execute the program from this recording medium, the function can be provided.

[0061] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs electrically, magnetically, optically, mechanically, or chemically and that can be read by a computer. Among such recording media, those that are removable from a computer include, for example, flexible disks, magneto-optical disks, Compact Disc Read Only Memory (CD-ROM), Compact Disc-Recordable (CD-R), Compact Disc-Rewriteable (CD-RW), Digital Versatile Disc (DVD), Blu-ray Disc (BD), Digital Audio Tape (DAT), 8mm tape, flash memory, external hard disk drives, and solid state drives (SSDs). Furthermore, recording media that are fixed to a computer include built-in hard disk drives, SSDs, ROMs, etc. [Explanation of symbols]

[0062] 1. Entertainment System 2. Control device 3. Display 4. VR Chair 21...1st acquisition part 22. Decision section 23...Second Acquisition Department 24. Executive Department 25··Improved Realism Model 26··Detection Model 41··Seat 42 Backrest 43...legs 201 CPU 202...Main memory 203...Auxiliary storage section 204··Communications Department 221 Management Table 221A Management Table 401 CPU 402...Main memory 403...Auxiliary storage section 404··Communications Department 411L··Speaker 411R··Speaker 412 Backrest vibration part 413··Seat vibration part 414··Chair rocker 415··Ventilation section 900··Content B. Content playback system B1 Content playback device B101 Content playback unit B102: Second scene determination unit B103: Second Scene Database B2...Presence enhancement device B201: First scene determination unit B202: First Scene Database B203 Parameter selection section B204 Video Processing Unit B205 Acoustic Processing Unit B206 Vibration processing section B207 Parameter Database B3 Display B4 Speaker B5 Exciter CB1 Content Media C1 Connection Bus C2 Connection Bus N1 Network

Claims

1. 1. A device for enhancing a sense of realism that performs, on content, a sense of realism enhancement process corresponding to first scene information that is scene information identifying a scene in the content and is included in a scene group made up of a plurality of pieces of scene information that are set in advance, acquiring second scene information corresponding to a scene in the content; selecting the first scene information from the scene group according to a similarity with the second scene information; performing a sense of realism enhancement process on the content corresponding to the selected first scene information; A device that enhances the sense of reality.

2. a storage unit that stores a first model that manages correspondence between a scene included in content and first scene information that indicates the scene, and a second model that manages correspondence between second scene information that indicates the scene and a sense of realism enhancement process that is executed for the scene; a processor connected to the storage unit, The processor: using the first model to obtain the first scene information corresponding to a first scene included in the content; calculating a similarity between each of the second scene information managed by the second model and the first scene information acquired from the first model; determining the second scene information corresponding to the first scene information obtained from the first model based on the similarity; inputting the determined second scene information into the second model to obtain the realism enhancement process to be performed on the first scene; instructing an output device to output the first scene on which the acquired realism enhancement processing has been performed; A device that enhances the sense of reality.

3. the first scene information is a first name indicating a feature of the scene; the second scene information is a second name indicating a feature of the scene, the similarity indicates a similarity between the meaning indicated by the first name and the meaning indicated by the second name; The device for enhancing a sense of realism according to claim 2 .

4. the content includes at least one of video and audio; the first scene information is a variation of a first frequency in the scene; the second scene information is a variation of a second frequency in the scene; the similarity indicates a difference between the fluctuation of the first frequency and the fluctuation of the second frequency. The device for enhancing a sense of realism according to claim 2 .

5. the content includes at least one of video and audio; the first scene information is a first signal waveform in the scene; the second scene information is a second signal waveform in the scene, the similarity indicates a similarity between the first signal waveform and the second signal waveform; The device for enhancing a sense of realism according to claim 2 .

6. the second model is associated with a priority for each of the second scene information; When there are a plurality of pieces of second scene information having the same similarity to the first scene information acquired from the first model, the processor selects the second scene information having the highest priority. prioritize the scene information of the first model and determine it as the second scene information corresponding to the first scene information acquired from the first model; The device for enhancing a sense of realism according to claim 2 .

7. 1. A method for enhancing a sense of realism for performing, on content, a sense of realism enhancement process corresponding to first scene information, the first scene information being scene information identifying a scene in the content and included in a scene group made up of a plurality of preset scene information, the method comprising: acquiring second scene information corresponding to a scene in the content; selecting the first scene information from the scene group according to a similarity with the second scene information; performing a sense of realism enhancement process on the content corresponding to the selected first scene information; How to improve sense of reality.

8. a computer including a storage unit that stores a first model that manages correspondence between a scene included in content and first scene information that indicates the scene, and a second model that manages correspondence between second scene information that indicates the scene and a sense of realism enhancement process that is executed for the scene, and a processor connected to the storage unit; using the first model to obtain the first scene information corresponding to a first scene included in the content; calculating a similarity between each of the second scene information managed by the second model and the first scene information acquired from the first model; determining the second scene information corresponding to the first scene information obtained from the first model based on the similarity; inputting the determined second scene information into the second model to obtain the realism enhancement process to be performed on the first scene; instructing an output device to output the first scene on which the acquired realism enhancement processing has been performed; How to improve sense of reality.

9. A computer-executable program for enhancing a sense of realism, the program performing on the content a sense of realism enhancement process corresponding to first scene information included in a scene group made up of a plurality of pieces of scene information that are set in advance and identify scenes in the content, the program comprising: acquiring second scene information corresponding to a scene in the content; selecting the first scene information from the scene group according to a similarity with the second scene information; performing a sense of realism enhancement process on the content corresponding to the selected first scene information; A program to enhance realism.

10. a storage unit that stores a first model that manages correspondence between a scene included in content and first scene information that indicates the scene, and a second model that manages correspondence between second scene information that indicates the scene and a sense of realism enhancement process that is executed for the scene, and a processor connected to the storage unit; using the first model to obtain the first scene information corresponding to a first scene included in the content; calculating a similarity between each of the second scene information managed by the second model and the first scene information acquired from the first model; determining the second scene information corresponding to the first scene information acquired from the first model based on the similarity; inputting the determined second scene information into the second model to obtain the realism enhancement process to be performed on the first scene; instructing an output device to output the first scene on which the realism enhancement processing has been performed; A program to enhance realism.

11. A system for enhancing a sense of presence comprising: a device for enhancing a sense of presence that performs processing to enhance a sense of presence of a video signal, an audio signal, and a vibration signal in a content; a display that displays a video based on the video signal; a speaker that outputs a sound based on the audio signal; and a vibration device that outputs a vibration based on the vibration signal, the device for enhancing a sense of realism performs, on the content, a sense of realism enhancement process corresponding to first scene information that is scene information for identifying a scene in the content and is included in a scene group made up of a plurality of pieces of scene information that are set in advance, acquiring second scene information that is scene information indicating a scene in the content; selecting the first scene information from the scene group according to a similarity with the second scene information; performing a sense of realism enhancement process corresponding to the selected first scene information on a video signal, an audio signal, and a vibration signal in the content; The display includes: displaying an image based on the video signal that has been subjected to the realism enhancement processing; The speaker is outputting audio based on the audio signal that has been subjected to realism enhancement processing; The vibration device is outputting vibrations based on the vibration signals that have been subjected to the realism enhancement processing; Realism enhancement system.

12. A realistic sensation enhancement system including a realistic sensation enhancement device and an output device, The realism enhancing device includes: a storage unit that stores a first model that manages correspondence between a scene included in content and first scene information that indicates the scene, and a second model that manages correspondence between second scene information that indicates the scene and a sense of realism enhancement process that is executed for the scene; a processor connected to the storage unit, The processor: using the first model to obtain the first scene information corresponding to a first scene included in the content; calculating a similarity between each of the second scene information managed by the second model and the first scene information acquired from the first model; determining the second scene information corresponding to the first scene information obtained from the first model based on the similarity; inputting the determined second scene information into the second model to obtain the realism enhancement process to be performed on the first scene; instructing the output device to output the first scene on which the realism enhancement processing has been performed; the output device outputs the first scene on which the realism enhancement processing has been performed. Realism enhancement system.

Images