System

Abstract

A system is provided.SOLUTION: The system includes a means for analyzing video data and recognizing an important event, a means for analyzing audio data and recognizing an important sound field, a means for generating effect data based on the analyzed video data and audio data, a means for transmitting the generated effect data to a control device, and a means for controlling a vibration device and a blowing device based on the effect data received by the control device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, movie theaters have begun offering 4D immersive movies, using effects such as vibrations and wind to enhance the immersive experience. However, systems for realizing these immersive movie experiences at home are limited and expensive, making them difficult to install in the average household. Furthermore, the means for people with visual or hearing impairments to enjoy movies are limited. The purpose of this invention is to realize a cinema-like immersive movie viewing experience at home, providing a new viewing experience for all users. [Means for solving the problem]

[0005] The present invention provides an immersive home theater system that includes the following means: means for analyzing video data to recognize important events and means for analyzing audio data to recognize important sound fields; means for generating effect data such as vibration and wind based on the analyzed video and audio data; and means for transmitting the generated effect data to a control device, and means for controlling a vibration device and a wind blower device based on the effect data received by the control device. This allows users to enjoy immersive movies like those in a movie theater at home, providing a new movie experience for all users.

[0006] "Video data" refers to digital data of visual information contained in movies and video content.

[0007] "Audio data" refers to digital data of auditory information contained in movies and video content.

[0008] An "important event" is a specific scene or movement in a movie that the user needs to experience, and is a visually identifiable occurrence.

[0009] "Important sound fields" are specific acoustic events or acoustic changes in a movie that the user needs to experience, and are auditorily identifiable events.

[0010] "Effect data" is control information for generating bodily sensation effects such as vibrations and wind in response to important events and sound fields analyzed from video data and audio data.

[0011] The "control device" is a device for controlling the operation of the vibration device and the air blowing device based on the effect data.

[0012] A "vibration device" is a device that generates physical vibrations in a sofa or chair on which a user sits, in response to an instruction from a control device.

[0013] The "air blowing device" is a device that, in response to an instruction from a control device, blows air to a user with a specified force and direction.

[0014] "Generative AI model" refers to the algorithms and models of artificial intelligence technology used to recognize important events and sound fields from video and audio data and generate effect data.

[0015] A "timestamp" is data that indicates the time at which a particular event or sound field occurred based on the analyzed video data and audio data. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] ---

[0038] The present invention is an immersive home theater system that utilizes AI. Specific embodiments of the system are described below, and the system's program processing is explained in natural language.

[0039] System Configuration

[0040] The system mainly includes the following components:

[0041] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[0042] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[0043] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[0044] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[0045] 5. User: A person who enjoys immersive movie watching through this system.

[0046] Program processing

[0047] Data analysis by server

[0048] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[0049] Generating efficacy data

[0050] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[0051] Sending and executing effect data

[0052] The server sends the generated effect data to the terminal in real time, and the terminal keeps the received effect data in a standby state and executes the appropriate effect in synchronization with the movie playback timing.

[0053] The device controls the vibration device and the air blowing device. For example, when a movie is about to play and an explosion scene is about to occur, the device instructs the vibration device to generate a "strong vibration" and the air blowing device to generate a "strong wind" moment. This allows the user to experience the movie scene not only visually but also through touch and the wind.

[0054] Specific examples

[0055] Processing thunder scenes

[0056] The user starts playing a movie. If there is thunder in one scene, the following happens:

[0057] 1. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[0058] 2. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[0059] 3. The server sends the effect data to the terminal.

[0060] 4. The terminal synchronizes the timing during movie playback to instruct the vibration device to vibrate strongly and the air blowing device to generate a momentary strong wind.

[0061] 5. While watching this scene, the user will experience the thunder scene in a realistic way, as the sofa or chair vibrates strongly and they will feel a sudden gust of wind.

[0062] Accommodating visually and hearing impaired users

[0063] This system is also suitable for users with visual or hearing impairments. By providing a haptic effect that is independent of video or audio, users can enjoy movie scenes and action through vibrations and wind. This allows all users to be equally immersed in the world of the movie.

[0064] As a result, the present invention provides users with a cinema-like movie-watching experience at home, making the overall movie-watching experience more interactive and immersive.

[0065] ---

[0066] The processing flow will be explained below.

[0067] ---

[0068] Step 1: Enter movie data

[0069] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[0070] Step 2: Analyzing the video data

[0071] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[0072] Step 3: Analyzing the audio data

[0073] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[0074] Step 4: Generate efficacy data

[0075] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[0076] Step 5: Submitting your effectiveness data

[0077] The server transmits the generated effect data to the device in real time, including the vibration intensity, duration, pattern, wind power, direction, and wind duration.

[0078] Step 6: Receiving and preparing effectiveness data

[0079] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[0080] Step 7: Synchronize movie playback

[0081] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[0082] Step 8: Implementing the vibration effect

[0083] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration for an explosion scene, a medium vibration for a fall scene, etc.

[0084] Step 9: Implementing the blowing effect

[0085] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[0086] Step 10: User Experience

[0087] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[0088] ---

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Conventional home theater systems can only provide a limited visual and auditory experience, making it difficult to fully experience the reality and immersion of a movie. Furthermore, users with visual or hearing impairments have difficulty fully enjoying the content of a movie. Therefore, a means was needed to enable all users to enjoy watching movies equally.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes: means for analyzing video data and identifying important events; means for analyzing audio data and identifying important sound fields; means for generating physical effect data based on the analyzed video data and audio data; means for transmitting the generated physical effect data to a control device; means for controlling a vibration device and an air blower based on the physical effect data received by the control device; and means for simultaneously analyzing the video data and audio data and generating time-stamped effect data for a specific scene. This allows users to experience movie scenes not only visually but also through touch and wind, significantly improving the sense of immersion in the movie. Furthermore, all users, including those with visual or hearing impairments, can enjoy a realistic movie-watching experience.

[0094] "Video data" refers to electronic data containing visual information such as movies, videos, and animations.

[0095] "Audio data" refers to electronic data containing auditory information such as movies, videos, and music.

[0096] Analysis is the process of examining and breaking down data to identify meaning and patterns.

[0097] A "significant event" is a scene or action in a film or video that is particularly notable visually or audibly.

[0098] "Important sound fields" refer to areas in the audio data where specific frequencies or sound pressure levels are prominent.

[0099] "Physical effects data" refers to data for instructing physical effects such as vibrations and wind that correspond to specific scenes in a movie or video.

[0100] The "control device" is a device that operates the vibration device and the air blower based on the received physical effect data.

[0101] A "vibrator" is a device that delivers physical vibrations to a user at a specified intensity and pattern.

[0102] A "blower" is a device that delivers physical wind to a user at a specified wind force and direction.

[0103] A "timestamp" indicates the date and time when a particular data or event occurred.

[0104] A "generative AI model" refers to a trained model that uses artificial intelligence to analyze data and generate effectiveness data.

[0105] This invention is an immersive home theater system that utilizes AI technology. This system is designed to allow users to experience realistic vibrations and wind while watching a movie. Specifically, the system analyzes video and audio data and controls vibration and airflow devices to generate physical effects according to the movie scene.

[0106] System Configuration

[0107] This system consists of the following hardware and software:

[0108] 1. Server: Analyzes video and audio data and generates physical effect data.

[0109] 2. Terminal: Receives effect data sent from the server and controls the vibration device and the air blower.

[0110] 3. Vibration device: Attached to the sofa or chair on which the user sits, it generates specified vibrations.

[0111] 4. Blower: Placed around the user and generates a specified amount of wind.

[0112] Program processing

[0113] Data analysis by server

[0114] The server receives the movie video and audio data sent by the user. The server is equipped with a generative AI model (e.g., a general-purpose AI model) that analyzes the video data frame by frame. For example, the generative AI model detects sudden changes in brightness and specific movements to recognize explosions and action scenes. It also performs spectral analysis of the audio data to identify specific sound fields (e.g., explosions, thunder). The server generates time-stamped physical effect data based on the results of these analyses.

[0115] Generating efficacy data

[0116] Based on the analysis results, the server generates vibration and wind effect data corresponding to a specific scene. For example, it generates strong vibrations for an explosion scene and instantaneous strong winds for a thunder scene. The generated effect data includes the strength, duration, and pattern of vibrations, as well as the wind power, direction, and duration.

[0117] Sending and executing effect data

[0118] The server transmits the generated physical effect data to the device in real time. The device receives this data and controls the vibration device and the air blower in accordance with the timing of the movie playback. For example, when the movie playback reaches an explosion scene, the device instructs the vibration device to generate a vibration of a specified strength and the air blower to generate a specified strong wind.

[0119] Specific examples

[0120] Processing thunder scenes

[0121] The user starts playing a movie. The server detects flashes of lightning from the video data and analyzes the peaks of thunder from the audio data. It then generates data on the effects of strong vibrations and instantaneous wind and sends it to the device. The device then instructs the vibration device to generate strong vibrations and the air blower to generate instantaneous strong winds, synchronizing with the movie playback. The user experiences this scene, greatly enhancing the realism of the movie.

[0122] Prompt Sentence Examples

[0123] An example of a prompt to input to a generative AI model is as follows:

[0124] Generate effects data to control vibration and airflow devices in a thunder scene in a movie. The effects data should include the following information:

[0125] 1. Vibration strength (strong, medium, weak)

[0126] 2. Duration (seconds)

[0127] 3. Vibration pattern (continuous, intermittent)

[0128] 4. Wind power (strong, medium, weak)

[0129] 5. Wind direction (front, back, left, right)

[0130] 6. Wind duration (seconds)

[0131] As a result, by using this invention, users can enjoy a more realistic and immersive movie viewing experience. The system provides tactile and wind experiences in addition to visual and auditory experiences, allowing all users to enjoy movies with a highly immersive feeling.

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1: (User) Play the movie

[0134] A user plays a movie on a device that contains the movie's video and audio data and is configured to stream it to a server in real time. The input is the movie's video and audio data, and the output is the data sent to the server.

[0135] Step 2: (Server) Receiving Data

[0136] The server receives the video and audio data of the movie sent by the user. The input is the video and audio data of the movie, and the output is that this data is stored in the server and prepared for analysis.

[0137] Step 3: (Server) Video data analysis

[0138] A generative AI model (e.g., a general-purpose AI model) in the server analyzes the received video data frame by frame. Specifically, the generative AI model detects sudden changes in brightness and specific movements, and identifies important events such as explosions or action scenes. The input is the video data, and the output is a list of identified events.

[0139] Step 4: (Server) Audio data analysis

[0140] The server analyzes the audio data received in synchronization with the video data. It performs a spectral analysis of the audio data to detect peaks and frequencies of specific sound fields (e.g., explosions, thunder). Specifically, it analyzes specific frequency peaks and sound pressure levels within the audio data to identify important sound fields. The input is the audio data, and the output is a list of identified sound fields.

[0141] Step 5: (Server) Generate effect data

[0142] The server generates physical effect data based on the analysis results. Specifically, it generates vibration effect data and wind effect data corresponding to a specific scene based on the analysis results of the video data and audio data. For example, it generates "strong vibration" for an explosion scene and "instantaneous strong wind" for a thunder scene. The input is a list of identified events and sound fields, and the generated physical effect data is obtained as output.

[0143] Step 6: (Server) Sending effect data

[0144] The server sends the generated physical effect data to the device in real time. Specifically, it assigns a timestamp to the effect data and sends it to the device at the appropriate time. The input is the generated physical effect data, and the output is the physical effect data with the timestamp sent to the device.

[0145] Step 7: (Terminal) Receive effect data and wait for execution

[0146] The device holds the physical effect data received from the server in a standby state. Specifically, the received data is stored in memory and placed in a standby state as appropriate in accordance with the timing of movie playback. The input is the time-stamped physical effect data sent from the server, and the output is the data in a standby state.

[0147] Step 8: (Terminal) Control of vibration and blower devices

[0148] The device executes the received effect data in synchronization with the movie scenes. Specifically, it issues instructions to a vibration device and a blower to generate physical effects in accordance with the movie playback. For example, when the movie playback reaches an explosion scene, it issues an instruction to a vibration device to generate a "strong vibration" and an instruction to a blower to generate a "momentary strong wind." The input is the physical effect data in a standby state, and the output is the executed physical effect.

[0149] Step 9: (User) Enjoyment of the experience

[0150] The user experiences vibrations and wind that occur in real time in response to movie scenes. For example, during a thunder scene, the sofa vibrates strongly, and at the same time, a momentary breeze is felt, bringing the scene to life. The input is the physical effect that is being performed, and the output is a realistic movie experience that is felt by the user.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] Conventional immersive systems in home theaters and entertainment facilities rely on visual and auditory perception, which can prevent users from fully experiencing the virtual reality world. This also makes it difficult for visually or auditory impaired users to enjoy the content. Furthermore, physical feedback during events and important scenes is incomplete, creating a need for improved realism and immersion.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for providing physical feedback to the user using a virtual reality device based on the effect data, thereby enabling the user to have a more realistic and immersive experience not only through video and audio but also through tactile sensations and wind.

[0156] "Video data" refers to data containing visual information captured by a video camera or other device.

[0157] "Analysis" is the process of extracting meaningful information from data and understanding and recognizing it.

[0158] A "significant event" is something that is of particular interest to the user, causing a particular action or change in the video or audio.

[0159] "Audio data" refers to data that includes auditory information collected by a microphone or the like.

[0160] An "important sound field" is an acoustic phenomenon that occurs at a particular time or place and is particularly noticeable to the user.

[0161] "Effect data" is information for controlling physical devices that is generated as a result of analyzing video data and audio data.

[0162] The "control device" is a device that receives the transmitted effect data and operates a physical device such as a vibration device or a ventilation device based on the data.

[0163] A "vibration device" is a device that generates physical vibrations in response to input instructions.

[0164] A "blowing device" is a device that generates air based on input instructions.

[0165] A "virtual reality device" is a device that allows a user to visually and aurally experience a virtual environment or situation, and typically includes a head-mounted display and sound system.

[0166] "Physical feedback" refers to providing the user with physical sensations such as vibrations or wind in response to changes in video and audio.

[0167] This invention describes an immersive VR ride attraction system that applies an immersive home theater system using AI. Specific embodiments and system program processing are described below.

[0168] System Configuration

[0169] The system mainly includes the following components:

[0170] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[0171] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. It works in conjunction with the virtual reality equipment.

[0172] 3. Vibration device: A device that generates vibrations in the seat on which the user sits or in the equipment worn by the user.

[0173] 4. Blowing device: A device that makes the user feel the wind at a specific force and direction.

[0174] 5. Virtual reality equipment: A device that allows the user to experience a virtual environment visually and audibly.

[0175] 6. User: A user who enjoys an experiential VR ride attraction through this system.

[0176] Data analysis by server

[0177] The server uses Python to analyze video and audio data. It uses a generative AI model to recognize important events in each frame of the video data. It uses the SciPy library to perform spectral analysis on the audio data to identify important sound fields. Specifically, it detects data corresponding to specific events, such as explosions, falls, or thunder. Based on the information obtained from this analysis, it generates effect data to control vibration and airflow devices.

[0178] Generating efficacy data

[0179] The server generates vibration and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. Based on the audio data, it generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[0180] Sending and executing effect data

[0181] The server sends the generated effect data to the device in real time. The device keeps the received effect data in a standby state and synchronizes with the virtual reality device to execute the effect at the appropriate time. The device controls the vibration device and air blowing device using a control program in Unity or Arduino. This allows the user to experience the movie scenes in a very realistic way, not only through their eyes but also through their sense of touch and wind.

[0182] Specific examples

[0183] A specific example will be described below, where thunder rumbles in a certain scene.

[0184] 1. The user starts the attraction and puts on the VR headset.

[0185] 2. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[0186] 3. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[0187] 4. The server sends the effect data to the terminal.

[0188] 5. The terminal instructs the vibration device to vibrate strongly and the air blowing device to blow a momentary strong wind in time with the playback timing of the VR attraction.

[0189] 6. While experiencing this scene, the user feels the seat and the wearable device vibrate strongly, and at the same time, they feel a momentary gust of wind, allowing them to experience the thunder scene in a realistic way.

[0190] Prompt Sentence Examples

[0191] "I want to design a system that will create strong vibrations and wind effects when an explosion occurs during a VR experience. To achieve this, I will create an AI program that will detect important events and generate the corresponding vibration and wind effects."

[0192] As a result, the present invention provides users with a more immersive and realistic experience in virtual reality, making the overall entertainment experience more interactive and immersive.

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] The server receives video data. The input is video data obtained from a video camera or other capture device. The server uses a generative AI model to analyze each frame of the video and recognize important events (e.g., explosions, falls). The output is a list of important events detected in the video.

[0196] Step 2:

[0197] The server receives audio data. The input is audio data collected from a microphone or other device. The server uses the SciPy library to perform spectral analysis of the audio data and identify important sound fields (e.g., thunder, explosions). The output is a list of important sound fields recognized in the audio.

[0198] Step 3:

[0199] The server generates effect data based on the analysis results of the video and audio data obtained in steps 1 and 2. The inputs are a list of important events and a list of sound fields. The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. The output is a set of corresponding effect data.

[0200] Step 4:

[0201] The server sends the generated effect data to the terminal. The input is a set of effect data. The effect data is sent from the server in real time. The output is the terminal that receives the effect data.

[0202] Step 5:

[0203] The terminal keeps the received effect data in a standby state. The input is the effect data sent from the server. The terminal synchronizes with the playback timing of the movie or VR attraction and prepares to execute the appropriate effect. The output is in a ready state.

[0204] Step 6:

[0205] The terminal controls the vibration device and the air blowing device in accordance with the playback timing. The input is the effect data and playback timing. The terminal uses a control program in Unity or Arduino to operate the physical devices with the specified intensity and duration. Specific actions include the seat or wearable device vibrating and the air blowing device generating wind. The output is the vibration and wind experienced by the user.

[0206] Through these steps, users can experience realistic changes in images and sounds not only through their eyes but also through their sense of touch and wind, making it possible to provide a truly immersive entertainment experience in the virtual reality world.

[0207] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0208] ---

[0209] This invention combines an AI-based immersive home theater system with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are shown below, and the system's program processing is explained in natural language.

[0210] System Configuration

[0211] The system mainly includes the following components:

[0212] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[0213] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[0214] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[0215] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[0216] 5. Emotion engine: A device that recognizes a user's emotions by analyzing their facial expressions, voice tone, or biometric data.

[0217] 6. User: A person who enjoys immersive movie watching through this system.

[0218] Program processing

[0219] Data analysis by server

[0220] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[0221] Efficacy data generation and adjustment

[0222] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[0223] Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, or biometric data to recognize the user's current emotion. For example, if the user is feeling scared, the emotion engine will detect this and adjust the effects of the vibration and airflow devices.

[0224] Sending and executing effect data

[0225] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing. It is also possible to dynamically adjust the effect data in real time based on data from the emotion engine.

[0226] For example, if the user is very tense, the vibration effect can be reduced and the airflow can be gentler to soothe the experience, while conversely, if the user is relaxed, the effect can be intensified.

[0227] Specific examples

[0228] Processing scary scenes from horror movies

[0229] The user plays a horror movie and reaches a scary scene. The following happens:

[0230] 1. The server detects sudden changes in brightness from the video data and frightening sounds from the audio data, such as a shadow appearing in the dark or a sudden scream.

[0231] 2. The server generates the appropriate vibration and cold wind effect data for the scary scene.

[0232] 3. The emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize that the user is already feeling very scared.

[0233] 4. The server takes into account the data from the emotion engine and reduces the vibration intensity and gently adjusts the fan effect.

[0234] 5. The device will then execute coordinated vibration and airflow effects when the movie reaches a scary scene.

[0235] 6. By experiencing a moderately adjusted immersive effect, users can enjoy a realistic immersive movie experience while mitigating excessive fear.

[0236] Accommodating visually and hearing impaired users

[0237] This system is also suitable for users with visual or hearing impairments. By providing immersive effects that are independent of video and audio, users can enjoy movie scenes and action through vibrations and wind. In addition, the emotion engine understands the user's emotional state and adjusts the appropriate effects, allowing users to enjoy movies comfortably.

[0238] As a result, the present invention provides users with a cinema-like experience of watching movies at home, and by utilizing an emotion engine to create an experience optimized for each user, the overall movie-watching experience becomes more interactive and personalized.

[0239] The processing flow will be explained below.

[0240] ---

[0241] Step 1: Enter movie data

[0242] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[0243] Step 2: Analyzing the video data

[0244] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[0245] Step 3: Analyzing the audio data

[0246] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[0247] Step 4: Generate efficacy data

[0248] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[0249] Step 5: Obtaining emotion data

[0250] The emotion engine monitors the user's facial expressions, voice tone, or biometric data to analyze the user's emotions in real time. For example, the emotion engine can determine whether the user is scared or excited.

[0251] Step 6: Adjusting the effect data

[0252] The server dynamically adjusts the effect data based on the user's emotional data acquired by the emotion engine. For example, if the user is already feeling scared, the server reduces the intensity of the vibrations and softens the airflow.

[0253] Step 7: Submitting your effectiveness data

[0254] The server transmits the generated and adjusted effect data to the device in real time, including vibration intensity, duration, pattern, wind power, direction, and wind duration.

[0255] Step 8: Receiving and preparing effectiveness data

[0256] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[0257] Step 9: Synchronize movie playback

[0258] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[0259] Step 10: Implementing the vibration effect

[0260] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration during an explosion scene, a medium vibration during a fall scene, etc.

[0261] Step 11: Implementing the blowing effect

[0262] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[0263] Step 12: User Experience

[0264] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[0265] ---

[0266] The above is the specific processing flow for an invention that combines an emotion engine with an immersive home theater system that utilizes AI.

[0267] Example 2

[0268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Conventional immersive home theater systems generate effect data based only on video and audio data, without considering the user's emotional state. This results in a problem where a uniform immersive effect is provided regardless of the user's emotional state. In particular, if the user is feeling fear or tension, the experience may become overly stressful due to the lack of appropriate adjustments.

[0270] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for analyzing user emotion data and adjusting the effect data based on the emotion data, means for transmitting the generated effect data to the control device, and means for controlling the vibration device and the air blowing device based on the effect data received by the control device. This makes it possible to dynamically provide optimal bodily sensation effects according to the user's emotional state.

[0271] "Video data" is digital data containing visual information such as movies and videos.

[0272] "Audio data" is digital data containing the sound information of a film or video.

[0273] A "significant event" refers to a specific occurrence in a movie or video scene that has a strong visual and emotional impact on the user.

[0274] "Critical sound fields" refer to specific acoustic environments that have a strong auditory and emotional impact on the user in film and video audio.

[0275] "Effect data" is instruction data for realizing physical effects such as vibrations and airflow that are generated based on the analyzed video data and audio data.

[0276] The "control device" is a device that receives effect data sent from the server and operates the vibration device and the air blowing device based on the effect data.

[0277] A "vibration device" is a device that generates physical vibrations based on generated effect data.

[0278] The "air blowing device" is a device that generates air based on the generated effect data.

[0279] "User's emotional data" is data that indicates the user's current emotional state, analyzed from the user's facial expression, voice tone, biometric information, and the like.

[0280] The "means for adjusting effect data based on emotional data" refers to a method for dynamically changing the intensity and duration of vibrations and airflow, etc., in accordance with the emotional state of the user.

[0281] The present invention is an immersive home theater system that uses AI technology to recognize a user's emotions and adjust the immersive effects according to those emotions, providing a more personalized movie-watching experience. Specific embodiments of the present invention are described in detail below.

[0282] System Configuration

[0283] The system includes the following components:

[0284] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[0285] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[0286] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[0287] 4. Air Blowing Device: A device that blows air at a specified force and direction to the user.

[0288] 5. Emotion Engine: A device that recognizes the user's emotions by analyzing their facial expressions, voice tone, and biometric data.

[0289] 6. User: A person who enjoys immersive movie viewing through this system.

[0290] Program processing

[0291] Data analysis by server

[0292] The server receives the video and audio data of the movie sent by the user. The server analyzes this data using generative AI models such as "TENSORFLOW (registered trademark)" and "PyTorch." For the video data, it recognizes important events in each frame, and for the audio data, it performs spectral analysis using libraries such as "LibROSA" to recognize important sound fields.

[0293] As a specific example of processing, the server detects sudden changes in brightness from video data and specific acoustic patterns (e.g., explosions or thunder) from audio data. Based on these analysis results, effect data is generated.

[0294] Efficacy data generation and adjustment

[0295] The server generates vibration and wind effect data based on the analysis results. For example, it uses SciPy and NumPy to process and calculate the data. For example, it generates "strong vibration" (3 seconds) for an explosion scene, and "instantaneous strong wind" (1 second) for thunder.

[0296] Furthermore, the emotion engine uses OpenCV and DeepFace to analyze the user's facial expressions, voice tone, and biometric data to detect the user's current emotional state. If the user is feeling fear or tension, the emotion engine will collect that information and adjust the effect data.

[0297] For example, if the user is very nervous, the emotion engine will adjust the vibration intensity to 50% and the airflow strength to 20%.

[0298] Sending and executing effect data

[0299] The server sends the generated effect data to the device in real time using a low-latency protocol such as WebSocket. The device keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing.

[0300] As a concrete example, consider the case where a user plays a scary scene from a horror movie. The server adjusts the effect data based on the scene analysis results and emotion engine data, and sends it to the device. The device then executes the adjusted vibration and airflow effects in accordance with the timing of the movie playback. This allows the user to enjoy an appropriately adjusted sensory effect without feeling excessive fear.

[0301] Specific operation example

[0302] Prompt Sentence Examples

[0303] As a specific example of operation, the following prompt sentence is input into the generative AI model.

[0304] 1. Video data analysis:

[0305] "Analyze video data and extract important events using generative AI models that identify explosions and action scenes."

[0306] 2. Analysis of audio data:

[0307] "Use spectral analysis to analyze audio data and recognize important sound fields such as explosions and thunder."

[0308] 3. Emotion Engine Analysis:

[0309] "Detect the user's current emotional state by analyzing their facial expressions, voice tone, and biometric data."

[0310] In this way, the present invention provides users with an optimal immersive movie-watching experience in real time, and utilizes an emotion engine to dynamically adjust effects according to the user's individual emotional state, making the movie-watching experience more interactive and personalized.

[0311] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0312] Processing Steps

[0313] Step 1:

[0314] Receiving movie data

[0315] The server receives the video and audio data of the movie from the user.

[0316] When the user presses the play button on the movie, the device transmits this data to the server in real time.

[0317] Input: Video data, audio data

[0318] Output: Received video and audio data

[0319] Specific behavior:

[0320] When a user presses the play button on a movie, the device uses the HTTP protocol to send video and audio data to the server, which receives it and prepares it for analysis.

[0321] Step 2:

[0322] Video data analysis

[0323] The server analyzes the received video data.

[0324] Uses a generative AI model (e.g., TensorFlow) to recognize important events in each frame.

[0325] Input: Video data

[0326] Output: List of significant events (e.g. explosions, action scenes)

[0327] Specific behavior:

[0328] The server analyzes each frame of video data, detecting specific patterns or changes (such as sudden changes in brightness), and uses a generative AI model to write something like "Frame 1500-2000: Explosion detected."

[0329] Step 3:

[0330] Analysis of audio data

[0331] The server analyzes the received voice data.

[0332] Spectral analysis is performed using libraries such as "LibROSA" to recognize important sound fields.

[0333] Input: Audio data

[0334] Output: List of important sound fields (e.g. explosions, thunder)

[0335] Specific behavior:

[0336] The server performs spectral analysis on the audio data to detect specific acoustic patterns (e.g. explosions, thunder), and lists information such as "Timestamp 00:10:30 - Thunderstorm sound detected."

[0337] Step 4:

[0338] Generating efficacy data

[0339] The server generates effect data based on the analyzed video data and audio data.

[0340] Calculate the vibration and ventilation effect data using "SciPy" and "NumPy".

[0341] Input: List of important events, list of important sound fields

[0342] Output: Effect data (e.g. strong vibration, momentary strong wind)

[0343] Specific behavior:

[0344] The server generates effect data such as "strong vibration" for explosion scenes and "instant strong wind" for thunder. For example, create "Effect 1: Strong vibration for 3 seconds" and "Effect 2: Quick strong wind for 1 second."

[0345] Step 5:

[0346] Emotional data analysis and effect adjustment

[0347] The emotion engine analyzes the user's facial expressions, voice tone, and biometric data.

[0348] "OpenCV" and "DeepFace" are used to recognize the user's emotions and adjust the effect data.

[0349] Input: User biometric data, emotional data

[0350] Output: Adjusted effect data

[0351] Specific behavior:

[0352] While the user is watching a movie, the emotion engine analyzes the user's face and obtains data such as "User appears scared." Based on this, the server makes adjustments such as "Reduce vibration to 50%" and "Lower wind intensity to 20%."

[0353] Step 6:

[0354] Sending effect data

[0355] The server transmits the generated effect data to the terminal.

[0356] Use low latency protocols (e.g. "WebSocket").

[0357] Input: Efficacy data

[0358] Output: Sends effect data to the terminal.

[0359] Specific behavior:

[0360] The server sends data to the device such as "Send Effect 1: Strong vibration for 3 seconds" and "Send Effect 2: Quick strong wind for 1 second."

[0361] Step 7:

[0362] Execution of effectiveness data

[0363] The terminal executes the received effect data in synchronization with the playback timing of the movie.

[0364] Controls vibration and air blowing devices.

[0365] Input: Efficacy data

[0366] Output: Execute effect device

[0367] Specific behavior:

[0368] Based on the effect data received by the device, it sends commands such as "execute strong vibration for 3 seconds" or "execute a momentary strong wind for 1 second" to the vibration device or air blowing device, allowing the user to experience the sensory effects in real time in line with the movie.

[0369] (Application example 2)

[0370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0371] Conventional immersive home theater systems provide vibration and airflow effects based on video and audio data, but lack the technology to adjust these effects based on the user's emotional state. As a result, excessive vibration and airflow can make users uncomfortable, making it difficult to provide an optimal experience tailored to each individual user's emotions. The present invention aims to solve this problem and provide an immersive movie-watching experience optimized for each user.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for analyzing the user's emotional state and adjusting the effect data based on the results. This makes it possible to provide a bodily sensation effect that is appropriately adjusted according to the user's emotions.

[0373] "Video data" is digital data that contains visual information such as movies and videos.

[0374] "Audio data" is digital data containing audio information contained in movies and videos.

[0375] "Important events" are information that refer to scenes or actions in a movie or video that deserve particular attention.

[0376] "Important sound fields" refers to information that indicates scenes in movies or videos that have particularly impressive sound effects.

[0377] "Effect data" is information relating to bodily sensation effects such as vibration and airflow that are generated based on video data and audio data.

[0378] The "control device" is a device that operates the vibration device and the air blowing device based on the generated effect data.

[0379] A "vibration device" is a device that generates physical vibrations on a sofa, chair, or other device on which a user sits.

[0380] A "blowing device" is a device that allows a user to feel wind with a specified wind force and direction.

[0381] "User's emotional state" refers to the psychological state obtained by analyzing the user's facial expressions, voice, and biometric data.

[0382] "Analyzing" is the process of examining data in detail to reveal its components and patterns.

[0383] "Adjusting" is a means of making appropriate corrections or amendments based on the data and results obtained.

[0384] System configuration and hardware and software usage

[0385] The present invention is implemented by a system that mainly includes the following components, all of which are used to generate sensory effects tailored to a user's emotions and enhance the movie-watching experience:

[0386] 1. Server: A system that analyzes video and audio data and recognizes important events and sound fields. The server performs data analysis using a generative AI model (e.g., OpenAI's GPT-4).

[0387] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. Examples of this include smartphones and smart glasses.

[0388] 3. Vibration device: A device that is attached to a sofa or chair and generates physical vibrations.

[0389] 4. Blower device: A small blower that allows the user to feel the wind.

[0390] 5. Emotion engine: A device that can recognize a user's emotional state by analyzing their facial expressions, voice tone, or biometric data. Examples of this include Microsoft® Azure® Cognitive Services and IBM Watson®.

[0391] Program processing and data processing

[0392] Server-based data analysis and generation of AI models

[0393] The server receives the video and audio data of the movie sent by the user. It then analyzes this data using a generative AI model to recognize important events and sound fields. For example, it identifies sudden scene changes and explosions. Based on the analysis results, the server generates vibration and airflow effect data. Generative AI models such as OpenAI's GPT-4 are used.

[0394] Generating effect data and analyzing the user's emotional state

[0395] The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize the user's emotional state. For example, if the user is feeling fear, the emotion engine can detect this and the server can dynamically adjust the effect data.

[0396] Sending and executing effect data

[0397] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and controls the vibration device and air blower device in sync with the movie playback timing. The effect data can be adjusted in real time based on the emotion engine data.

[0398] Specific examples

[0399] For example, if a user plays a horror movie and reaches a scary scene, the process is as follows:

[0400] 1. The user plays a movie.

[0401] 2. The server analyzes the video and audio data to detect shadows appearing in the darkness or sudden screams.

[0402] 3. The server generates vibration and cold wind effect data suitable for the horror scene.

[0403] 4. The emotion engine analyzes the user's facial expressions and biometric data and recognizes that they are already feeling extremely scared.

[0404] 5. The server takes these data into account and reduces the vibration intensity and gently adjusts the airflow effect.

[0405] 6. The device will execute vibration and airflow effects according to the adjusted effect data, timed to coincide with the scary scenes in the movie.

[0406] Example prompts to input to the generative AI model

[0407] The following prompt sentence can be used to have the system analyze the important events in the movie:

[0408] Prompt statement:

[0409] "Analyze the following video and audio data and recognize important events (e.g. explosions, sudden scene changes). Then generate an algorithm that adjusts the vibration and airflow effects based on the user's emotional state."

[0410] Video data: (video stream)

[0411] Audio data: (audio stream)

[0412] User emotion data: (facial expression and voice analysis data)

[0413] output:

[0414] Scene Events: Important events with time stamps

[0415] Effect data: appropriate vibration and airflow effects for each event (intensity, duration, pattern, wind force, direction)

[0416] Adjustment data: Adjustment data based on the user's emotional state

[0417] In this way, it is possible to provide a sensory effect that corresponds to the user's emotions, and to realize a home theater experience that is optimized for each individual user.

[0418] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0419] Step 1:

[0420] When a user plays a movie, the device transmits the movie's video and audio data to the server. This allows the server to receive the movie stream in real time. The input data are video and audio streams. Specifically, the video and audio captured by a smartphone or smart glasses are transferred to the server via the Internet.

[0421] Step 2:

[0422] The server inputs the video data into a generative AI model (such as GPT-4) to recognize important events. The video data is analyzed frame by frame to identify scenes such as explosions and falls. The generative AI model then timestamps these important events and creates an event list. The input data is the video data, and the output data is a list of important events with timestamps.

[0423] Step 3:

[0424] The server performs spectral analysis of the audio data to recognize important sound fields. For example, it identifies specific sound fields, such as roars or screams, by detecting sound peaks and specific frequency components. This is also done using a generative AI model. The input data is the audio data, and the output data is a time-stamped list of important sound fields.

[0425] Step 4:

[0426] The server generates effect data based on the important event list and the important sound field list. For example, it generates strong vibration effect data for an explosion scene, and instantaneous strong wind effect data for thunder. The input data are the important event list and the important sound field list, and the output data is the effect data. Specifically, detailed information such as the intensity, duration, pattern, wind force, direction, and wind duration of the vibrations is included.

[0427] Step 5:

[0428] The server transmits the generated effect data to the terminal in real time. The terminal stores the received effect data and keeps it in a standby state for execution. The input data is the effect data, and the output data is the effect data transmitted to the terminal. Specifically, the effect data is transmitted via wireless communication or the Internet.

[0429] Step 6:

[0430] The emotion engine analyzes the user's facial expression, voice tone, and biometric data to recognize the user's emotional state. For example, it detects whether the user's eyes are wide open in surprise or their voice is trembling in fear. The input data is the user's facial expression data, voice data, and biometric data, and the output data is the user's emotional state data.

[0431] Step 7:

[0432] The server dynamically adjusts the effect data based on the emotion engine data. For example, if the user is very nervous, it will reduce the vibration intensity and shorten the duration of the airflow. The input data is the emotional state data and existing effect data, and the output data is the adjusted effect data.

[0433] Step 8:

[0434] The terminal synchronizes with the playback timing of the movie and controls the vibration device and the air blowing device according to the adjusted effect data. This allows the user to experience appropriate bodily sensations according to the scene in the movie. The input data is the adjusted effect data, and the output data is the execution of physical vibrations and air blowing. Specifically, the terminal transmits control signals to operate the vibration device and the air blowing device based on the effect data.

[0435] In this way, the present invention provides a user with a personalized, immersive movie-watching experience.

[0436] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0437] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0438] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0439] [Second embodiment]

[0440] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0441] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0442] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0443] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0444] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0445] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0446] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0447] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0448] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0449] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0450] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0451] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0452] ---

[0453] The present invention is an immersive home theater system that utilizes AI. Specific embodiments of the system are described below, and the system's program processing is explained in natural language.

[0454] System Configuration

[0455] The system mainly includes the following components:

[0456] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[0457] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[0458] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[0459] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[0460] 5. User: A person who enjoys immersive movie watching through this system.

[0461] Program processing

[0462] Data analysis by server

[0463] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[0464] Generating efficacy data

[0465] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[0466] Sending and executing effect data

[0467] The server sends the generated effect data to the terminal in real time, and the terminal keeps the received effect data in a standby state and executes the appropriate effect in synchronization with the movie playback timing.

[0468] The device controls the vibration device and the air blowing device. For example, when a movie is about to play and an explosion scene is about to occur, the device instructs the vibration device to generate a "strong vibration" and the air blowing device to generate a "strong wind" moment. This allows the user to experience the movie scene not only visually but also through touch and the wind.

[0469] Specific examples

[0470] Processing thunder scenes

[0471] The user starts playing a movie. If there is thunder in one scene, the following happens:

[0472] 1. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[0473] 2. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[0474] 3. The server sends the effect data to the terminal.

[0475] 4. The terminal synchronizes the timing during movie playback to instruct the vibration device to vibrate strongly and the air blowing device to generate a momentary strong wind.

[0476] 5. While watching this scene, the user will experience the thunder scene in a realistic way, as the sofa or chair vibrates strongly and they will feel a sudden gust of wind.

[0477] Accommodating visually and hearing impaired users

[0478] This system is also suitable for users with visual or hearing impairments. By providing a haptic effect that is independent of video or audio, users can enjoy movie scenes and action through vibrations and wind. This allows all users to be equally immersed in the world of the movie.

[0479] As a result, the present invention provides users with a cinema-like movie-watching experience at home, making the overall movie-watching experience more interactive and immersive.

[0480] ---

[0481] The processing flow will be explained below.

[0482] ---

[0483] Step 1: Enter movie data

[0484] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[0485] Step 2: Analyzing the video data

[0486] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[0487] Step 3: Analyzing the audio data

[0488] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[0489] Step 4: Generate efficacy data

[0490] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[0491] Step 5: Submitting your effectiveness data

[0492] The server transmits the generated effect data to the device in real time, including the vibration intensity, duration, pattern, wind power, direction, and wind duration.

[0493] Step 6: Receiving and preparing effectiveness data

[0494] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[0495] Step 7: Synchronize movie playback

[0496] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[0497] Step 8: Implementing the vibration effect

[0498] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration for an explosion scene, a medium vibration for a fall scene, etc.

[0499] Step 9: Implementing the blowing effect

[0500] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[0501] Step 10: User Experience

[0502] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[0503] ---

[0504] Example 1

[0505] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0506] Conventional home theater systems can only provide a limited visual and auditory experience, making it difficult to fully experience the reality and immersion of a movie. Furthermore, users with visual or hearing impairments have difficulty fully enjoying the content of a movie. Therefore, a means was needed to enable all users to enjoy watching movies equally.

[0507] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0508] In this invention, the server includes: means for analyzing video data and identifying important events; means for analyzing audio data and identifying important sound fields; means for generating physical effect data based on the analyzed video data and audio data; means for transmitting the generated physical effect data to a control device; means for controlling a vibration device and an air blower based on the physical effect data received by the control device; and means for simultaneously analyzing the video data and audio data and generating time-stamped effect data for a specific scene. This allows users to experience movie scenes not only visually but also through touch and wind, significantly improving the sense of immersion in the movie. Furthermore, all users, including those with visual or hearing impairments, can enjoy a realistic movie-watching experience.

[0509] "Video data" refers to electronic data containing visual information such as movies, videos, and animations.

[0510] "Audio data" refers to electronic data containing auditory information such as movies, videos, and music.

[0511] Analysis is the process of examining and breaking down data to identify meaning and patterns.

[0512] A "significant event" is a scene or action in a film or video that is particularly notable visually or audibly.

[0513] "Important sound fields" refer to areas in the audio data where specific frequencies or sound pressure levels are prominent.

[0514] "Physical effects data" refers to data for instructing physical effects such as vibrations and wind that correspond to specific scenes in a movie or video.

[0515] The "control device" is a device that operates the vibration device and the air blower based on the received physical effect data.

[0516] A "vibrator" is a device that delivers physical vibrations to a user at a specified intensity and pattern.

[0517] A "blower" is a device that delivers physical wind to a user at a specified wind force and direction.

[0518] A "timestamp" indicates the date and time when a particular data or event occurred.

[0519] A "generative AI model" refers to a trained model that uses artificial intelligence to analyze data and generate effectiveness data.

[0520] This invention is an immersive home theater system that utilizes AI technology. This system is designed to allow users to experience realistic vibrations and wind while watching a movie. Specifically, the system analyzes video and audio data and controls vibration and airflow devices to generate physical effects according to the movie scene.

[0521] System Configuration

[0522] This system consists of the following hardware and software:

[0523] 1. Server: Analyzes video and audio data and generates physical effect data.

[0524] 2. Terminal: Receives effect data sent from the server and controls the vibration device and the air blower.

[0525] 3. Vibration device: Attached to the sofa or chair on which the user sits, it generates specified vibrations.

[0526] 4. Blower: Placed around the user and generates a specified amount of wind.

[0527] Program processing

[0528] Data analysis by server

[0529] The server receives the movie video and audio data sent by the user. The server is equipped with a generative AI model (e.g., a general-purpose AI model) that analyzes the video data frame by frame. For example, the generative AI model detects sudden changes in brightness and specific movements to recognize explosions and action scenes. It also performs spectral analysis of the audio data to identify specific sound fields (e.g., explosions, thunder). The server generates time-stamped physical effect data based on the results of these analyses.

[0530] Generating efficacy data

[0531] Based on the analysis results, the server generates vibration and wind effect data corresponding to a specific scene. For example, it generates strong vibrations for an explosion scene and instantaneous strong winds for a thunder scene. The generated effect data includes the strength, duration, and pattern of vibrations, as well as the wind power, direction, and duration.

[0532] Sending and executing effect data

[0533] The server transmits the generated physical effect data to the device in real time. The device receives this data and controls the vibration device and the air blower in accordance with the timing of the movie playback. For example, when the movie playback reaches an explosion scene, the device instructs the vibration device to generate a vibration of a specified strength and the air blower to generate a specified strong wind.

[0534] Specific examples

[0535] Processing thunder scenes

[0536] The user starts playing a movie. The server detects flashes of lightning from the video data and analyzes the peaks of thunder from the audio data. It then generates data on the effects of strong vibrations and instantaneous wind and sends it to the device. The device then instructs the vibration device to generate strong vibrations and the air blower to generate instantaneous strong winds, synchronizing with the movie playback. The user experiences this scene, greatly enhancing the realism of the movie.

[0537] Prompt Sentence Examples

[0538] An example of a prompt to input to a generative AI model is as follows:

[0539] Generate effects data to control vibration and airflow devices in a thunder scene in a movie. The effects data should include the following information:

[0540] 1. Vibration strength (strong, medium, weak)

[0541] 2. Duration (seconds)

[0542] 3. Vibration pattern (continuous, intermittent)

[0543] 4. Wind power (strong, medium, weak)

[0544] 5. Wind direction (front, back, left, right)

[0545] 6. Wind duration (seconds)

[0546] As a result, by using this invention, users can enjoy a more realistic and immersive movie viewing experience. The system provides tactile and wind experiences in addition to visual and auditory experiences, allowing all users to enjoy movies with a highly immersive feeling.

[0547] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0548] Step 1: (User) Play the movie

[0549] A user plays a movie on a device that contains the movie's video and audio data and is configured to stream it to a server in real time. The input is the movie's video and audio data, and the output is the data sent to the server.

[0550] Step 2: (Server) Receiving Data

[0551] The server receives the video and audio data of the movie sent by the user. The input is the video and audio data of the movie, and the output is that this data is stored in the server and prepared for analysis.

[0552] Step 3: (Server) Video data analysis

[0553] A generative AI model (e.g., a general-purpose AI model) in the server analyzes the received video data frame by frame. Specifically, the generative AI model detects sudden changes in brightness and specific movements, and identifies important events such as explosions or action scenes. The input is the video data, and the output is a list of identified events.

[0554] Step 4: (Server) Audio data analysis

[0555] The server analyzes the audio data received in synchronization with the video data. It performs a spectral analysis of the audio data to detect peaks and frequencies of specific sound fields (e.g., explosions, thunder). Specifically, it analyzes specific frequency peaks and sound pressure levels within the audio data to identify important sound fields. The input is the audio data, and the output is a list of identified sound fields.

[0556] Step 5: (Server) Generate effect data

[0557] The server generates physical effect data based on the analysis results. Specifically, it generates vibration effect data and wind effect data corresponding to a specific scene based on the analysis results of the video data and audio data. For example, it generates "strong vibration" for an explosion scene and "instantaneous strong wind" for a thunder scene. The input is a list of identified events and sound fields, and the generated physical effect data is obtained as output.

[0558] Step 6: (Server) Sending effect data

[0559] The server sends the generated physical effect data to the device in real time. Specifically, it assigns a timestamp to the effect data and sends it to the device at the appropriate time. The input is the generated physical effect data, and the output is the physical effect data with the timestamp sent to the device.

[0560] Step 7: (Terminal) Receive effect data and wait for execution

[0561] The device holds the physical effect data received from the server in a standby state. Specifically, the received data is stored in memory and placed in a standby state as appropriate in accordance with the timing of movie playback. The input is the time-stamped physical effect data sent from the server, and the output is the data in a standby state.

[0562] Step 8: (Terminal) Control of vibration and blower devices

[0563] The device executes the received effect data in synchronization with the movie scenes. Specifically, it issues instructions to a vibration device and a blower to generate physical effects in accordance with the movie playback. For example, when the movie playback reaches an explosion scene, it issues an instruction to a vibration device to generate a "strong vibration" and an instruction to a blower to generate a "momentary strong wind." The input is the physical effect data in a standby state, and the output is the executed physical effect.

[0564] Step 9: (User) Enjoyment of the experience

[0565] The user experiences vibrations and wind that occur in real time in response to movie scenes. For example, during a thunder scene, the sofa vibrates strongly, and at the same time, a momentary breeze is felt, bringing the scene to life. The input is the physical effect that is being performed, and the output is a realistic movie experience that is felt by the user.

[0566] (Application example 1)

[0567] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0568] Conventional immersive systems in home theaters and entertainment facilities rely on visual and auditory perception, which can prevent users from fully experiencing the virtual reality world. This also makes it difficult for visually or auditory impaired users to enjoy the content. Furthermore, physical feedback during events and important scenes is incomplete, creating a need for improved realism and immersion.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0570] In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for providing physical feedback to the user using a virtual reality device based on the effect data, thereby enabling the user to have a more realistic and immersive experience not only through video and audio but also through tactile sensations and wind.

[0571] "Video data" refers to data containing visual information captured by a video camera or other device.

[0572] "Analysis" is the process of extracting meaningful information from data and understanding and recognizing it.

[0573] A "significant event" is something that is of particular interest to the user, causing a particular action or change in the video or audio.

[0574] "Audio data" refers to data that includes auditory information collected by a microphone or the like.

[0575] An "important sound field" is an acoustic phenomenon that occurs at a particular time or place and is particularly noticeable to the user.

[0576] "Effect data" is information for controlling physical devices that is generated as a result of analyzing video data and audio data.

[0577] The "control device" is a device that receives the transmitted effect data and operates a physical device such as a vibration device or a ventilation device based on the data.

[0578] A "vibration device" is a device that generates physical vibrations in response to input instructions.

[0579] A "blowing device" is a device that generates air based on input instructions.

[0580] A "virtual reality device" is a device that allows a user to visually and aurally experience a virtual environment or situation, and typically includes a head-mounted display and sound system.

[0581] "Physical feedback" refers to providing the user with physical sensations such as vibrations or wind in response to changes in video and audio.

[0582] This invention describes an immersive VR ride attraction system that applies an immersive home theater system using AI. Specific embodiments and system program processing are described below.

[0583] System Configuration

[0584] The system mainly includes the following components:

[0585] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[0586] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. It works in conjunction with the virtual reality equipment.

[0587] 3. Vibration device: A device that generates vibrations in the seat on which the user sits or in the equipment worn by the user.

[0588] 4. Blowing device: A device that makes the user feel the wind at a specific force and direction.

[0589] 5. Virtual reality equipment: A device that allows the user to experience a virtual environment visually and audibly.

[0590] 6. User: A user who enjoys an experiential VR ride attraction through this system.

[0591] Data analysis by server

[0592] The server uses Python to analyze video and audio data. It uses a generative AI model to recognize important events in each frame of the video data. It uses the SciPy library to perform spectral analysis on the audio data to identify important sound fields. Specifically, it detects data corresponding to specific events, such as explosions, falls, or thunder. Based on the information obtained from this analysis, it generates effect data to control vibration and airflow devices.

[0593] Generating efficacy data

[0594] The server generates vibration and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. Based on the audio data, it generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[0595] Sending and executing effect data

[0596] The server sends the generated effect data to the device in real time. The device keeps the received effect data in a standby state and synchronizes with the virtual reality device to execute the effect at the appropriate time. The device controls the vibration device and air blowing device using a control program in Unity or Arduino. This allows the user to experience the movie scenes in a very realistic way, not only through their eyes but also through their sense of touch and wind.

[0597] Specific examples

[0598] A specific example will be described below, where thunder rumbles in a certain scene.

[0599] 1. The user starts the attraction and puts on the VR headset.

[0600] 2. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[0601] 3. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[0602] 4. The server sends the effect data to the terminal.

[0603] 5. The terminal instructs the vibration device to vibrate strongly and the air blowing device to blow a momentary strong wind in time with the playback timing of the VR attraction.

[0604] 6. While experiencing this scene, the user feels the seat and the wearable device vibrate strongly, and at the same time, they feel a momentary gust of wind, allowing them to experience the thunder scene in a realistic way.

[0605] Prompt Sentence Examples

[0606] "I want to design a system that will create strong vibrations and wind effects when an explosion occurs during a VR experience. To achieve this, I will create an AI program that will detect important events and generate the corresponding vibration and wind effects."

[0607] As a result, the present invention provides users with a more immersive and realistic experience in virtual reality, making the overall entertainment experience more interactive and immersive.

[0608] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0609] Step 1:

[0610] The server receives video data. The input is video data obtained from a video camera or other capture device. The server uses a generative AI model to analyze each frame of the video and recognize important events (e.g., explosions, falls). The output is a list of important events detected in the video.

[0611] Step 2:

[0612] The server receives audio data. The input is audio data collected from a microphone or other device. The server uses the SciPy library to perform spectral analysis of the audio data and identify important sound fields (e.g., thunder, explosions). The output is a list of important sound fields recognized in the audio.

[0613] Step 3:

[0614] The server generates effect data based on the analysis results of the video and audio data obtained in steps 1 and 2. The inputs are a list of important events and a list of sound fields. The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. The output is a set of corresponding effect data.

[0615] Step 4:

[0616] The server sends the generated effect data to the terminal. The input is a set of effect data. The effect data is sent from the server in real time. The output is the terminal that receives the effect data.

[0617] Step 5:

[0618] The terminal keeps the received effect data in a standby state. The input is the effect data sent from the server. The terminal synchronizes with the playback timing of the movie or VR attraction and prepares to execute the appropriate effect. The output is in a ready state.

[0619] Step 6:

[0620] The terminal controls the vibration device and the air blowing device in accordance with the playback timing. The input is the effect data and playback timing. The terminal uses a control program in Unity or Arduino to operate the physical devices with the specified intensity and duration. Specific actions include the seat or wearable device vibrating and the air blowing device generating wind. The output is the vibration and wind experienced by the user.

[0621] Through these steps, users can experience realistic changes in images and sounds not only through their eyes but also through their sense of touch and wind, making it possible to provide a truly immersive entertainment experience in the virtual reality world.

[0622] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0623] ---

[0624] This invention combines an AI-based immersive home theater system with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are shown below, and the system's program processing is explained in natural language.

[0625] System Configuration

[0626] The system mainly includes the following components:

[0627] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[0628] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[0629] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[0630] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[0631] 5. Emotion engine: A device that recognizes a user's emotions by analyzing their facial expressions, voice tone, or biometric data.

[0632] 6. User: A person who enjoys immersive movie watching through this system.

[0633] Program processing

[0634] Data analysis by server

[0635] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[0636] Efficacy data generation and adjustment

[0637] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[0638] Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, or biometric data to recognize the user's current emotion. For example, if the user is feeling scared, the emotion engine will detect this and adjust the effects of the vibration and airflow devices.

[0639] Sending and executing effect data

[0640] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing. It is also possible to dynamically adjust the effect data in real time based on data from the emotion engine.

[0641] For example, if the user is very tense, the vibration effect can be reduced and the airflow can be gentler to soothe the experience, while conversely, if the user is relaxed, the effect can be intensified.

[0642] Specific examples

[0643] Processing scary scenes from horror movies

[0644] The user plays a horror movie and reaches a scary scene. The following happens:

[0645] 1. The server detects sudden changes in brightness from the video data and frightening sounds from the audio data, such as a shadow appearing in the dark or a sudden scream.

[0646] 2. The server generates the appropriate vibration and cold wind effect data for the scary scene.

[0647] 3. The emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize that the user is already feeling very scared.

[0648] 4. The server takes into account the data from the emotion engine and reduces the vibration intensity and gently adjusts the fan effect.

[0649] 5. The device will then execute coordinated vibration and airflow effects when the movie reaches a scary scene.

[0650] 6. By experiencing a moderately adjusted immersive effect, users can enjoy a realistic immersive movie experience while mitigating excessive fear.

[0651] Accommodating visually and hearing impaired users

[0652] This system is also suitable for users with visual or hearing impairments. By providing immersive effects that are independent of video and audio, users can enjoy movie scenes and action through vibrations and wind. In addition, the emotion engine understands the user's emotional state and adjusts the appropriate effects, allowing users to enjoy movies comfortably.

[0653] As a result, the present invention provides users with a cinema-like experience of watching movies at home, and by utilizing an emotion engine to create an experience optimized for each user, the overall movie-watching experience becomes more interactive and personalized.

[0654] The processing flow will be explained below.

[0655] ---

[0656] Step 1: Enter movie data

[0657] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[0658] Step 2: Analyzing the video data

[0659] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[0660] Step 3: Analyzing the audio data

[0661] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[0662] Step 4: Generate efficacy data

[0663] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[0664] Step 5: Obtaining emotion data

[0665] The emotion engine monitors the user's facial expressions, voice tone, or biometric data to analyze the user's emotions in real time. For example, the emotion engine can determine whether the user is scared or excited.

[0666] Step 6: Adjusting the effect data

[0667] The server dynamically adjusts the effect data based on the user's emotional data acquired by the emotion engine. For example, if the user is already feeling scared, the server reduces the intensity of the vibrations and softens the airflow.

[0668] Step 7: Submitting your effectiveness data

[0669] The server transmits the generated and adjusted effect data to the device in real time, including vibration intensity, duration, pattern, wind power, direction, and wind duration.

[0670] Step 8: Receiving and preparing effectiveness data

[0671] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[0672] Step 9: Synchronize movie playback

[0673] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[0674] Step 10: Implementing the vibration effect

[0675] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration during an explosion scene, a medium vibration during a fall scene, etc.

[0676] Step 11: Implementing the blowing effect

[0677] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[0678] Step 12: User Experience

[0679] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[0680] ---

[0681] The above is the specific processing flow for an invention that combines an emotion engine with an immersive home theater system that utilizes AI.

[0682] Example 2

[0683] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0684] Conventional immersive home theater systems generate effect data based only on video and audio data, without considering the user's emotional state. This results in a problem where a uniform immersive effect is provided regardless of the user's emotional state. In particular, if the user is feeling fear or tension, the experience may become overly stressful due to the lack of appropriate adjustments.

[0685] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for analyzing user emotion data and adjusting the effect data based on the emotion data, means for transmitting the generated effect data to the control device, and means for controlling the vibration device and the air blowing device based on the effect data received by the control device. This makes it possible to dynamically provide optimal bodily sensation effects according to the user's emotional state.

[0686] "Video data" is digital data containing visual information such as movies and videos.

[0687] "Audio data" is digital data containing the sound information of a film or video.

[0688] A "significant event" refers to a specific occurrence in a movie or video scene that has a strong visual and emotional impact on the user.

[0689] "Critical sound fields" refer to specific acoustic environments that have a strong auditory and emotional impact on the user in film and video audio.

[0690] "Effect data" is instruction data for realizing physical effects such as vibrations and airflow that are generated based on the analyzed video data and audio data.

[0691] The "control device" is a device that receives effect data sent from the server and operates the vibration device and the air blowing device based on the effect data.

[0692] A "vibration device" is a device that generates physical vibrations based on generated effect data.

[0693] The "air blowing device" is a device that generates air based on the generated effect data.

[0694] "User's emotional data" is data that indicates the user's current emotional state, analyzed from the user's facial expression, voice tone, biometric information, and the like.

[0695] The "means for adjusting effect data based on emotional data" refers to a method for dynamically changing the intensity and duration of vibrations and airflow, etc., in accordance with the emotional state of the user.

[0696] The present invention is an immersive home theater system that uses AI technology to recognize a user's emotions and adjust the immersive effects according to those emotions, providing a more personalized movie-watching experience. Specific embodiments of the present invention are described in detail below.

[0697] System Configuration

[0698] The system includes the following components:

[0699] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[0700] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[0701] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[0702] 4. Air Blowing Device: A device that blows air at a specified force and direction to the user.

[0703] 5. Emotion Engine: A device that recognizes the user's emotions by analyzing their facial expressions, voice tone, and biometric data.

[0704] 6. User: A person who enjoys immersive movie viewing through this system.

[0705] Program processing

[0706] Data analysis by server

[0707] The server receives the video and audio data of the movie sent by the user. It analyzes this data using generative AI models such as TensorFlow and PyTorch. The server recognizes important events in each frame of the video data, and performs spectral analysis of the audio data using libraries such as LibROSA to recognize important sound fields.

[0708] As a specific example of processing, the server detects sudden changes in brightness from video data and specific acoustic patterns (e.g., explosions or thunder) from audio data. Based on these analysis results, effect data is generated.

[0709] Efficacy data generation and adjustment

[0710] The server generates vibration and wind effect data based on the analysis results. For example, it uses SciPy and NumPy to process and calculate the data. For example, it generates "strong vibration" (3 seconds) for an explosion scene, and "instantaneous strong wind" (1 second) for thunder.

[0711] Furthermore, the emotion engine uses OpenCV and DeepFace to analyze the user's facial expressions, voice tone, and biometric data to detect the user's current emotional state. If the user is feeling fear or tension, the emotion engine will collect that information and adjust the effect data.

[0712] For example, if the user is very nervous, the emotion engine will adjust the vibration intensity to 50% and the airflow strength to 20%.

[0713] Sending and executing effect data

[0714] The server sends the generated effect data to the device in real time using a low-latency protocol such as WebSocket. The device keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing.

[0715] As a concrete example, consider the case where a user plays a scary scene from a horror movie. The server adjusts the effect data based on the scene analysis results and emotion engine data, and sends it to the device. The device then executes the adjusted vibration and airflow effects in accordance with the timing of the movie playback. This allows the user to enjoy an appropriately adjusted sensory effect without feeling excessive fear.

[0716] Specific operation example

[0717] Prompt Sentence Examples

[0718] As a specific example of operation, the following prompt sentence is input into the generative AI model.

[0719] 1. Video data analysis:

[0720] "Analyze video data and extract important events using generative AI models that identify explosions and action scenes."

[0721] 2. Analysis of audio data:

[0722] "Use spectral analysis to analyze audio data and recognize important sound fields such as explosions and thunder."

[0723] 3. Emotion Engine Analysis:

[0724] "Detect the user's current emotional state by analyzing their facial expressions, voice tone, and biometric data."

[0725] In this way, the present invention provides users with an optimal immersive movie-watching experience in real time, and utilizes an emotion engine to dynamically adjust effects according to the user's individual emotional state, making the movie-watching experience more interactive and personalized.

[0726] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0727] Processing Steps

[0728] Step 1:

[0729] Receiving movie data

[0730] The server receives the video and audio data of the movie from the user.

[0731] When the user presses the play button on the movie, the device transmits this data to the server in real time.

[0732] Input: Video data, audio data

[0733] Output: Received video and audio data

[0734] Specific behavior:

[0735] When a user presses the play button on a movie, the device uses the HTTP protocol to send video and audio data to the server, which receives it and prepares it for analysis.

[0736] Step 2:

[0737] Video data analysis

[0738] The server analyzes the received video data.

[0739] Uses a generative AI model (e.g., TensorFlow) to recognize important events in each frame.

[0740] Input: Video data

[0741] Output: List of significant events (e.g. explosions, action scenes)

[0742] Specific behavior:

[0743] The server analyzes each frame of video data, detecting specific patterns or changes (such as sudden changes in brightness), and uses a generative AI model to write something like "Frame 1500-2000: Explosion detected."

[0744] Step 3:

[0745] Analysis of audio data

[0746] The server analyzes the received voice data.

[0747] Spectral analysis is performed using libraries such as "LibROSA" to recognize important sound fields.

[0748] Input: Audio data

[0749] Output: List of important sound fields (e.g. explosions, thunder)

[0750] Specific behavior:

[0751] The server performs spectral analysis on the audio data to detect specific acoustic patterns (e.g. explosions, thunder), and lists information such as "Timestamp 00:10:30 - Thunderstorm sound detected."

[0752] Step 4:

[0753] Generating efficacy data

[0754] The server generates effect data based on the analyzed video data and audio data.

[0755] Calculate the vibration and ventilation effect data using "SciPy" and "NumPy".

[0756] Input: List of important events, list of important sound fields

[0757] Output: Effect data (e.g. strong vibration, momentary strong wind)

[0758] Specific behavior:

[0759] The server generates effect data such as "strong vibration" for explosion scenes and "instant strong wind" for thunder. For example, create "Effect 1: Strong vibration for 3 seconds" and "Effect 2: Quick strong wind for 1 second."

[0760] Step 5:

[0761] Emotional data analysis and effect adjustment

[0762] The emotion engine analyzes the user's facial expressions, voice tone, and biometric data.

[0763] "OpenCV" and "DeepFace" are used to recognize the user's emotions and adjust the effect data.

[0764] Input: User biometric data, emotional data

[0765] Output: Adjusted effect data

[0766] Specific behavior:

[0767] While the user is watching a movie, the emotion engine analyzes the user's face and obtains data such as "User appears scared." Based on this, the server makes adjustments such as "Reduce vibration to 50%" and "Lower wind intensity to 20%."

[0768] Step 6:

[0769] Sending effect data

[0770] The server transmits the generated effect data to the terminal.

[0771] Use low latency protocols (e.g. "WebSocket").

[0772] Input: Efficacy data

[0773] Output: Sends effect data to the terminal.

[0774] Specific behavior:

[0775] The server sends data to the device such as "Send Effect 1: Strong vibration for 3 seconds" and "Send Effect 2: Quick strong wind for 1 second."

[0776] Step 7:

[0777] Execution of effectiveness data

[0778] The terminal executes the received effect data in synchronization with the playback timing of the movie.

[0779] Controls vibration and air blowing devices.

[0780] Input: Efficacy data

[0781] Output: Execute effect device

[0782] Specific behavior:

[0783] Based on the effect data received by the device, it sends commands such as "execute strong vibration for 3 seconds" or "execute a momentary strong wind for 1 second" to the vibration device or air blowing device, allowing the user to experience the sensory effects in real time in line with the movie.

[0784] (Application example 2)

[0785] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0786] Conventional immersive home theater systems provide vibration and airflow effects based on video and audio data, but lack the technology to adjust these effects based on the user's emotional state. As a result, excessive vibration and airflow can make users uncomfortable, making it difficult to provide an optimal experience tailored to each individual user's emotions. The present invention aims to solve this problem and provide an immersive movie-watching experience optimized for each user.

[0787] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for analyzing the user's emotional state and adjusting the effect data based on the results. This makes it possible to provide a bodily sensation effect that is appropriately adjusted according to the user's emotions.

[0788] "Video data" is digital data that contains visual information such as movies and videos.

[0789] "Audio data" is digital data containing audio information contained in movies and videos.

[0790] "Important events" are information that refer to scenes or actions in a movie or video that deserve particular attention.

[0791] "Important sound fields" refers to information that indicates scenes in movies or videos that have particularly impressive sound effects.

[0792] "Effect data" is information relating to bodily sensation effects such as vibration and airflow that are generated based on video data and audio data.

[0793] The "control device" is a device that operates the vibration device and the air blowing device based on the generated effect data.

[0794] A "vibration device" is a device that generates physical vibrations on a sofa, chair, or other device on which a user sits.

[0795] A "blowing device" is a device that allows a user to feel wind with a specified wind force and direction.

[0796] "User's emotional state" refers to the psychological state obtained by analyzing the user's facial expressions, voice, and biometric data.

[0797] "Analyzing" is the process of examining data in detail to reveal its components and patterns.

[0798] "Adjusting" is a means of making appropriate corrections or amendments based on the data and results obtained.

[0799] System configuration and hardware and software usage

[0800] The present invention is implemented by a system that mainly includes the following components, all of which are used to generate sensory effects tailored to a user's emotions and enhance the movie-watching experience:

[0801] 1. Server: A system that analyzes video and audio data and recognizes important events and sound fields. The server performs data analysis using a generative AI model (e.g., OpenAI's GPT-4).

[0802] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. Examples of this include smartphones and smart glasses.

[0803] 3. Vibration device: A device that is attached to a sofa or chair and generates physical vibrations.

[0804] 4. Blower device: A small blower that allows the user to feel the wind.

[0805] 5. Emotion engine: A device that can recognize a user's emotional state by analyzing their facial expressions, voice tone, or biometric data. Examples include Microsoft Azure Cognitive Services and IBM Watson.

[0806] Program processing and data processing

[0807] Server-based data analysis and generation of AI models

[0808] The server receives the video and audio data of the movie sent by the user. It then analyzes this data using a generative AI model to recognize important events and sound fields. For example, it identifies sudden scene changes and explosions. Based on the analysis results, the server generates vibration and airflow effect data. Generative AI models such as OpenAI's GPT-4 are used.

[0809] Generating effect data and analyzing the user's emotional state

[0810] The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize the user's emotional state. For example, if the user is feeling fear, the emotion engine can detect this and the server can dynamically adjust the effect data.

[0811] Sending and executing effect data

[0812] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and controls the vibration device and air blower device in sync with the movie playback timing. The effect data can be adjusted in real time based on the emotion engine data.

[0813] Specific examples

[0814] For example, if a user plays a horror movie and reaches a scary scene, the process is as follows:

[0815] 1. The user plays a movie.

[0816] 2. The server analyzes the video and audio data to detect shadows appearing in the darkness or sudden screams.

[0817] 3. The server generates vibration and cold wind effect data suitable for the horror scene.

[0818] 4. The emotion engine analyzes the user's facial expressions and biometric data and recognizes that they are already feeling extremely scared.

[0819] 5. The server takes these data into account and reduces the vibration intensity and gently adjusts the airflow effect.

[0820] 6. The device will execute vibration and airflow effects according to the adjusted effect data, timed to coincide with the scary scenes in the movie.

[0821] Example prompts to input to the generative AI model

[0822] The following prompt sentence can be used to have the system analyze the important events in the movie:

[0823] Prompt statement:

[0824] "Analyze the following video and audio data and recognize important events (e.g. explosions, sudden scene changes). Then generate an algorithm that adjusts the vibration and airflow effects based on the user's emotional state."

[0825] Video data: (video stream)

[0826] Audio data: (audio stream)

[0827] User emotion data: (facial expression and voice analysis data)

[0828] output:

[0829] Scene Events: Important events with time stamps

[0830] Effect data: appropriate vibration and airflow effects for each event (intensity, duration, pattern, wind force, direction)

[0831] Adjustment data: Adjustment data based on the user's emotional state

[0832] In this way, it is possible to provide a sensory effect that corresponds to the user's emotions, and to realize a home theater experience that is optimized for each individual user.

[0833] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0834] Step 1:

[0835] When a user plays a movie, the device transmits the movie's video and audio data to the server. This allows the server to receive the movie stream in real time. The input data are video and audio streams. Specifically, the video and audio captured by a smartphone or smart glasses are transferred to the server via the Internet.

[0836] Step 2:

[0837] The server inputs the video data into a generative AI model (such as GPT-4) to recognize important events. The video data is analyzed frame by frame to identify scenes such as explosions and falls. The generative AI model then timestamps these important events and creates an event list. The input data is the video data, and the output data is a list of important events with timestamps.

[0838] Step 3:

[0839] The server performs spectral analysis of the audio data to recognize important sound fields. For example, it identifies specific sound fields, such as roars or screams, by detecting sound peaks and specific frequency components. This is also done using a generative AI model. The input data is the audio data, and the output data is a time-stamped list of important sound fields.

[0840] Step 4:

[0841] The server generates effect data based on the important event list and the important sound field list. For example, it generates strong vibration effect data for an explosion scene, and instantaneous strong wind effect data for thunder. The input data are the important event list and the important sound field list, and the output data is the effect data. Specifically, detailed information such as the intensity, duration, pattern, wind force, direction, and wind duration of the vibrations is included.

[0842] Step 5:

[0843] The server transmits the generated effect data to the terminal in real time. The terminal stores the received effect data and keeps it in a standby state for execution. The input data is the effect data, and the output data is the effect data transmitted to the terminal. Specifically, the effect data is transmitted via wireless communication or the Internet.

[0844] Step 6:

[0845] The emotion engine analyzes the user's facial expression, voice tone, and biometric data to recognize the user's emotional state. For example, it detects whether the user's eyes are wide open in surprise or their voice is trembling in fear. The input data is the user's facial expression data, voice data, and biometric data, and the output data is the user's emotional state data.

[0846] Step 7:

[0847] The server dynamically adjusts the effect data based on the emotion engine data. For example, if the user is very nervous, it will reduce the vibration intensity and shorten the duration of the airflow. The input data is the emotional state data and existing effect data, and the output data is the adjusted effect data.

[0848] Step 8:

[0849] The terminal synchronizes with the playback timing of the movie and controls the vibration device and the air blowing device according to the adjusted effect data. This allows the user to experience appropriate bodily sensations according to the scene in the movie. The input data is the adjusted effect data, and the output data is the execution of physical vibrations and air blowing. Specifically, the terminal transmits control signals to operate the vibration device and the air blowing device based on the effect data.

[0850] In this way, the present invention provides a user with a personalized, immersive movie-watching experience.

[0851] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0852] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0853] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0854] [Third embodiment]

[0855] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0856] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0857] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0858] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0859] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0860] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0861] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0862] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0863] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0864] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0865] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0866] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0867] ---

[0868] The present invention is an immersive home theater system that utilizes AI. Specific embodiments of the system are described below, and the system's program processing is explained in natural language.

[0869] System Configuration

[0870] The system mainly includes the following components:

[0871] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[0872] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[0873] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[0874] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[0875] 5. User: A person who enjoys immersive movie watching through this system.

[0876] Program processing

[0877] Data analysis by server

[0878] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[0879] Generating efficacy data

[0880] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[0881] Sending and executing effect data

[0882] The server sends the generated effect data to the terminal in real time, and the terminal keeps the received effect data in a standby state and executes the appropriate effect in synchronization with the movie playback timing.

[0883] The device controls the vibration device and the air blowing device. For example, when a movie is about to play and an explosion scene is about to occur, the device instructs the vibration device to generate a "strong vibration" and the air blowing device to generate a "strong wind" moment. This allows the user to experience the movie scene not only visually but also through touch and the wind.

[0884] Specific examples

[0885] Processing thunder scenes

[0886] The user starts playing a movie. If there is thunder in one scene, the following happens:

[0887] 1. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[0888] 2. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[0889] 3. The server sends the effect data to the terminal.

[0890] 4. The terminal synchronizes the timing during movie playback to instruct the vibration device to vibrate strongly and the air blowing device to generate a momentary strong wind.

[0891] 5. While watching this scene, the user will experience the thunder scene in a realistic way, as the sofa or chair vibrates strongly and they will feel a sudden gust of wind.

[0892] Accommodating visually and hearing impaired users

[0893] This system is also suitable for users with visual or hearing impairments. By providing a haptic effect that is independent of video or audio, users can enjoy movie scenes and action through vibrations and wind. This allows all users to be equally immersed in the world of the movie.

[0894] As a result, the present invention provides users with a cinema-like movie-watching experience at home, making the overall movie-watching experience more interactive and immersive.

[0895] ---

[0896] The processing flow will be explained below.

[0897] ---

[0898] Step 1: Enter movie data

[0899] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[0900] Step 2: Analyzing the video data

[0901] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[0902] Step 3: Analyzing the audio data

[0903] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[0904] Step 4: Generate efficacy data

[0905] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[0906] Step 5: Submitting your effectiveness data

[0907] The server transmits the generated effect data to the device in real time, including the vibration intensity, duration, pattern, wind power, direction, and wind duration.

[0908] Step 6: Receiving and preparing effectiveness data

[0909] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[0910] Step 7: Synchronize movie playback

[0911] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[0912] Step 8: Implementing the vibration effect

[0913] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration for an explosion scene, a medium vibration for a fall scene, etc.

[0914] Step 9: Implementing the blowing effect

[0915] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[0916] Step 10: User Experience

[0917] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[0918] ---

[0919] Example 1

[0920] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0921] Conventional home theater systems can only provide a limited visual and auditory experience, making it difficult to fully experience the reality and immersion of a movie. Furthermore, users with visual or hearing impairments have difficulty fully enjoying the content of a movie. Therefore, a means was needed to enable all users to enjoy watching movies equally.

[0922] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0923] In this invention, the server includes: means for analyzing video data and identifying important events; means for analyzing audio data and identifying important sound fields; means for generating physical effect data based on the analyzed video data and audio data; means for transmitting the generated physical effect data to a control device; means for controlling a vibration device and an air blower based on the physical effect data received by the control device; and means for simultaneously analyzing the video data and audio data and generating time-stamped effect data for a specific scene. This allows users to experience movie scenes not only visually but also through touch and wind, significantly improving the sense of immersion in the movie. Furthermore, all users, including those with visual or hearing impairments, can enjoy a realistic movie-watching experience.

[0924] "Video data" refers to electronic data containing visual information such as movies, videos, and animations.

[0925] "Audio data" refers to electronic data containing auditory information such as movies, videos, and music.

[0926] Analysis is the process of examining and breaking down data to identify meaning and patterns.

[0927] A "significant event" is a scene or action in a film or video that is particularly notable visually or audibly.

[0928] "Important sound fields" refer to areas in the audio data where specific frequencies or sound pressure levels are prominent.

[0929] "Physical effects data" refers to data for instructing physical effects such as vibrations and wind that correspond to specific scenes in a movie or video.

[0930] The "control device" is a device that operates the vibration device and the air blower based on the received physical effect data.

[0931] A "vibrator" is a device that delivers physical vibrations to a user at a specified intensity and pattern.

[0932] A "blower" is a device that delivers physical wind to a user at a specified wind force and direction.

[0933] A "timestamp" indicates the date and time when a particular data or event occurred.

[0934] A "generative AI model" refers to a trained model that uses artificial intelligence to analyze data and generate effectiveness data.

[0935] This invention is an immersive home theater system that utilizes AI technology. This system is designed to allow users to experience realistic vibrations and wind while watching a movie. Specifically, the system analyzes video and audio data and controls vibration and airflow devices to generate physical effects according to the movie scene.

[0936] System Configuration

[0937] This system consists of the following hardware and software:

[0938] 1. Server: Analyzes video and audio data and generates physical effect data.

[0939] 2. Terminal: Receives effect data sent from the server and controls the vibration device and the air blower.

[0940] 3. Vibration device: Attached to the sofa or chair on which the user sits, it generates specified vibrations.

[0941] 4. Blower: Placed around the user and generates a specified amount of wind.

[0942] Program processing

[0943] Data analysis by server

[0944] The server receives the movie video and audio data sent by the user. The server is equipped with a generative AI model (e.g., a general-purpose AI model) that analyzes the video data frame by frame. For example, the generative AI model detects sudden changes in brightness and specific movements to recognize explosions and action scenes. It also performs spectral analysis of the audio data to identify specific sound fields (e.g., explosions, thunder). The server generates time-stamped physical effect data based on the results of these analyses.

[0945] Generating efficacy data

[0946] Based on the analysis results, the server generates vibration and wind effect data corresponding to a specific scene. For example, it generates strong vibrations for an explosion scene and instantaneous strong winds for a thunder scene. The generated effect data includes the strength, duration, and pattern of vibrations, as well as the wind power, direction, and duration.

[0947] Sending and executing effect data

[0948] The server transmits the generated physical effect data to the device in real time. The device receives this data and controls the vibration device and the air blower in accordance with the timing of the movie playback. For example, when the movie playback reaches an explosion scene, the device instructs the vibration device to generate a vibration of a specified strength and the air blower to generate a specified strong wind.

[0949] Specific examples

[0950] Processing thunder scenes

[0951] The user starts playing a movie. The server detects flashes of lightning from the video data and analyzes the peaks of thunder from the audio data. It then generates data on the effects of strong vibrations and instantaneous wind and sends it to the device. The device then instructs the vibration device to generate strong vibrations and the air blower to generate instantaneous strong winds, synchronizing with the movie playback. The user experiences this scene, greatly enhancing the realism of the movie.

[0952] Prompt Sentence Examples

[0953] An example of a prompt to input to a generative AI model is as follows:

[0954] Generate effects data to control vibration and airflow devices in a thunder scene in a movie. The effects data should include the following information:

[0955] 1. Vibration strength (strong, medium, weak)

[0956] 2. Duration (seconds)

[0957] 3. Vibration pattern (continuous, intermittent)

[0958] 4. Wind power (strong, medium, weak)

[0959] 5. Wind direction (front, back, left, right)

[0960] 6. Wind duration (seconds)

[0961] As a result, by using this invention, users can enjoy a more realistic and immersive movie viewing experience. The system provides tactile and wind experiences in addition to visual and auditory experiences, allowing all users to enjoy movies with a highly immersive feeling.

[0962] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0963] Step 1: (User) Play the movie

[0964] A user plays a movie on a device that contains the movie's video and audio data and is configured to stream it to a server in real time. The input is the movie's video and audio data, and the output is the data sent to the server.

[0965] Step 2: (Server) Receiving Data

[0966] The server receives the video and audio data of the movie sent by the user. The input is the video and audio data of the movie, and the output is that this data is stored in the server and prepared for analysis.

[0967] Step 3: (Server) Video data analysis

[0968] A generative AI model (e.g., a general-purpose AI model) in the server analyzes the received video data frame by frame. Specifically, the generative AI model detects sudden changes in brightness and specific movements, and identifies important events such as explosions or action scenes. The input is the video data, and the output is a list of identified events.

[0969] Step 4: (Server) Audio data analysis

[0970] The server analyzes the audio data received in synchronization with the video data. It performs a spectral analysis of the audio data to detect peaks and frequencies of specific sound fields (e.g., explosions, thunder). Specifically, it analyzes specific frequency peaks and sound pressure levels within the audio data to identify important sound fields. The input is the audio data, and the output is a list of identified sound fields.

[0971] Step 5: (Server) Generate effect data

[0972] The server generates physical effect data based on the analysis results. Specifically, it generates vibration effect data and wind effect data corresponding to a specific scene based on the analysis results of the video data and audio data. For example, it generates "strong vibration" for an explosion scene and "instantaneous strong wind" for a thunder scene. The input is a list of identified events and sound fields, and the generated physical effect data is obtained as output.

[0973] Step 6: (Server) Sending effect data

[0974] The server sends the generated physical effect data to the device in real time. Specifically, it assigns a timestamp to the effect data and sends it to the device at the appropriate time. The input is the generated physical effect data, and the output is the physical effect data with the timestamp sent to the device.

[0975] Step 7: (Terminal) Receive effect data and wait for execution

[0976] The device holds the physical effect data received from the server in a standby state. Specifically, the received data is stored in memory and placed in a standby state as appropriate in accordance with the timing of movie playback. The input is the time-stamped physical effect data sent from the server, and the output is the data in a standby state.

[0977] Step 8: (Terminal) Control of vibration and blower devices

[0978] The device executes the received effect data in synchronization with the movie scenes. Specifically, it issues instructions to a vibration device and a blower to generate physical effects in accordance with the movie playback. For example, when the movie playback reaches an explosion scene, it issues an instruction to a vibration device to generate a "strong vibration" and an instruction to a blower to generate a "momentary strong wind." The input is the physical effect data in a standby state, and the output is the executed physical effect.

[0979] Step 9: (User) Enjoyment of the experience

[0980] The user experiences vibrations and wind that occur in real time in response to movie scenes. For example, during a thunder scene, the sofa vibrates strongly, and at the same time, a momentary breeze is felt, bringing the scene to life. The input is the physical effect that is being performed, and the output is a realistic movie experience that is felt by the user.

[0981] (Application example 1)

[0982] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0983] Conventional immersive systems in home theaters and entertainment facilities rely on visual and auditory perception, which can prevent users from fully experiencing the virtual reality world. This also makes it difficult for visually or auditory impaired users to enjoy the content. Furthermore, physical feedback during events and important scenes is incomplete, creating a need for improved realism and immersion.

[0984] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0985] In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for providing physical feedback to the user using a virtual reality device based on the effect data, thereby enabling the user to have a more realistic and immersive experience not only through video and audio but also through tactile sensations and wind.

[0986] "Video data" refers to data containing visual information captured by a video camera or other device.

[0987] "Analysis" is the process of extracting meaningful information from data and understanding and recognizing it.

[0988] A "significant event" is something that is of particular interest to the user, causing a particular action or change in the video or audio.

[0989] "Audio data" refers to data that includes auditory information collected by a microphone or the like.

[0990] An "important sound field" is an acoustic phenomenon that occurs at a particular time or place and is particularly noticeable to the user.

[0991] "Effect data" is information for controlling physical devices that is generated as a result of analyzing video data and audio data.

[0992] The "control device" is a device that receives the transmitted effect data and operates a physical device such as a vibration device or a ventilation device based on the data.

[0993] A "vibration device" is a device that generates physical vibrations in response to input instructions.

[0994] A "blowing device" is a device that generates air based on input instructions.

[0995] A "virtual reality device" is a device that allows a user to visually and aurally experience a virtual environment or situation, and typically includes a head-mounted display and sound system.

[0996] "Physical feedback" refers to providing the user with physical sensations such as vibrations or wind in response to changes in video and audio.

[0997] This invention describes an immersive VR ride attraction system that applies an immersive home theater system using AI. Specific embodiments and system program processing are described below.

[0998] System Configuration

[0999] The system mainly includes the following components:

[1000] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[1001] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. It works in conjunction with the virtual reality equipment.

[1002] 3. Vibration device: A device that generates vibrations in the seat on which the user sits or in the equipment worn by the user.

[1003] 4. Blowing device: A device that makes the user feel the wind at a specific force and direction.

[1004] 5. Virtual reality equipment: A device that allows the user to experience a virtual environment visually and audibly.

[1005] 6. User: A user who enjoys an experiential VR ride attraction through this system.

[1006] Data analysis by server

[1007] The server uses Python to analyze video and audio data. It uses a generative AI model to recognize important events in each frame of the video data. It uses the SciPy library to perform spectral analysis on the audio data to identify important sound fields. Specifically, it detects data corresponding to specific events, such as explosions, falls, or thunder. Based on the information obtained from this analysis, it generates effect data to control vibration and airflow devices.

[1008] Generating efficacy data

[1009] The server generates vibration and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. Based on the audio data, it generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[1010] Sending and executing effect data

[1011] The server sends the generated effect data to the device in real time. The device keeps the received effect data in a standby state and synchronizes with the virtual reality device to execute the effect at the appropriate time. The device controls the vibration device and air blowing device using a control program in Unity or Arduino. This allows the user to experience the movie scenes in a very realistic way, not only through their eyes but also through their sense of touch and wind.

[1012] Specific examples

[1013] A specific example will be described below, where thunder rumbles in a certain scene.

[1014] 1. The user starts the attraction and puts on the VR headset.

[1015] 2. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[1016] 3. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[1017] 4. The server sends the effect data to the terminal.

[1018] 5. The terminal instructs the vibration device to vibrate strongly and the air blowing device to blow a momentary strong wind in time with the playback timing of the VR attraction.

[1019] 6. While experiencing this scene, the user feels the seat and the wearable device vibrate strongly, and at the same time, they feel a momentary gust of wind, allowing them to experience the thunder scene in a realistic way.

[1020] Prompt Sentence Examples

[1021] "I want to design a system that will create strong vibrations and wind effects when an explosion occurs during a VR experience. To achieve this, I will create an AI program that will detect important events and generate the corresponding vibration and wind effects."

[1022] As a result, the present invention provides users with a more immersive and realistic experience in virtual reality, making the overall entertainment experience more interactive and immersive.

[1023] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1024] Step 1:

[1025] The server receives video data. The input is video data obtained from a video camera or other capture device. The server uses a generative AI model to analyze each frame of the video and recognize important events (e.g., explosions, falls). The output is a list of important events detected in the video.

[1026] Step 2:

[1027] The server receives audio data. The input is audio data collected from a microphone or other device. The server uses the SciPy library to perform spectral analysis of the audio data and identify important sound fields (e.g., thunder, explosions). The output is a list of important sound fields recognized in the audio.

[1028] Step 3:

[1029] The server generates effect data based on the analysis results of the video and audio data obtained in steps 1 and 2. The inputs are a list of important events and a list of sound fields. The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. The output is a set of corresponding effect data.

[1030] Step 4:

[1031] The server sends the generated effect data to the terminal. The input is a set of effect data. The effect data is sent from the server in real time. The output is the terminal that receives the effect data.

[1032] Step 5:

[1033] The terminal keeps the received effect data in a standby state. The input is the effect data sent from the server. The terminal synchronizes with the playback timing of the movie or VR attraction and prepares to execute the appropriate effect. The output is in a ready state.

[1034] Step 6:

[1035] The terminal controls the vibration device and the air blowing device in accordance with the playback timing. The input is the effect data and playback timing. The terminal uses a control program in Unity or Arduino to operate the physical devices with the specified intensity and duration. Specific actions include the seat or wearable device vibrating and the air blowing device generating wind. The output is the vibration and wind experienced by the user.

[1036] Through these steps, users can experience realistic changes in images and sounds not only through their eyes but also through their sense of touch and wind, making it possible to provide a truly immersive entertainment experience in the virtual reality world.

[1037] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1038] ---

[1039] This invention combines an AI-based immersive home theater system with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are shown below, and the system's program processing is explained in natural language.

[1040] System Configuration

[1041] The system mainly includes the following components:

[1042] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[1043] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[1044] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[1045] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[1046] 5. Emotion engine: A device that recognizes a user's emotions by analyzing their facial expressions, voice tone, or biometric data.

[1047] 6. User: A person who enjoys immersive movie watching through this system.

[1048] Program processing

[1049] Data analysis by server

[1050] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[1051] Efficacy data generation and adjustment

[1052] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[1053] Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, or biometric data to recognize the user's current emotion. For example, if the user is feeling scared, the emotion engine will detect this and adjust the effects of the vibration and airflow devices.

[1054] Sending and executing effect data

[1055] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing. It is also possible to dynamically adjust the effect data in real time based on data from the emotion engine.

[1056] For example, if the user is very tense, the vibration effect can be reduced and the airflow can be gentler to soothe the experience, while conversely, if the user is relaxed, the effect can be intensified.

[1057] Specific examples

[1058] Processing scary scenes from horror movies

[1059] The user plays a horror movie and reaches a scary scene. The following happens:

[1060] 1. The server detects sudden changes in brightness from the video data and frightening sounds from the audio data, such as a shadow appearing in the dark or a sudden scream.

[1061] 2. The server generates the appropriate vibration and cold wind effect data for the scary scene.

[1062] 3. The emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize that the user is already feeling very scared.

[1063] 4. The server takes into account the data from the emotion engine and reduces the vibration intensity and gently adjusts the fan effect.

[1064] 5. The device will then execute coordinated vibration and airflow effects when the movie reaches a scary scene.

[1065] 6. By experiencing a moderately adjusted immersive effect, users can enjoy a realistic immersive movie experience while mitigating excessive fear.

[1066] Accommodating visually and hearing impaired users

[1067] This system is also suitable for users with visual or hearing impairments. By providing immersive effects that are independent of video and audio, users can enjoy movie scenes and action through vibrations and wind. In addition, the emotion engine understands the user's emotional state and adjusts the appropriate effects, allowing users to enjoy movies comfortably.

[1068] As a result, the present invention provides users with a cinema-like experience of watching movies at home, and by utilizing an emotion engine to create an experience optimized for each user, the overall movie-watching experience becomes more interactive and personalized.

[1069] The processing flow will be explained below.

[1070] ---

[1071] Step 1: Enter movie data

[1072] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[1073] Step 2: Analyzing the video data

[1074] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[1075] Step 3: Analyzing the audio data

[1076] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[1077] Step 4: Generate efficacy data

[1078] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[1079] Step 5: Obtaining emotion data

[1080] The emotion engine monitors the user's facial expressions, voice tone, or biometric data to analyze the user's emotions in real time. For example, the emotion engine can determine whether the user is scared or excited.

[1081] Step 6: Adjusting the effect data

[1082] The server dynamically adjusts the effect data based on the user's emotional data acquired by the emotion engine. For example, if the user is already feeling scared, the server reduces the intensity of the vibrations and softens the airflow.

[1083] Step 7: Submitting your effectiveness data

[1084] The server transmits the generated and adjusted effect data to the device in real time, including vibration intensity, duration, pattern, wind power, direction, and wind duration.

[1085] Step 8: Receiving and preparing effectiveness data

[1086] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[1087] Step 9: Synchronize movie playback

[1088] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[1089] Step 10: Implementing the vibration effect

[1090] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration during an explosion scene, a medium vibration during a fall scene, etc.

[1091] Step 11: Implementing the blowing effect

[1092] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[1093] Step 12: User Experience

[1094] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[1095] ---

[1096] The above is the specific processing flow for an invention that combines an emotion engine with an immersive home theater system that utilizes AI.

[1097] Example 2

[1098] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1099] Conventional immersive home theater systems generate effect data based only on video and audio data, without considering the user's emotional state. This results in a problem where a uniform immersive effect is provided regardless of the user's emotional state. In particular, if the user is feeling fear or tension, the experience may become overly stressful due to the lack of appropriate adjustments.

[1100] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for analyzing user emotion data and adjusting the effect data based on the emotion data, means for transmitting the generated effect data to the control device, and means for controlling the vibration device and the air blowing device based on the effect data received by the control device. This makes it possible to dynamically provide optimal bodily sensation effects according to the user's emotional state.

[1101] "Video data" is digital data containing visual information such as movies and videos.

[1102] "Audio data" is digital data containing the sound information of a film or video.

[1103] A "significant event" refers to a specific occurrence in a movie or video scene that has a strong visual and emotional impact on the user.

[1104] "Critical sound fields" refer to specific acoustic environments that have a strong auditory and emotional impact on the user in film and video audio.

[1105] "Effect data" is instruction data for realizing physical effects such as vibrations and airflow that are generated based on the analyzed video data and audio data.

[1106] The "control device" is a device that receives effect data sent from the server and operates the vibration device and the air blowing device based on the effect data.

[1107] A "vibration device" is a device that generates physical vibrations based on generated effect data.

[1108] The "air blowing device" is a device that generates air based on the generated effect data.

[1109] "User's emotional data" is data that indicates the user's current emotional state, analyzed from the user's facial expression, voice tone, biometric information, and the like.

[1110] The "means for adjusting effect data based on emotional data" refers to a method for dynamically changing the intensity and duration of vibrations and airflow, etc., in accordance with the emotional state of the user.

[1111] The present invention is an immersive home theater system that uses AI technology to recognize a user's emotions and adjust the immersive effects according to those emotions, providing a more personalized movie-watching experience. Specific embodiments of the present invention are described in detail below.

[1112] System Configuration

[1113] The system includes the following components:

[1114] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[1115] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[1116] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[1117] 4. Air Blowing Device: A device that blows air at a specified force and direction to the user.

[1118] 5. Emotion Engine: A device that recognizes the user's emotions by analyzing their facial expressions, voice tone, and biometric data.

[1119] 6. User: A person who enjoys immersive movie viewing through this system.

[1120] Program processing

[1121] Data analysis by server

[1122] The server receives the video and audio data of the movie sent by the user. It analyzes this data using generative AI models such as TensorFlow and PyTorch. The server recognizes important events in each frame of the video data, and performs spectral analysis of the audio data using libraries such as LibROSA to recognize important sound fields.

[1123] As a specific example of processing, the server detects sudden changes in brightness from video data and specific acoustic patterns (e.g., explosions or thunder) from audio data. Based on these analysis results, effect data is generated.

[1124] Efficacy data generation and adjustment

[1125] The server generates vibration and wind effect data based on the analysis results. For example, it uses SciPy and NumPy to process and calculate the data. For example, it generates "strong vibration" (3 seconds) for an explosion scene, and "instantaneous strong wind" (1 second) for thunder.

[1126] Furthermore, the emotion engine uses OpenCV and DeepFace to analyze the user's facial expressions, voice tone, and biometric data to detect the user's current emotional state. If the user is feeling fear or tension, the emotion engine will collect that information and adjust the effect data.

[1127] For example, if the user is very nervous, the emotion engine will adjust the vibration intensity to 50% and the airflow strength to 20%.

[1128] Sending and executing effect data

[1129] The server sends the generated effect data to the device in real time using a low-latency protocol such as WebSocket. The device keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing.

[1130] As a concrete example, consider the case where a user plays a scary scene from a horror movie. The server adjusts the effect data based on the scene analysis results and emotion engine data, and sends it to the device. The device then executes the adjusted vibration and airflow effects in accordance with the timing of the movie playback. This allows the user to enjoy an appropriately adjusted sensory effect without feeling excessive fear.

[1131] Specific operation example

[1132] Prompt Sentence Examples

[1133] As a specific example of operation, the following prompt sentence is input into the generative AI model.

[1134] 1. Video data analysis:

[1135] "Analyze video data and extract important events using generative AI models that identify explosions and action scenes."

[1136] 2. Analysis of audio data:

[1137] "Use spectral analysis to analyze audio data and recognize important sound fields such as explosions and thunder."

[1138] 3. Emotion Engine Analysis:

[1139] "Detect the user's current emotional state by analyzing their facial expressions, voice tone, and biometric data."

[1140] In this way, the present invention provides users with an optimal immersive movie-watching experience in real time, and utilizes an emotion engine to dynamically adjust effects according to the user's individual emotional state, making the movie-watching experience more interactive and personalized.

[1141] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1142] Processing Steps

[1143] Step 1:

[1144] Receiving movie data

[1145] The server receives the video and audio data of the movie from the user.

[1146] When the user presses the play button on the movie, the device transmits this data to the server in real time.

[1147] Input: Video data, audio data

[1148] Output: Received video and audio data

[1149] Specific behavior:

[1150] When a user presses the play button on a movie, the device uses the HTTP protocol to send video and audio data to the server, which receives it and prepares it for analysis.

[1151] Step 2:

[1152] Video data analysis

[1153] The server analyzes the received video data.

[1154] Uses a generative AI model (e.g., TensorFlow) to recognize important events in each frame.

[1155] Input: Video data

[1156] Output: List of significant events (e.g. explosions, action scenes)

[1157] Specific behavior:

[1158] The server analyzes each frame of video data, detecting specific patterns or changes (such as sudden changes in brightness), and uses a generative AI model to write something like "Frame 1500-2000: Explosion detected."

[1159] Step 3:

[1160] Analysis of audio data

[1161] The server analyzes the received voice data.

[1162] Spectral analysis is performed using libraries such as "LibROSA" to recognize important sound fields.

[1163] Input: Audio data

[1164] Output: List of important sound fields (e.g. explosions, thunder)

[1165] Specific behavior:

[1166] The server performs spectral analysis on the audio data to detect specific acoustic patterns (e.g. explosions, thunder), and lists information such as "Timestamp 00:10:30 - Thunderstorm sound detected."

[1167] Step 4:

[1168] Generating efficacy data

[1169] The server generates effect data based on the analyzed video data and audio data.

[1170] Calculate the vibration and ventilation effect data using "SciPy" and "NumPy".

[1171] Input: List of important events, list of important sound fields

[1172] Output: Effect data (e.g. strong vibration, momentary strong wind)

[1173] Specific behavior:

[1174] The server generates effect data such as "strong vibration" for explosion scenes and "instant strong wind" for thunder. For example, create "Effect 1: Strong vibration for 3 seconds" and "Effect 2: Quick strong wind for 1 second."

[1175] Step 5:

[1176] Emotional data analysis and effect adjustment

[1177] The emotion engine analyzes the user's facial expressions, voice tone, and biometric data.

[1178] "OpenCV" and "DeepFace" are used to recognize the user's emotions and adjust the effect data.

[1179] Input: User biometric data, emotional data

[1180] Output: Adjusted effect data

[1181] Specific behavior:

[1182] While the user is watching a movie, the emotion engine analyzes the user's face and obtains data such as "User appears scared." Based on this, the server makes adjustments such as "Reduce vibration to 50%" and "Lower wind intensity to 20%."

[1183] Step 6:

[1184] Sending effect data

[1185] The server transmits the generated effect data to the terminal.

[1186] Use low latency protocols (e.g. "WebSocket").

[1187] Input: Efficacy data

[1188] Output: Sends effect data to the terminal.

[1189] Specific behavior:

[1190] The server sends data to the device such as "Send Effect 1: Strong vibration for 3 seconds" and "Send Effect 2: Quick strong wind for 1 second."

[1191] Step 7:

[1192] Execution of effectiveness data

[1193] The terminal executes the received effect data in synchronization with the playback timing of the movie.

[1194] Controls vibration and air blowing devices.

[1195] Input: Efficacy data

[1196] Output: Execute effect device

[1197] Specific behavior:

[1198] Based on the effect data received by the device, it sends commands such as "execute strong vibration for 3 seconds" or "execute a momentary strong wind for 1 second" to the vibration device or air blowing device, allowing the user to experience the sensory effects in real time in line with the movie.

[1199] (Application example 2)

[1200] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1201] Conventional immersive home theater systems provide vibration and airflow effects based on video and audio data, but lack the technology to adjust these effects based on the user's emotional state. As a result, excessive vibration and airflow can make users uncomfortable, making it difficult to provide an optimal experience tailored to each individual user's emotions. The present invention aims to solve this problem and provide an immersive movie-watching experience optimized for each user.

[1202] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for analyzing the user's emotional state and adjusting the effect data based on the results. This makes it possible to provide a bodily sensation effect that is appropriately adjusted according to the user's emotions.

[1203] "Video data" is digital data that contains visual information such as movies and videos.

[1204] "Audio data" is digital data containing audio information contained in movies and videos.

[1205] "Important events" are information that refer to scenes or actions in a movie or video that deserve particular attention.

[1206] "Important sound fields" refers to information that indicates scenes in movies or videos that have particularly impressive sound effects.

[1207] "Effect data" is information relating to bodily sensation effects such as vibration and airflow that are generated based on video data and audio data.

[1208] The "control device" is a device that operates the vibration device and the air blowing device based on the generated effect data.

[1209] A "vibration device" is a device that generates physical vibrations on a sofa, chair, or other device on which a user sits.

[1210] A "blowing device" is a device that allows a user to feel wind with a specified wind force and direction.

[1211] "User's emotional state" refers to the psychological state obtained by analyzing the user's facial expressions, voice, and biometric data.

[1212] "Analyzing" is the process of examining data in detail to reveal its components and patterns.

[1213] "Adjusting" is a means of making appropriate corrections or amendments based on the data and results obtained.

[1214] System configuration and hardware and software usage

[1215] The present invention is implemented by a system that mainly includes the following components, all of which are used to generate sensory effects tailored to a user's emotions and enhance the movie-watching experience:

[1216] 1. Server: A system that analyzes video and audio data and recognizes important events and sound fields. The server performs data analysis using a generative AI model (e.g., OpenAI's GPT-4).

[1217] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. Examples of this include smartphones and smart glasses.

[1218] 3. Vibration device: A device that is attached to a sofa or chair and generates physical vibrations.

[1219] 4. Blower device: A small blower that allows the user to feel the wind.

[1220] 5. Emotion engine: A device that can recognize a user's emotional state by analyzing their facial expressions, voice tone, or biometric data. Examples include Microsoft Azure Cognitive Services and IBM Watson.

[1221] Program processing and data processing

[1222] Server-based data analysis and generation of AI models

[1223] The server receives the video and audio data of the movie sent by the user. It then analyzes this data using a generative AI model to recognize important events and sound fields. For example, it identifies sudden scene changes and explosions. Based on the analysis results, the server generates vibration and airflow effect data. Generative AI models such as OpenAI's GPT-4 are used.

[1224] Generating effect data and analyzing the user's emotional state

[1225] The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize the user's emotional state. For example, if the user is feeling fear, the emotion engine can detect this and the server can dynamically adjust the effect data.

[1226] Sending and executing effect data

[1227] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and controls the vibration device and air blower device in sync with the movie playback timing. The effect data can be adjusted in real time based on the emotion engine data.

[1228] Specific examples

[1229] For example, if a user plays a horror movie and reaches a scary scene, the process is as follows:

[1230] 1. The user plays a movie.

[1231] 2. The server analyzes the video and audio data to detect shadows appearing in the darkness or sudden screams.

[1232] 3. The server generates vibration and cold wind effect data suitable for the horror scene.

[1233] 4. The emotion engine analyzes the user's facial expressions and biometric data and recognizes that they are already feeling extremely scared.

[1234] 5. The server takes these data into account and reduces the vibration intensity and gently adjusts the airflow effect.

[1235] 6. The device will execute vibration and airflow effects according to the adjusted effect data, timed to coincide with the scary scenes in the movie.

[1236] Example prompts to input to the generative AI model

[1237] The following prompt sentence can be used to have the system analyze the important events in the movie:

[1238] Prompt statement:

[1239] "Analyze the following video and audio data and recognize important events (e.g. explosions, sudden scene changes). Then generate an algorithm that adjusts the vibration and airflow effects based on the user's emotional state."

[1240] Video data: (video stream)

[1241] Audio data: (audio stream)

[1242] User emotion data: (facial expression and voice analysis data)

[1243] output:

[1244] Scene Events: Important events with time stamps

[1245] Effect data: appropriate vibration and airflow effects for each event (intensity, duration, pattern, wind force, direction)

[1246] Adjustment data: Adjustment data based on the user's emotional state

[1247] In this way, it is possible to provide a sensory effect that corresponds to the user's emotions, and to realize a home theater experience that is optimized for each individual user.

[1248] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1249] Step 1:

[1250] When a user plays a movie, the device transmits the movie's video and audio data to the server. This allows the server to receive the movie stream in real time. The input data are video and audio streams. Specifically, the video and audio captured by a smartphone or smart glasses are transferred to the server via the Internet.

[1251] Step 2:

[1252] The server inputs the video data into a generative AI model (such as GPT-4) to recognize important events. The video data is analyzed frame by frame to identify scenes such as explosions and falls. The generative AI model then timestamps these important events and creates an event list. The input data is the video data, and the output data is a list of important events with timestamps.

[1253] Step 3:

[1254] The server performs spectral analysis of the audio data to recognize important sound fields. For example, it identifies specific sound fields, such as roars or screams, by detecting sound peaks and specific frequency components. This is also done using a generative AI model. The input data is the audio data, and the output data is a time-stamped list of important sound fields.

[1255] Step 4:

[1256] The server generates effect data based on the important event list and the important sound field list. For example, it generates strong vibration effect data for an explosion scene, and instantaneous strong wind effect data for thunder. The input data are the important event list and the important sound field list, and the output data is the effect data. Specifically, detailed information such as the intensity, duration, pattern, wind force, direction, and wind duration of the vibrations is included.

[1257] Step 5:

[1258] The server transmits the generated effect data to the terminal in real time. The terminal stores the received effect data and keeps it in a standby state for execution. The input data is the effect data, and the output data is the effect data transmitted to the terminal. Specifically, the effect data is transmitted via wireless communication or the Internet.

[1259] Step 6:

[1260] The emotion engine analyzes the user's facial expression, voice tone, and biometric data to recognize the user's emotional state. For example, it detects whether the user's eyes are wide open in surprise or their voice is trembling in fear. The input data is the user's facial expression data, voice data, and biometric data, and the output data is the user's emotional state data.

[1261] Step 7:

[1262] The server dynamically adjusts the effect data based on the emotion engine data. For example, if the user is very nervous, it will reduce the vibration intensity and shorten the duration of the airflow. The input data is the emotional state data and existing effect data, and the output data is the adjusted effect data.

[1263] Step 8:

[1264] The terminal synchronizes with the playback timing of the movie and controls the vibration device and the air blowing device according to the adjusted effect data. This allows the user to experience appropriate bodily sensations according to the scene in the movie. The input data is the adjusted effect data, and the output data is the execution of physical vibrations and air blowing. Specifically, the terminal transmits control signals to operate the vibration device and the air blowing device based on the effect data.

[1265] In this way, the present invention provides a user with a personalized, immersive movie-watching experience.

[1266] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1267] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1268] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1269] [Fourth embodiment]

[1270] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1271] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1272] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1273] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1274] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1275] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1276] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1277] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1278] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1279] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1280] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1281] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1282] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1283] ---

[1284] The present invention is an immersive home theater system that utilizes AI. Specific embodiments of the system are described below, and the system's program processing is explained in natural language.

[1285] System Configuration

[1286] The system mainly includes the following components:

[1287] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[1288] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[1289] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[1290] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[1291] 5. User: A person who enjoys immersive movie watching through this system.

[1292] Program processing

[1293] Data analysis by server

[1294] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[1295] Generating efficacy data

[1296] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[1297] Sending and executing effect data

[1298] The server sends the generated effect data to the terminal in real time, and the terminal keeps the received effect data in a standby state and executes the appropriate effect in synchronization with the movie playback timing.

[1299] The device controls the vibration device and the air blowing device. For example, when a movie is about to play and an explosion scene is about to occur, the device instructs the vibration device to generate a "strong vibration" and the air blowing device to generate a "strong wind" moment. This allows the user to experience the movie scene not only visually but also through touch and the wind.

[1300] Specific examples

[1301] Processing thunder scenes

[1302] The user starts playing a movie. If there is thunder in one scene, the following happens:

[1303] 1. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[1304] 2. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[1305] 3. The server sends the effect data to the terminal.

[1306] 4. The terminal synchronizes the timing during movie playback to instruct the vibration device to vibrate strongly and the air blowing device to generate a momentary strong wind.

[1307] 5. While watching this scene, the user will experience the thunder scene in a realistic way, as the sofa or chair vibrates strongly and they will feel a sudden gust of wind.

[1308] Accommodating visually and hearing impaired users

[1309] This system is also suitable for users with visual or hearing impairments. By providing a haptic effect that is independent of video or audio, users can enjoy movie scenes and action through vibrations and wind. This allows all users to be equally immersed in the world of the movie.

[1310] As a result, the present invention provides users with a cinema-like movie-watching experience at home, making the overall movie-watching experience more interactive and immersive.

[1311] ---

[1312] The processing flow will be explained below.

[1313] ---

[1314] Step 1: Enter movie data

[1315] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[1316] Step 2: Analyzing the video data

[1317] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[1318] Step 3: Analyzing the audio data

[1319] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[1320] Step 4: Generate efficacy data

[1321] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[1322] Step 5: Submitting your effectiveness data

[1323] The server transmits the generated effect data to the device in real time, including the vibration intensity, duration, pattern, wind power, direction, and wind duration.

[1324] Step 6: Receiving and preparing effectiveness data

[1325] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[1326] Step 7: Synchronize movie playback

[1327] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[1328] Step 8: Implementing the vibration effect

[1329] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration for an explosion scene, a medium vibration for a fall scene, etc.

[1330] Step 9: Implementing the blowing effect

[1331] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[1332] Step 10: User Experience

[1333] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[1334] ---

[1335] Example 1

[1336] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1337] Conventional home theater systems can only provide a limited visual and auditory experience, making it difficult to fully experience the reality and immersion of a movie. Furthermore, users with visual or hearing impairments have difficulty fully enjoying the content of a movie. Therefore, a means was needed to enable all users to enjoy watching movies equally.

[1338] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1339] In this invention, the server includes: means for analyzing video data and identifying important events; means for analyzing audio data and identifying important sound fields; means for generating physical effect data based on the analyzed video data and audio data; means for transmitting the generated physical effect data to a control device; means for controlling a vibration device and an air blower based on the physical effect data received by the control device; and means for simultaneously analyzing the video data and audio data and generating time-stamped effect data for a specific scene. This allows users to experience movie scenes not only visually but also through touch and wind, significantly improving the sense of immersion in the movie. Furthermore, all users, including those with visual or hearing impairments, can enjoy a realistic movie-watching experience.

[1340] "Video data" refers to electronic data containing visual information such as movies, videos, and animations.

[1341] "Audio data" refers to electronic data containing auditory information such as movies, videos, and music.

[1342] Analysis is the process of examining and breaking down data to identify meaning and patterns.

[1343] A "significant event" is a scene or action in a film or video that is particularly notable visually or audibly.

[1344] "Important sound fields" refer to areas in the audio data where specific frequencies or sound pressure levels are prominent.

[1345] "Physical effects data" refers to data for instructing physical effects such as vibrations and wind that correspond to specific scenes in a movie or video.

[1346] The "control device" is a device that operates the vibration device and the air blower based on the received physical effect data.

[1347] A "vibrator" is a device that delivers physical vibrations to a user at a specified intensity and pattern.

[1348] A "blower" is a device that delivers physical wind to a user at a specified wind force and direction.

[1349] A "timestamp" indicates the date and time when a particular data or event occurred.

[1350] A "generative AI model" refers to a trained model that uses artificial intelligence to analyze data and generate effectiveness data.

[1351] This invention is an immersive home theater system that utilizes AI technology. This system is designed to allow users to experience realistic vibrations and wind while watching a movie. Specifically, the system analyzes video and audio data and controls vibration and airflow devices to generate physical effects according to the movie scene.

[1352] System Configuration

[1353] This system consists of the following hardware and software:

[1354] 1. Server: Analyzes video and audio data and generates physical effect data.

[1355] 2. Terminal: Receives effect data sent from the server and controls the vibration device and the air blower.

[1356] 3. Vibration device: Attached to the sofa or chair on which the user sits, it generates specified vibrations.

[1357] 4. Blower: Placed around the user and generates a specified amount of wind.

[1358] Program processing

[1359] Data analysis by server

[1360] The server receives the movie video and audio data sent by the user. The server is equipped with a generative AI model (e.g., a general-purpose AI model) that analyzes the video data frame by frame. For example, the generative AI model detects sudden changes in brightness and specific movements to recognize explosions and action scenes. It also performs spectral analysis of the audio data to identify specific sound fields (e.g., explosions, thunder). The server generates time-stamped physical effect data based on the results of these analyses.

[1361] Generating efficacy data

[1362] Based on the analysis results, the server generates vibration and wind effect data corresponding to a specific scene. For example, it generates strong vibrations for an explosion scene and instantaneous strong winds for a thunder scene. The generated effect data includes the strength, duration, and pattern of vibrations, as well as the wind power, direction, and duration.

[1363] Sending and executing effect data

[1364] The server transmits the generated physical effect data to the device in real time. The device receives this data and controls the vibration device and the air blower in accordance with the timing of the movie playback. For example, when the movie playback reaches an explosion scene, the device instructs the vibration device to generate a vibration of a specified strength and the air blower to generate a specified strong wind.

[1365] Specific examples

[1366] Processing thunder scenes

[1367] The user starts playing a movie. The server detects flashes of lightning from the video data and analyzes the peaks of thunder from the audio data. It then generates data on the effects of strong vibrations and instantaneous wind and sends it to the device. The device then instructs the vibration device to generate strong vibrations and the air blower to generate instantaneous strong winds, synchronizing with the movie playback. The user experiences this scene, greatly enhancing the realism of the movie.

[1368] Prompt Sentence Examples

[1369] An example of a prompt to input to a generative AI model is as follows:

[1370] Generate effects data to control vibration and airflow devices in a thunder scene in a movie. The effects data should include the following information:

[1371] 1. Vibration strength (strong, medium, weak)

[1372] 2. Duration (seconds)

[1373] 3. Vibration pattern (continuous, intermittent)

[1374] 4. Wind power (strong, medium, weak)

[1375] 5. Wind direction (front, back, left, right)

[1376] 6. Wind duration (seconds)

[1377] As a result, by using this invention, users can enjoy a more realistic and immersive movie viewing experience. The system provides tactile and wind experiences in addition to visual and auditory experiences, allowing all users to enjoy movies with a highly immersive feeling.

[1378] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1379] Step 1: (User) Play the movie

[1380] A user plays a movie on a device that contains the movie's video and audio data and is configured to stream it to a server in real time. The input is the movie's video and audio data, and the output is the data sent to the server.

[1381] Step 2: (Server) Receiving Data

[1382] The server receives the video and audio data of the movie sent by the user. The input is the video and audio data of the movie, and the output is that this data is stored in the server and prepared for analysis.

[1383] Step 3: (Server) Video data analysis

[1384] A generative AI model (e.g., a general-purpose AI model) in the server analyzes the received video data frame by frame. Specifically, the generative AI model detects sudden changes in brightness and specific movements, and identifies important events such as explosions or action scenes. The input is the video data, and the output is a list of identified events.

[1385] Step 4: (Server) Audio data analysis

[1386] The server analyzes the audio data received in synchronization with the video data. It performs a spectral analysis of the audio data to detect peaks and frequencies of specific sound fields (e.g., explosions, thunder). Specifically, it analyzes specific frequency peaks and sound pressure levels within the audio data to identify important sound fields. The input is the audio data, and the output is a list of identified sound fields.

[1387] Step 5: (Server) Generate effect data

[1388] The server generates physical effect data based on the analysis results. Specifically, it generates vibration effect data and wind effect data corresponding to a specific scene based on the analysis results of the video data and audio data. For example, it generates "strong vibration" for an explosion scene and "instantaneous strong wind" for a thunder scene. The input is a list of identified events and sound fields, and the generated physical effect data is obtained as output.

[1389] Step 6: (Server) Sending effect data

[1390] The server sends the generated physical effect data to the device in real time. Specifically, it assigns a timestamp to the effect data and sends it to the device at the appropriate time. The input is the generated physical effect data, and the output is the physical effect data with the timestamp sent to the device.

[1391] Step 7: (Terminal) Receive effect data and wait for execution

[1392] The device holds the physical effect data received from the server in a standby state. Specifically, the received data is stored in memory and placed in a standby state as appropriate in accordance with the timing of movie playback. The input is the time-stamped physical effect data sent from the server, and the output is the data in a standby state.

[1393] Step 8: (Terminal) Control of vibration and blower devices

[1394] The device executes the received effect data in synchronization with the movie scenes. Specifically, it issues instructions to a vibration device and a blower to generate physical effects in accordance with the movie playback. For example, when the movie playback reaches an explosion scene, it issues an instruction to a vibration device to generate a "strong vibration" and an instruction to a blower to generate a "momentary strong wind." The input is the physical effect data in a standby state, and the output is the executed physical effect.

[1395] Step 9: (User) Enjoyment of the experience

[1396] The user experiences vibrations and wind that occur in real time in response to movie scenes. For example, during a thunder scene, the sofa vibrates strongly, and at the same time, a momentary breeze is felt, bringing the scene to life. The input is the physical effect that is being performed, and the output is a realistic movie experience that is felt by the user.

[1397] (Application example 1)

[1398] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1399] Conventional immersive systems in home theaters and entertainment facilities rely on visual and auditory perception, which can prevent users from fully experiencing the virtual reality world. This also makes it difficult for visually or auditory impaired users to enjoy the content. Furthermore, physical feedback during events and important scenes is incomplete, creating a need for improved realism and immersion.

[1400] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1401] In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for providing physical feedback to the user using a virtual reality device based on the effect data, thereby enabling the user to have a more realistic and immersive experience not only through video and audio but also through tactile sensations and wind.

[1402] "Video data" refers to data containing visual information captured by a video camera or other device.

[1403] "Analysis" is the process of extracting meaningful information from data and understanding and recognizing it.

[1404] A "significant event" is something that is of particular interest to the user, causing a particular action or change in the video or audio.

[1405] "Audio data" refers to data that includes auditory information collected by a microphone or the like.

[1406] An "important sound field" is an acoustic phenomenon that occurs at a particular time or place and is particularly noticeable to the user.

[1407] "Effect data" is information for controlling physical devices that is generated as a result of analyzing video data and audio data.

[1408] The "control device" is a device that receives the transmitted effect data and operates a physical device such as a vibration device or a ventilation device based on the data.

[1409] A "vibration device" is a device that generates physical vibrations in response to input instructions.

[1410] A "blowing device" is a device that generates air based on input instructions.

[1411] A "virtual reality device" is a device that allows a user to visually and aurally experience a virtual environment or situation, and typically includes a head-mounted display and sound system.

[1412] "Physical feedback" refers to providing the user with physical sensations such as vibrations or wind in response to changes in video and audio.

[1413] This invention describes an immersive VR ride attraction system that applies an immersive home theater system using AI. Specific embodiments and system program processing are described below.

[1414] System Configuration

[1415] The system mainly includes the following components:

[1416] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[1417] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. It works in conjunction with the virtual reality equipment.

[1418] 3. Vibration device: A device that generates vibrations in the seat on which the user sits or in the equipment worn by the user.

[1419] 4. Blowing device: A device that makes the user feel the wind at a specific force and direction.

[1420] 5. Virtual reality equipment: A device that allows the user to experience a virtual environment visually and audibly.

[1421] 6. User: A user who enjoys an experiential VR ride attraction through this system.

[1422] Data analysis by server

[1423] The server uses Python to analyze video and audio data. It uses a generative AI model to recognize important events in each frame of the video data. It uses the SciPy library to perform spectral analysis on the audio data to identify important sound fields. Specifically, it detects data corresponding to specific events, such as explosions, falls, or thunder. Based on the information obtained from this analysis, it generates effect data to control vibration and airflow devices.

[1424] Generating efficacy data

[1425] The server generates vibration and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. Based on the audio data, it generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[1426] Sending and executing effect data

[1427] The server sends the generated effect data to the device in real time. The device keeps the received effect data in a standby state and synchronizes with the virtual reality device to execute the effect at the appropriate time. The device controls the vibration device and air blowing device using a control program in Unity or Arduino. This allows the user to experience the movie scenes in a very realistic way, not only through their eyes but also through their sense of touch and wind.

[1428] Specific examples

[1429] A specific example will be described below, where thunder rumbles in a certain scene.

[1430] 1. The user starts the attraction and puts on the VR headset.

[1431] 2. The server detects lightning flashes from the video data and analyzes the peaks of thunder from the audio data.

[1432] 3. The server generates data on the effects of strong vibrations and instantaneous winds on lightning flashes.

[1433] 4. The server sends the effect data to the terminal.

[1434] 5. The terminal instructs the vibration device to vibrate strongly and the air blowing device to blow a momentary strong wind in time with the playback timing of the VR attraction.

[1435] 6. While experiencing this scene, the user feels the seat and the wearable device vibrate strongly, and at the same time, they feel a momentary gust of wind, allowing them to experience the thunder scene in a realistic way.

[1436] Prompt Sentence Examples

[1437] "I want to design a system that will create strong vibrations and wind effects when an explosion occurs during a VR experience. To achieve this, I will create an AI program that will detect important events and generate the corresponding vibration and wind effects."

[1438] As a result, the present invention provides users with a more immersive and realistic experience in virtual reality, making the overall entertainment experience more interactive and immersive.

[1439] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1440] Step 1:

[1441] The server receives video data. The input is video data obtained from a video camera or other capture device. The server uses a generative AI model to analyze each frame of the video and recognize important events (e.g., explosions, falls). The output is a list of important events detected in the video.

[1442] Step 2:

[1443] The server receives audio data. The input is audio data collected from a microphone or other device. The server uses the SciPy library to perform spectral analysis of the audio data and identify important sound fields (e.g., thunder, explosions). The output is a list of important sound fields recognized in the audio.

[1444] Step 3:

[1445] The server generates effect data based on the analysis results of the video and audio data obtained in steps 1 and 2. The inputs are a list of important events and a list of sound fields. The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. The output is a set of corresponding effect data.

[1446] Step 4:

[1447] The server sends the generated effect data to the terminal. The input is a set of effect data. The effect data is sent from the server in real time. The output is the terminal that receives the effect data.

[1448] Step 5:

[1449] The terminal keeps the received effect data in a standby state. The input is the effect data sent from the server. The terminal synchronizes with the playback timing of the movie or VR attraction and prepares to execute the appropriate effect. The output is in a ready state.

[1450] Step 6:

[1451] The terminal controls the vibration device and the air blowing device in accordance with the playback timing. The input is the effect data and playback timing. The terminal uses a control program in Unity or Arduino to operate the physical devices with the specified intensity and duration. Specific actions include the seat or wearable device vibrating and the air blowing device generating wind. The output is the vibration and wind experienced by the user.

[1452] Through these steps, users can experience realistic changes in images and sounds not only through their eyes but also through their sense of touch and wind, making it possible to provide a truly immersive entertainment experience in the virtual reality world.

[1453] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1454] ---

[1455] This invention combines an AI-based immersive home theater system with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are shown below, and the system's program processing is explained in natural language.

[1456] System Configuration

[1457] The system mainly includes the following components:

[1458] 1. Server: A device that analyzes video and audio data and recognizes important events and sound fields.

[1459] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[1460] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[1461] 4. Blowing device: A device that makes the user feel the wind at a specified force and direction.

[1462] 5. Emotion engine: A device that recognizes a user's emotions by analyzing their facial expressions, voice tone, or biometric data.

[1463] 6. User: A person who enjoys immersive movie watching through this system.

[1464] Program processing

[1465] Data analysis by server

[1466] The server receives the video and audio data of the movie sent by the user. The generative AI model on the server analyzes this data and recognizes important events in each video frame. For example, it identifies explosions and action scenes. It also performs spectral analysis on the audio data and detects peaks and specific frequencies in the sound field to recognize important sound fields such as explosions and roars. The server assigns timestamps to each important event and sound field and generates corresponding effect data.

[1467] Efficacy data generation and adjustment

[1468] The server generates vibration effect data and wind effect data based on the analysis results. For example, it generates "strong vibration" for an explosion scene and "medium vibration" for a fall scene. For audio data, it also generates "instant strong wind" for thunder. This effect data includes the strength, duration, pattern, wind force, direction, and wind duration of the vibration.

[1469] Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, or biometric data to recognize the user's current emotion. For example, if the user is feeling scared, the emotion engine will detect this and adjust the effects of the vibration and airflow devices.

[1470] Sending and executing effect data

[1471] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing. It is also possible to dynamically adjust the effect data in real time based on data from the emotion engine.

[1472] For example, if the user is very tense, the vibration effect can be reduced and the airflow can be gentler to soothe the experience, while conversely, if the user is relaxed, the effect can be intensified.

[1473] Specific examples

[1474] Processing scary scenes from horror movies

[1475] The user plays a horror movie and reaches a scary scene. The following happens:

[1476] 1. The server detects sudden changes in brightness from the video data and frightening sounds from the audio data, such as a shadow appearing in the dark or a sudden scream.

[1477] 2. The server generates the appropriate vibration and cold wind effect data for the scary scene.

[1478] 3. The emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize that the user is already feeling very scared.

[1479] 4. The server takes into account the data from the emotion engine and reduces the vibration intensity and gently adjusts the fan effect.

[1480] 5. The device will then execute coordinated vibration and airflow effects when the movie reaches a scary scene.

[1481] 6. By experiencing a moderately adjusted immersive effect, users can enjoy a realistic immersive movie experience while mitigating excessive fear.

[1482] Accommodating visually and hearing impaired users

[1483] This system is also suitable for users with visual or hearing impairments. By providing immersive effects that are independent of video and audio, users can enjoy movie scenes and action through vibrations and wind. In addition, the emotion engine understands the user's emotional state and adjusts the appropriate effects, allowing users to enjoy movies comfortably.

[1484] As a result, the present invention provides users with a cinema-like experience of watching movies at home, and by utilizing an emotion engine to create an experience optimized for each user, the overall movie-watching experience becomes more interactive and personalized.

[1485] The processing flow will be explained below.

[1486] ---

[1487] Step 1: Enter movie data

[1488] Users input movie video and audio data into the system by providing a streaming URL or uploading a digital file, which is then received by the server.

[1489] Step 2: Analyzing the video data

[1490] The server analyzes the received video data frame by frame, using a generative AI model to detect important events (e.g., explosions, falls) in each frame and assign them a timestamp.

[1491] Step 3: Analyzing the audio data

[1492] The server performs spectral analysis of the audio data to detect volume peaks and specific frequencies, recognizing important sound fields (e.g. explosions, roars) and adding timestamps.

[1493] Step 4: Generate efficacy data

[1494] The server combines the results of video and audio analysis to generate effect data (vibration and wind patterns). For example, strong vibration and strong wind effect data is generated for explosion scenes, and light vibration or no wind effect is generated for quiet conversation scenes.

[1495] Step 5: Obtaining emotion data

[1496] The emotion engine monitors the user's facial expressions, voice tone, or biometric data to analyze the user's emotions in real time. For example, the emotion engine can determine whether the user is scared or excited.

[1497] Step 6: Adjusting the effect data

[1498] The server dynamically adjusts the effect data based on the user's emotional data acquired by the emotion engine. For example, if the user is already feeling scared, the server reduces the intensity of the vibrations and softens the airflow.

[1499] Step 7: Submitting your effectiveness data

[1500] The server transmits the generated and adjusted effect data to the device in real time, including vibration intensity, duration, pattern, wind power, direction, and wind duration.

[1501] Step 8: Receiving and preparing effectiveness data

[1502] The device receives the effect data sent from the server and keeps it in a standby state, ready to activate the effect at the correct timing when the movie is played.

[1503] Step 9: Synchronize movie playback

[1504] While the user is playing the movie, the device synchronizes the movie's progress with the timestamps of the effect data, and executes the effect when the movie reaches the analyzed event or sound field.

[1505] Step 10: Implementing the vibration effect

[1506] Based on the events analyzed in the movie, the device will instruct the vibration device on the intensity, duration and pattern of vibrations, for example, a strong vibration during an explosion scene, a medium vibration during a fall scene, etc.

[1507] Step 11: Implementing the blowing effect

[1508] Based on the sound field analyzed in the movie, the device will instruct the fan device on the power, direction and duration of the wind, for example, generating a strong wind in a stormy scene and a gentle breeze in a calm scene.

[1509] Step 12: User Experience

[1510] While watching a movie, users can experience the effects provided by the vibration and airflow devices, allowing them to become immersed in the movie scenes and enjoy a realistic, immersive movie viewing experience.

[1511] ---

[1512] The above is the specific processing flow for an invention that combines an emotion engine with an immersive home theater system that utilizes AI.

[1513] Example 2

[1514] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1515] Conventional immersive home theater systems generate effect data based only on video and audio data, without considering the user's emotional state. This results in a problem where a uniform immersive effect is provided regardless of the user's emotional state. In particular, if the user is feeling fear or tension, the experience may become overly stressful due to the lack of appropriate adjustments.

[1516] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for analyzing user emotion data and adjusting the effect data based on the emotion data, means for transmitting the generated effect data to the control device, and means for controlling the vibration device and the air blowing device based on the effect data received by the control device. This makes it possible to dynamically provide optimal bodily sensation effects according to the user's emotional state.

[1517] "Video data" is digital data containing visual information such as movies and videos.

[1518] "Audio data" is digital data containing the sound information of a film or video.

[1519] A "significant event" refers to a specific occurrence in a movie or video scene that has a strong visual and emotional impact on the user.

[1520] "Critical sound fields" refer to specific acoustic environments that have a strong auditory and emotional impact on the user in film and video audio.

[1521] "Effect data" is instruction data for realizing physical effects such as vibrations and airflow that are generated based on the analyzed video data and audio data.

[1522] The "control device" is a device that receives effect data sent from the server and operates the vibration device and the air blowing device based on the effect data.

[1523] A "vibration device" is a device that generates physical vibrations based on generated effect data.

[1524] The "air blowing device" is a device that generates air based on the generated effect data.

[1525] "User's emotional data" is data that indicates the user's current emotional state, analyzed from the user's facial expression, voice tone, biometric information, and the like.

[1526] The "means for adjusting effect data based on emotional data" refers to a method for dynamically changing the intensity and duration of vibrations and airflow, etc., in accordance with the emotional state of the user.

[1527] The present invention is an immersive home theater system that uses AI technology to recognize a user's emotions and adjust the immersive effects according to those emotions, providing a more personalized movie-watching experience. Specific embodiments of the present invention are described in detail below.

[1528] System Configuration

[1529] The system includes the following components:

[1530] 1. Server: A device that analyzes video and audio data, recognizes important events and sound fields, and generates effect data.

[1531] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and the air blowing device.

[1532] 3. Vibration device: A device that generates physical vibrations in the sofa or chair on which the user sits.

[1533] 4. Air Blowing Device: A device that blows air at a specified force and direction to the user.

[1534] 5. Emotion Engine: A device that recognizes the user's emotions by analyzing their facial expressions, voice tone, and biometric data.

[1535] 6. User: A person who enjoys immersive movie viewing through this system.

[1536] Program processing

[1537] Data analysis by server

[1538] The server receives the video and audio data of the movie sent by the user. It analyzes this data using generative AI models such as TensorFlow and PyTorch. The server recognizes important events in each frame of the video data, and performs spectral analysis of the audio data using libraries such as LibROSA to recognize important sound fields.

[1539] As a specific example of processing, the server detects sudden changes in brightness from video data and specific acoustic patterns (e.g., explosions or thunder) from audio data. Based on these analysis results, effect data is generated.

[1540] Efficacy data generation and adjustment

[1541] The server generates vibration and wind effect data based on the analysis results. For example, it uses SciPy and NumPy to process and calculate the data. For example, it generates "strong vibration" (3 seconds) for an explosion scene, and "instantaneous strong wind" (1 second) for thunder.

[1542] Furthermore, the emotion engine uses OpenCV and DeepFace to analyze the user's facial expressions, voice tone, and biometric data to detect the user's current emotional state. If the user is feeling fear or tension, the emotion engine will collect that information and adjust the effect data.

[1543] For example, if the user is very nervous, the emotion engine will adjust the vibration intensity to 50% and the airflow strength to 20%.

[1544] Sending and executing effect data

[1545] The server sends the generated effect data to the device in real time using a low-latency protocol such as WebSocket. The device keeps the received effect data in a standby state and executes the appropriate effect in sync with the movie playback timing.

[1546] As a concrete example, consider the case where a user plays a scary scene from a horror movie. The server adjusts the effect data based on the scene analysis results and emotion engine data, and sends it to the device. The device then executes the adjusted vibration and airflow effects in accordance with the timing of the movie playback. This allows the user to enjoy an appropriately adjusted sensory effect without feeling excessive fear.

[1547] Specific operation example

[1548] Prompt Sentence Examples

[1549] As a specific example of operation, the following prompt sentence is input into the generative AI model.

[1550] 1. Video data analysis:

[1551] "Analyze video data and extract important events using generative AI models that identify explosions and action scenes."

[1552] 2. Analysis of audio data:

[1553] "Use spectral analysis to analyze audio data and recognize important sound fields such as explosions and thunder."

[1554] 3. Emotion Engine Analysis:

[1555] "Detect the user's current emotional state by analyzing their facial expressions, voice tone, and biometric data."

[1556] In this way, the present invention provides users with an optimal immersive movie-watching experience in real time, and utilizes an emotion engine to dynamically adjust effects according to the user's individual emotional state, making the movie-watching experience more interactive and personalized.

[1557] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1558] Processing Steps

[1559] Step 1:

[1560] Receiving movie data

[1561] The server receives the video and audio data of the movie from the user.

[1562] When the user presses the play button on the movie, the device transmits this data to the server in real time.

[1563] Input: Video data, audio data

[1564] Output: Received video and audio data

[1565] Specific behavior:

[1566] When a user presses the play button on a movie, the device uses the HTTP protocol to send video and audio data to the server, which receives it and prepares it for analysis.

[1567] Step 2:

[1568] Video data analysis

[1569] The server analyzes the received video data.

[1570] Uses a generative AI model (e.g., TensorFlow) to recognize important events in each frame.

[1571] Input: Video data

[1572] Output: List of significant events (e.g. explosions, action scenes)

[1573] Specific behavior:

[1574] The server analyzes each frame of video data, detecting specific patterns or changes (such as sudden changes in brightness), and uses a generative AI model to write something like "Frame 1500-2000: Explosion detected."

[1575] Step 3:

[1576] Analysis of audio data

[1577] The server analyzes the received voice data.

[1578] Spectral analysis is performed using libraries such as "LibROSA" to recognize important sound fields.

[1579] Input: Audio data

[1580] Output: List of important sound fields (e.g. explosions, thunder)

[1581] Specific behavior:

[1582] The server performs spectral analysis on the audio data to detect specific acoustic patterns (e.g. explosions, thunder), and lists information such as "Timestamp 00:10:30 - Thunderstorm sound detected."

[1583] Step 4:

[1584] Generating efficacy data

[1585] The server generates effect data based on the analyzed video data and audio data.

[1586] Calculate the vibration and ventilation effect data using "SciPy" and "NumPy".

[1587] Input: List of important events, list of important sound fields

[1588] Output: Effect data (e.g. strong vibration, momentary strong wind)

[1589] Specific behavior:

[1590] The server generates effect data such as "strong vibration" for explosion scenes and "instant strong wind" for thunder. For example, create "Effect 1: Strong vibration for 3 seconds" and "Effect 2: Quick strong wind for 1 second."

[1591] Step 5:

[1592] Emotional data analysis and effect adjustment

[1593] The emotion engine analyzes the user's facial expressions, voice tone, and biometric data.

[1594] "OpenCV" and "DeepFace" are used to recognize the user's emotions and adjust the effect data.

[1595] Input: User biometric data, emotional data

[1596] Output: Adjusted effect data

[1597] Specific behavior:

[1598] While the user is watching a movie, the emotion engine analyzes the user's face and obtains data such as "User appears scared." Based on this, the server makes adjustments such as "Reduce vibration to 50%" and "Lower wind intensity to 20%."

[1599] Step 6:

[1600] Sending effect data

[1601] The server transmits the generated effect data to the terminal.

[1602] Use low latency protocols (e.g. "WebSocket").

[1603] Input: Efficacy data

[1604] Output: Sends effect data to the terminal.

[1605] Specific behavior:

[1606] The server sends data to the device such as "Send Effect 1: Strong vibration for 3 seconds" and "Send Effect 2: Quick strong wind for 1 second."

[1607] Step 7:

[1608] Execution of effectiveness data

[1609] The terminal executes the received effect data in synchronization with the playback timing of the movie.

[1610] Controls vibration and air blowing devices.

[1611] Input: Efficacy data

[1612] Output: Execute effect device

[1613] Specific behavior:

[1614] Based on the effect data received by the device, it sends commands such as "execute strong vibration for 3 seconds" or "execute a momentary strong wind for 1 second" to the vibration device or air blowing device, allowing the user to experience the sensory effects in real time in line with the movie.

[1615] (Application example 2)

[1616] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1617] Conventional immersive home theater systems provide vibration and airflow effects based on video and audio data, but lack the technology to adjust these effects based on the user's emotional state. As a result, excessive vibration and airflow can make users uncomfortable, making it difficult to provide an optimal experience tailored to each individual user's emotions. The present invention aims to solve this problem and provide an immersive movie-watching experience optimized for each user.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and recognizing important events, means for analyzing audio data and recognizing important sound fields, means for generating effect data based on the analyzed video data and audio data, means for transmitting the generated effect data to the control device, means for controlling the vibration device and the air blowing device based on the effect data received by the control device, and means for analyzing the user's emotional state and adjusting the effect data based on the results. This makes it possible to provide a bodily sensation effect that is appropriately adjusted according to the user's emotions.

[1619] "Video data" is digital data that contains visual information such as movies and videos.

[1620] "Audio data" is digital data containing audio information contained in movies and videos.

[1621] "Important events" are information that refer to scenes or actions in a movie or video that deserve particular attention.

[1622] "Important sound fields" refers to information that indicates scenes in movies or videos that have particularly impressive sound effects.

[1623] "Effect data" is information relating to bodily sensation effects such as vibration and airflow that are generated based on video data and audio data.

[1624] The "control device" is a device that operates the vibration device and the air blowing device based on the generated effect data.

[1625] A "vibration device" is a device that generates physical vibrations on a sofa, chair, or other device on which a user sits.

[1626] A "blowing device" is a device that allows a user to feel wind with a specified wind force and direction.

[1627] "User's emotional state" refers to the psychological state obtained by analyzing the user's facial expressions, voice, and biometric data.

[1628] "Analyzing" is the process of examining data in detail to reveal its components and patterns.

[1629] "Adjusting" is a means of making appropriate corrections or amendments based on the data and results obtained.

[1630] System configuration and hardware and software usage

[1631] The present invention is implemented by a system that mainly includes the following components, all of which are used to generate sensory effects tailored to a user's emotions and enhance the movie-watching experience:

[1632] 1. Server: A system that analyzes video and audio data and recognizes important events and sound fields. The server performs data analysis using a generative AI model (e.g., OpenAI's GPT-4).

[1633] 2. Terminal: A device that receives effect data sent from the server and controls the vibration device and air blowing device. Examples of this include smartphones and smart glasses.

[1634] 3. Vibration device: A device that is attached to a sofa or chair and generates physical vibrations.

[1635] 4. Blower device: A small blower that allows the user to feel the wind.

[1636] 5. Emotion engine: A device that can recognize a user's emotional state by analyzing their facial expressions, voice tone, or biometric data. Examples include Microsoft Azure Cognitive Services and IBM Watson.

[1637] Program processing and data processing

[1638] Server-based data analysis and generation of AI models

[1639] The server receives the video and audio data of the movie sent by the user. It then analyzes this data using a generative AI model to recognize important events and sound fields. For example, it identifies sudden scene changes and explosions. Based on the analysis results, the server generates vibration and airflow effect data. Generative AI models such as OpenAI's GPT-4 are used.

[1640] Generating effect data and analyzing the user's emotional state

[1641] The generated effect data includes the intensity, duration, and pattern of vibrations, as well as the wind force, direction, and duration. Furthermore, the emotion engine analyzes the user's facial expressions, voice tone, and biometric data to recognize the user's emotional state. For example, if the user is feeling fear, the emotion engine can detect this and the server can dynamically adjust the effect data.

[1642] Sending and executing effect data

[1643] The server sends the generated effect data to the terminal in real time. The terminal keeps the received effect data in a standby state and controls the vibration device and air blower device in sync with the movie playback timing. The effect data can be adjusted in real time based on the emotion engine data.

[1644] Specific examples

[1645] For example, if a user plays a horror movie and reaches a scary scene, the process is as follows:

[1646] 1. The user plays a movie.

[1647] 2. The server analyzes the video and audio data to detect shadows appearing in the darkness or sudden screams.

[1648] 3. The server generates vibration and cold wind effect data suitable for the horror scene.

[1649] 4. The emotion engine analyzes the user's facial expressions and biometric data and recognizes that they are already feeling extremely scared.

[1650] 5. The server takes these data into account and reduces the vibration intensity and gently adjusts the airflow effect.

[1651] 6. The device will execute vibration and airflow effects according to the adjusted effect data, timed to coincide with the scary scenes in the movie.

[1652] Example prompts to input to the generative AI model

[1653] The following prompt sentence can be used to have the system analyze the important events in the movie:

[1654] Prompt statement:

[1655] "Analyze the following video and audio data and recognize important events (e.g. explosions, sudden scene changes). Then generate an algorithm that adjusts the vibration and airflow effects based on the user's emotional state."

[1656] Video data: (video stream)

[1657] Audio data: (audio stream)

[1658] User emotion data: (facial expression and voice analysis data)

[1659] output:

[1660] Scene Events: Important events with time stamps

[1661] Effect data: appropriate vibration and airflow effects for each event (intensity, duration, pattern, wind force, direction)

[1662] Adjustment data: Adjustment data based on the user's emotional state

[1663] In this way, it is possible to provide a sensory effect that corresponds to the user's emotions, and to realize a home theater experience that is optimized for each individual user.

[1664] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1665] Step 1:

[1666] When a user plays a movie, the device transmits the movie's video and audio data to the server. This allows the server to receive the movie stream in real time. The input data are video and audio streams. Specifically, the video and audio captured by a smartphone or smart glasses are transferred to the server via the Internet.

[1667] Step 2:

[1668] The server inputs the video data into a generative AI model (such as GPT-4) to recognize important events. The video data is analyzed frame by frame to identify scenes such as explosions and falls. The generative AI model then timestamps these important events and creates an event list. The input data is the video data, and the output data is a list of important events with timestamps.

[1669] Step 3:

[1670] The server performs spectral analysis of the audio data to recognize important sound fields. For example, it identifies specific sound fields, such as roars or screams, by detecting sound peaks and specific frequency components. This is also done using a generative AI model. The input data is the audio data, and the output data is a time-stamped list of important sound fields.

[1671] Step 4:

[1672] The server generates effect data based on the important event list and the important sound field list. For example, it generates strong vibration effect data for an explosion scene, and instantaneous strong wind effect data for thunder. The input data are the important event list and the important sound field list, and the output data is the effect data. Specifically, detailed information such as the intensity, duration, pattern, wind force, direction, and wind duration of the vibrations is included.

[1673] Step 5:

[1674] The server transmits the generated effect data to the terminal in real time. The terminal stores the received effect data and keeps it in a standby state for execution. The input data is the effect data, and the output data is the effect data transmitted to the terminal. Specifically, the effect data is transmitted via wireless communication or the Internet.

[1675] Step 6:

[1676] The emotion engine analyzes the user's facial expression, voice tone, and biometric data to recognize the user's emotional state. For example, it detects whether the user's eyes are wide open in surprise or their voice is trembling in fear. The input data is the user's facial expression data, voice data, and biometric data, and the output data is the user's emotional state data.

[1677] Step 7:

[1678] The server dynamically adjusts the effect data based on the emotion engine data. For example, if the user is very nervous, it will reduce the vibration intensity and shorten the duration of the airflow. The input data is the emotional state data and existing effect data, and the output data is the adjusted effect data.

[1679] Step 8:

[1680] The terminal synchronizes with the playback timing of the movie and controls the vibration device and the air blowing device according to the adjusted effect data. This allows the user to experience appropriate bodily sensations according to the scene in the movie. The input data is the adjusted effect data, and the output data is the execution of physical vibrations and air blowing. Specifically, the terminal transmits control signals to operate the vibration device and the air blowing device based on the effect data.

[1681] In this way, the present invention provides a user with a personalized, immersive movie-watching experience.

[1682] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1683] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1684] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1685] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1686] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1687] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1688] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1689] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1690] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1691] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1692] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1693] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1694] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1695] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1696] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1697] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1698] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1699] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1700] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1701] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1702] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1703] The following is further disclosed regarding the above embodiment.

[1704] (Claim 1)

[1705] A means for analyzing video data and recognizing important events;

[1706] A means for analyzing audio data and recognizing important sound fields;

[1707] means for generating effect data based on the analyzed video data and audio data;

[1708] means for transmitting the generated effect data to the control device;

[1709] means for controlling the vibration device and the air blowing device based on the effect data received by the control device;

[1710] A system including:

[1711] (Claim 2)

[1712] 2. The system of claim 1,

[1713] The system generates effect data including the intensity, duration and pattern of vibration.

[1714] (Claim 3)

[1715] 2. The system of claim 1,

[1716] The system generates effect data including wind force, direction and wind duration.

[1717] The above is the original draft of the patent claim for an immersive home theater system that utilizes AI.

[1718] "Example 1"

[1719] (Claim 1)

[1720] means for analyzing the video data and identifying significant events;

[1721] means for analyzing the audio data and identifying important sound fields;

[1722] means for generating physical effect data based on the analyzed video data and audio data;

[1723] means for transmitting the generated physical effect data to a control device;

[1724] means for controlling the vibration device and the blower device based on the physical effect data received by the controller;

[1725] means for simultaneously analyzing the video data and audio data and generating time-stamped effect data for a specific scene;

[1726] A system including:

[1727] (Claim 2)

[1728] 10. The system of claim 1,

[1729] A system in which the generated physical effects data includes the intensity, duration and pattern of vibrations.

[1730] (Claim 3)

[1731] 10. The system of claim 1,

[1732] The system generates physical effects data including wind force, direction and wind duration.

[1733] "Application Example 1"

[1734] (Claim 1)

[1735] A means for analyzing video data and recognizing important events;

[1736] A means for analyzing audio data and recognizing important sound fields;

[1737] means for generating effect data based on the analyzed video data and audio data;

[1738] means for transmitting the generated effect data to the control device;

[1739] means for controlling the vibration device and the air blowing device based on the effect data received by the control device;

[1740] a means for providing a user with physical feedback using a virtual reality device based on the effect data;

[1741] A system including:

[1742] (Claim 2)

[1743] 10. The system of claim 1, wherein the generated effect data includes the intensity, duration, and pattern of the vibrations.

[1744] (Claim 3)

[1745] 10. The system of claim 1, wherein the generated effect data includes wind force, direction, and duration of the wind.

[1746] "Example 2: Combining Emotion Engines"

[1747] (Claim 1)

[1748] A means for analyzing video data and recognizing important events;

[1749] A means for analyzing audio data and recognizing important sound fields;

[1750] means for generating effect data based on the analyzed video data and audio data;

[1751] means for analyzing emotion data of a user and adjusting effect data based on the emotion data;

[1752] means for transmitting the generated effect data to the control device;

[1753] means for controlling the vibration device and the air blowing device based on the effect data received by the control device;

[1754] A system including:

[1755] (Claim 2)

[1756] 10. The system of claim 1, wherein the generated effect data includes the intensity, duration, and pattern of the vibrations.

[1757] (Claim 3)

[1758] 10. The system of claim 1, wherein the generated effect data includes wind force, direction, and duration of the wind.

[1759] "Application example 2 when combining emotion engines"

[1760] (Claim 1)

[1761] A means for analyzing video data and recognizing important events;

[1762] A means for analyzing audio data and recognizing important sound fields;

[1763] means for generating effect data based on the analyzed video data and audio data;

[1764] means for transmitting the generated effect data to the control device;

[1765] means for controlling the vibration device and the air blowing device based on the effect data received by the control device;

[1766] means for analyzing the emotional state of the user and adjusting the effect data based on the result of the analysis;

[1767] A system including:

[1768] (Claim 2)

[1769] 10. The system of claim 1, wherein the generated effect data includes the intensity, duration, and pattern of the vibrations.

[1770] (Claim 3)

[1771] 10. The system of claim 1, wherein the generated effect data includes wind force, direction, and duration of the wind. [Explanation of symbols]

[1772] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing video data and recognizing important events; A means for analyzing audio data and recognizing important sound fields; means for generating effect data based on the analyzed video data and audio data; means for transmitting the generated effect data to the control device; means for controlling the vibration device and the air blowing device based on the effect data received by the control device; A system including:

2. 10. The system of claim 1, The system generates effect data including the intensity, duration and pattern of vibration.

3. 10. The system of claim 1, The system generates effect data including wind force, direction and wind duration. The above is the original draft of the patent claim for an immersive home theater system that utilizes AI.

Images