system

Abstract

The system according to this embodiment aims to provide a holistic sensory experience that integrates sight, sound, touch, smell, and taste. [Solution] The system according to the embodiment comprises a visual unit, an acoustic unit, a tactile unit, a simulation unit, and an optimization unit. The visual unit provides a visual experience. The acoustic unit provides an acoustic experience based on the visual experience provided by the visual unit. The tactile unit provides a tactile experience based on the acoustic experience provided by the acoustic unit. The simulation unit simulates smells and tastes based on the tactile experience provided by the tactile unit. The optimization unit optimizes the experience based on the experience provided by the simulation unit.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, it is difficult to provide a multisensory experience integrating vision, sound, touch, smell, and taste, and there is room for improvement.

[0005] The system according to the embodiment aims to provide a multisensory experience integrating vision, sound, touch, smell, and taste.

Means for Solving the Problems

[0006] The system according to this embodiment comprises a visual unit, an acoustic unit, a tactile unit, a simulation unit, and an optimization unit. The visual unit provides a visual experience. The acoustic unit provides an acoustic experience based on the visual experience provided by the visual unit. The tactile unit provides a tactile experience based on the acoustic experience provided by the acoustic unit. The simulation unit simulates smells and tastes based on the tactile experience provided by the tactile unit. The optimization unit optimizes the experience based on the experience provided by the simulation unit. [Effects of the Invention]

[0007] The system according to this embodiment can provide a holistic sensory experience that integrates sight, sound, touch, smell, and taste. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10]This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 comprises a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The receiving device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The total sensory experience system according to an embodiment of the present invention is a system that makes full use of sight, sound, and touch, utilizing virtual reality (VR) and augmented reality (AR) technologies and AI, while simulating smell and taste to a realistic extent. This total sensory experience system constantly optimizes the experience based on user responses and is customized to each individual user. First, the user puts on a VR or AR device and starts the experience. The visual experience is provided with clear and immersive images through a high-resolution display and real-time rendering technology. For example, if the user is walking through a virtual forest, the details of the trees and plants are realistically reproduced. Next, the acoustic experience realistically reproduces ambient sounds using spatial audio technology. For example, if the user is in a virtual concert hall, the sounds of instruments and the cheers of the audience are heard three-dimensionally, giving the user the feeling of being there. The tactile experience reproduces the feel of touching objects in the virtual space through a haptic feedback device. For example, if the user picks up a virtual stone, its weight and texture are transmitted to the hand. Furthermore, simulations of smell and taste are also performed. While currently limited to realistic simulations, this system allows users to experience the aroma and taste of coffee in a virtual cafe. This holistic sensory experience system is expected to have applications in various fields, including entertainment, education, healthcare, tourism, and rehabilitation. For example, it could be used in immersive games and virtual concerts in the entertainment industry, for recreating historical events and learning scientific concepts in education, and for telemedicine and rehabilitation in healthcare. The experience is constantly optimized based on user responses. AI recognizes user behavior and the surrounding environment, adjusting the virtual space in real time. For instance, if a user is climbing a virtual mountain, the experience is adjusted according to their fatigue level and heart rate, providing a more realistic experience. Thus, this holistic sensory experience system, utilizing virtual reality (VR) and augmented reality (AR) technologies along with AI, provides users with an immersive experience that fully engages all five senses—sight, sound, touch, smell, and taste—and is expected to have applications in various fields. This allows the holistic sensory experience system to optimize the experience based on user responses, providing a customized experience tailored to each individual user.

[0029] The total sensory experience system according to this embodiment comprises a visual unit, an acoustic unit, a haptic unit, a simulation unit, and an optimization unit. The visual unit provides a visual experience. The visual unit provides a visual experience using, for example, a high-resolution display and real-time rendering technology. For example, if the user is walking through a virtual forest, the visual unit can realistically reproduce the details of trees and plants. The acoustic unit provides an acoustic experience based on the visual experience provided by the visual unit. The acoustic unit provides an acoustic experience using, for example, spatial audio technology. For example, if the user is in a virtual concert hall, the acoustic unit can make the sounds of musical instruments and audience cheers sound three-dimensional. The haptic unit provides a haptic experience based on the acoustic experience provided by the acoustic unit. The haptic unit provides a haptic experience using, for example, a haptic feedback device. For example, if the user picks up a virtual stone, the haptic unit can convey its weight and texture to the user's hand. The simulation unit simulates smells and tastes based on the haptic experience provided by the haptic unit. For example, the simulation unit simulates smells and tastes within a realistic range. The simulation unit allows the user to experience the aroma and taste of coffee when drinking it in a virtual cafe. The optimization unit optimizes the experience based on the experience provided by the simulation unit. The optimization unit optimizes the experience based on the user's reactions, for example. If the user is climbing a virtual mountain, the optimization unit can adjust the experience according to their fatigue level and heart rate. As a result, the holistic sensory experience system according to this embodiment can provide an immersive experience that fully utilizes the five senses: sight, sound, touch, smell, and taste.

[0030] The visual unit provides a visual experience. For example, it uses high-resolution displays and real-time rendering technology to deliver a visual experience. Specifically, the visual unit uses eye-tracking technology to render visual information in real time according to the direction the user is looking. This allows the user to obtain a natural visual experience. Furthermore, the visual unit uses HDR (High Dynamic Range) technology to enhance the contrast between light and dark, providing more realistic images. For example, when a user is walking through a virtual forest, not only can the details of trees and plants be realistically reproduced, but the reflection of light and the movement of shadows are also expressed in real time. The visual unit can also provide three-dimensional images using 3D display technology. This allows the user to feel a sense of depth in the virtual space, resulting in a more immersive experience. The visual unit also dynamically updates visual information according to the user's movements, allowing the user to move freely within the virtual space. This enables the visual unit to provide the user with a highly realistic and dynamic visual experience.

[0031] The acoustic unit provides an acoustic experience based on the visual experience provided by the visual unit. For example, the acoustic unit uses spatial audio technology to deliver the acoustic experience. Specifically, the acoustic unit adjusts the direction and distance of sound in real time according to the user's position and movement. This allows the user to accurately perceive the location of the sound source and obtain a more three-dimensional acoustic experience. For example, if the user is in a virtual concert hall, the sounds of instruments and audience cheers can be heard in three dimensions. The acoustic unit also provides a more realistic acoustic environment by generating ambient sounds and sound effects in real time and synchronizing them with the visual experience. For example, when walking through a virtual forest, the sound of wind, birdsong, and rustling leaves are realistically reproduced. Furthermore, the acoustic unit can customize the acoustic experience according to the user's auditory characteristics. For example, it adjusts the acoustic profile based on the user's ear shape and hearing ability to provide an optimal acoustic experience. This allows the acoustic unit to provide the user with a highly realistic and personalized acoustic experience.

[0032] The haptic unit provides a tactile experience based on the acoustic experience provided by the acoustic unit. The haptic unit provides a tactile experience, for example, using a haptic feedback device. Specifically, the haptic unit provides tactile feedback such as vibration, pressure, and temperature through the user's device or wearable device. For example, when a user picks up a virtual stone, its weight and texture can be transmitted to the user's hand. The haptic unit also provides a more realistic tactile experience by adjusting the intensity and pattern of the haptic feedback in real time. For example, when walking through a virtual forest, the user can feel the texture of the ground under their feet and the rustling of the grass. Furthermore, the haptic unit dynamically updates the haptic feedback according to the user's movements and posture, allowing the user to feel their movements in the virtual space more realistically. In this way, the haptic unit can provide the user with a highly realistic and dynamic tactile experience.

[0033] The simulation unit simulates smells and tastes based on the tactile experience provided by the haptic unit. For example, the simulation unit simulates smells and tastes within a realistic range. Specifically, it generates specific smells and tastes through a device worn by the user. For instance, when a user drinks coffee in a virtual cafe, they can experience its aroma and taste. The simulation unit also provides a more realistic experience by adjusting the intensity and type of smells and tastes in real time. For example, when walking through a virtual forest, the user can experience the scent of flowers and the smell of damp earth. Furthermore, the simulation unit can customize the experience according to the user's olfactory and gustatory characteristics. For example, it adjusts the smell and taste profile based on the user's olfactory and gustatory sensitivity to provide an optimal experience. This allows the simulation unit to provide users with a highly realistic and personalized smell and taste experience.

[0034] The optimization unit optimizes the experience based on the experience provided by the simulation unit. For example, the optimization unit optimizes the experience based on user responses. Specifically, the optimization unit collects the user's biometric information and behavioral data in real time and dynamically adjusts the content and intensity of the experience. For example, if a user is climbing a virtual mountain, the experience can be adjusted according to their fatigue level and heart rate. The optimization unit can also customize the experience based on the user's preferences and past experience history. For example, if a user likes certain music or scenery, it will provide an experience accordingly. Furthermore, the optimization unit uses AI to analyze user responses and build a feedback loop to provide the optimal experience. This allows the optimization unit to always provide the user with the best possible experience and enhance immersion. The optimization unit can also individually optimize the experience for each user even when multiple users are sharing the experience simultaneously. This allows the optimization unit to provide users with a highly personalized experience and maximize the effectiveness of the whole-sensory experience system.

[0035] The recognition unit can recognize the user's actions and the surrounding environment. For example, the recognition unit can recognize the user's movement patterns and gestures. For example, the recognition unit can acquire the user's location information and recognize the surrounding environment. For example, the recognition unit can recognize the temperature, lighting, and acoustic environment. This allows the user experience to be adjusted based on the user's actions and environment. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input the user's movement patterns into the AI ​​and have the AI ​​perform the recognition of the movement patterns.

[0036] The adjustment unit can adjust the virtual space. For example, the adjustment unit can adjust 3D models and environment settings. For example, the adjustment unit can adjust interaction elements. This allows the virtual space to be adjusted in real time. Some or all of the above-described processes in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input a 3D model of the virtual space into the AI ​​and have the AI ​​perform adjustments to the 3D model.

[0037] The visual unit can provide a visual experience using a high-resolution display and real-time rendering technology. For example, the visual unit can provide clear images using a high-resolution display. For example, the visual unit can generate images in real time using real-time rendering technology. For example, if the user is walking through a virtual forest, the visual unit can realistically reproduce even the smallest details of trees and plants. This enables the provision of a high-resolution, real-time visual experience. Some or all of the above-described processes in the visual unit may be performed using AI or not. For example, the visual unit can have AI perform the generation of images.

[0038] The audio unit can provide an audio experience using spatial audio technology. For example, the audio unit can calculate the location of sound sources to provide a three-dimensional audio experience. For example, the audio unit can enhance the realism of sound using sound effects. For example, if the user is in a virtual concert hall, the audio unit can make the sounds of musical instruments and audience cheers sound three-dimensional. This provides a three-dimensional audio experience. Some or all of the above processing in the audio unit may be performed using AI or not. For example, the audio unit can have AI perform the calculation of the location of sound sources.

[0039] The haptic unit can provide a tactile experience using haptic feedback devices. For example, the haptic unit can provide haptic feedback using a vibration device. For example, the haptic unit can provide haptic feedback using a pressure device. For example, the haptic unit can provide haptic feedback using a temperature device. For example, when a user picks up a virtual stone, the haptic unit can convey its weight and texture to the user's hand. This can provide a realistic tactile experience. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can have AI perform the control of haptic feedback.

[0040] The simulation unit can simulate smells and tastes within a realistic range. For example, the simulation unit can adjust the intensity and duration of smells. For example, the simulation unit can adjust the intensity and duration of tastes. For example, when a user drinks coffee in a virtual cafe, the simulation unit can experience its aroma and taste. This allows for the provision of smell and taste simulations. Some or all of the above-described processes in the simulation unit may be performed using AI or not. For example, the simulation unit can have AI perform smell and taste simulations.

[0041] The optimization unit can optimize the experience based on user responses. For example, the optimization unit can recognize the user's facial expressions and movements and adjust the experience. For example, the optimization unit can monitor the user's vital signs and adjust the experience. For example, if the user is climbing a virtual mountain, the optimization unit can adjust the experience according to the user's fatigue level and heart rate. This allows the experience to be optimized based on user responses. Some or all of the above-described processes in the optimization unit may be performed using AI or not. For example, the optimization unit can input the user's vital signs into the AI ​​and have the AI ​​perform the optimization of the experience.

[0042] The visual unit can dynamically change the focus of the visual experience using the user's eye-tracking data. For example, if the user is looking at a specific object, the visual unit will highlight that object. For example, if the user moves their gaze, the visual unit will automatically focus on the object in their line of sight. For example, if the user's gaze is fixed, the visual unit will adjust the surrounding visual information around that viewpoint. This allows the focus of the visual experience to change according to the user's gaze. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input eye-tracking data into AI and have the AI ​​perform the change in the focus of the visual experience.

[0043] The visual unit can monitor the user's pupillary response during the visual experience and adjust the intensity of the visual stimulus. For example, if the user's pupils dilate, the visual unit may decrease the intensity of the visual stimulus. For example, if the user's pupils constrict, the visual unit may increase the intensity of the visual stimulus. For example, if the user's pupillary response is not constant, the visual unit may dynamically adjust the intensity of the visual stimulus. This allows the intensity of the visual stimulus to be adjusted based on the user's pupillary response. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input pupillary response data into the AI ​​and have the AI ​​perform the adjustment of the intensity of the visual stimulus.

[0044] The visual unit can detect the user's head movements during the visual experience and change the viewpoint of the visual content in real time. For example, if the user moves their head to the left, the visual unit will move the viewpoint to the left. For example, if the user moves their head up and down, the visual unit will move the viewpoint up and down. For example, if the user rotates their head, the visual unit will rotate the viewpoint. This allows the viewpoint of the visual content to be changed in response to the user's head movements. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input head movement data into the AI ​​and have the AI ​​perform the viewpoint change.

[0045] The visual unit can analyze the user's blinking frequency during the visual experience and adjust the display speed of the visual content. For example, if the user blinks frequently, the visual unit will slow down the display speed. For example, if the user blinks infrequently, the visual unit will speed up the display speed. For example, the visual unit can dynamically adjust the display speed based on the user's blinking pattern. This allows the display speed of the visual content to be adjusted based on the user's blinking frequency. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input blinking frequency data into the AI ​​and have the AI ​​perform the adjustment of the display speed.

[0046] The acoustic unit can track the user's ear position during the audio experience and dynamically change the sound source position. For example, if the user moves their head to the left, the acoustic unit will move the sound source to the left. For example, if the user moves their head up and down, the acoustic unit will move the sound source up and down. For example, if the user rotates their head, the acoustic unit will rotate the sound source. This allows the sound source position to be changed according to the user's ear position. Some or all of the above processing in the acoustic unit may be performed using AI or not. For example, the acoustic unit can input ear position data into the AI ​​and have the AI ​​perform the sound source position change.

[0047] The acoustic unit can monitor the user's heart rate during the acoustic experience and adjust the rhythm of the acoustic stimulation. For example, the acoustic unit can speed up the rhythm if the user's heart rate is high, or slow down if the user's heart rate is low. The acoustic unit can also dynamically adjust the rhythm based on fluctuations in the user's heart rate. This allows the rhythm of the acoustic stimulation to be adjusted based on the user's heart rate. Some or all of the above processing in the acoustic unit may be performed using AI or not. For example, the acoustic unit can input heart rate data into the AI ​​and have the AI ​​perform the rhythm adjustment.

[0048] The audio unit can detect the user's breathing pattern during the audio experience and adjust the tempo of the audio content. For example, if the user's breathing is fast, the audio unit will speed up the tempo. For example, if the user's breathing is slow, the audio unit will slow down the tempo. For example, the audio unit can dynamically adjust the tempo based on the user's breathing pattern. This allows the tempo of the audio content to be adjusted based on the user's breathing pattern. Some or all of the above processing in the audio unit may be performed using AI or not. For example, the audio unit can input breathing pattern data into the AI ​​and have the AI ​​perform the tempo adjustment.

[0049] The acoustics unit can analyze the user's voice tone during the audio experience and provide acoustic feedback. For example, if the user's voice is high-pitched, the acoustics unit provides bright acoustic feedback. For example, if the user's voice is low-pitched, the acoustics unit provides calm acoustic feedback. The acoustics unit can dynamically adjust the acoustic feedback based on the user's voice tone, for example. This allows the acoustics unit to provide acoustic feedback based on the user's voice tone. Some or all of the above processing in the acoustics unit may be performed using AI or not. For example, the acoustics unit can input voice tone data into an AI and have the AI ​​perform the task of providing acoustic feedback.

[0050] The haptic unit can monitor the user's skin temperature during the haptic experience and dynamically change the temperature of the haptic stimulus. For example, if the user's skin temperature is high, the haptic unit will lower the temperature of the haptic stimulus. For example, if the user's skin temperature is low, the haptic unit will raise the temperature of the haptic stimulus. For example, the haptic unit can dynamically adjust the temperature of the haptic stimulus based on fluctuations in the user's skin temperature. This allows the temperature of the haptic stimulus to be adjusted based on the user's skin temperature. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input skin temperature data into the AI ​​and have the AI ​​perform the adjustment of the haptic stimulus temperature.

[0051] The haptic unit can detect the user's muscle responses during the haptic experience and adjust the haptic feedback pattern. For example, if the user's muscles are tense, the haptic unit will soften the haptic feedback pattern. For example, if the user's muscles are relaxed, the haptic unit will strengthen the haptic feedback pattern. The haptic unit can dynamically adjust the haptic feedback pattern based on the user's muscle responses. This allows the haptic feedback pattern to be adjusted based on the user's muscle responses. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input muscle response data into the AI ​​and have the AI ​​perform the adjustment of the haptic feedback pattern.

[0052] The haptic unit can track the user's hand movements during the haptic experience and dynamically change the position of the haptic feedback. For example, if the user moves their hand to the left, the haptic unit will move the position of the haptic feedback to the left. For example, if the user moves their hand up and down, the haptic unit will move the position of the haptic feedback up and down. For example, if the user rotates their hand, the haptic unit will rotate the position of the haptic feedback. This allows the position of the haptic feedback to change according to the user's hand movements. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input hand movement data into the AI ​​and have the AI ​​perform the change in the position of the haptic feedback.

[0053] The haptic unit can analyze the user's pressure sensation during the haptic experience and adjust the pressure of the haptic feedback. For example, if the user feels strong pressure, the haptic unit will lower the pressure of the haptic feedback. For example, if the user feels weak pressure, the haptic unit will increase the pressure of the haptic feedback. The haptic unit can dynamically adjust the pressure of the haptic feedback based on the user's pressure sensation. This allows the pressure of the haptic feedback to be adjusted based on the user's pressure sensation. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input pressure sensation data into the AI ​​and have the AI ​​perform the adjustment of the haptic feedback pressure.

[0054] The simulation unit can monitor the user's olfactory response during smell and taste simulations and dynamically change the simulation content. For example, if the user's olfactory response is strong, the simulation unit may weaken the simulation content. For example, if the user's olfactory response is weak, the simulation unit may strengthen the simulation content. For example, the simulation unit may dynamically adjust the simulation content based on the user's olfactory response. This allows the simulation content to be adjusted based on the user's olfactory response. Some or all of the above processing in the simulation unit may be performed using AI or not. For example, the simulation unit can input olfactory response data into the AI ​​and have the AI ​​execute changes to the simulation content.

[0055] The simulation unit can detect the user's saliva secretion during smell and taste simulations to improve the realism of the simulation. For example, the simulation unit increases the realism of the simulation if the user's saliva secretion is high. For example, the simulation unit decreases the realism of the simulation if the user's saliva secretion is low. For example, the simulation unit dynamically adjusts the realism of the simulation based on the user's saliva secretion. This allows the realism of the simulation to be adjusted based on the user's saliva secretion. Some or all of the above processing in the simulation unit may be performed using AI or not. For example, the simulation unit can input saliva secretion data into the AI ​​and have the AI ​​perform the adjustment of the simulation realism.

[0056] The simulation unit can refer to the user's eating history during the simulation of smells and tastes and customize the simulation content. For example, the simulation unit can simulate the smells and tastes of dishes that the user has enjoyed eating in the past. For example, the simulation unit can simulate the smells and tastes of specific ingredients from the user's eating history. For example, the simulation unit can dynamically adjust the simulation content based on the user's eating history. This allows the simulation content to be customized based on the user's eating history. Some or all of the above processes in the simulation unit may be performed using AI or not. For example, the simulation unit can input eating history data into AI and have the AI ​​perform the customization of the simulation content.

[0057] The simulation unit can analyze user preference data during smell and taste simulations and adjust the type of simulation. For example, the simulation unit simulates preferred smells and tastes based on the user preference data. For example, the simulation unit emphasizes specific flavors based on the user preference data. For example, the simulation unit dynamically adjusts the type of simulation based on the user preference data. This allows the type of simulation to be adjusted based on the user preference data. Some or all of the above processing in the simulation unit may be performed using AI or not. For example, the simulation unit can input preference data into AI and have the AI ​​perform the adjustment of the type of simulation.

[0058] The optimization unit can analyze the user's biometric data in real time during optimization and dynamically change the experience content. For example, if the user's heart rate is high, the optimization unit will slow down the experience content. For example, if the user's heart rate is low, the optimization unit will intensify the experience content. For example, the optimization unit will dynamically adjust the experience content based on the user's biometric data. This allows the experience content to be dynamically changed based on the user's biometric data. Some or all of the above processing in the optimization unit may be performed using AI or not. For example, the optimization unit can input biometric data into AI and have the AI ​​perform the changes to the experience content.

[0059] The optimization unit can refer to the user's past experience history during optimization to improve the accuracy of the optimization. For example, the optimization unit provides the optimal experience content based on the user's past experience history. For example, the optimization unit extracts specific patterns from the user's past experience history to improve the accuracy of the optimization. For example, the optimization unit analyzes the user's past experience history and improves the optimization algorithm. This allows the accuracy of the optimization to be improved based on the user's past experience history. Some or all of the above processes in the optimization unit may be performed using AI or not. For example, the optimization unit can input past experience history data into AI and have the AI ​​perform the optimization accuracy improvement.

[0060] The optimization unit can refer to the user's environmental data during optimization and customize the user experience. For example, if the temperature around the user is high, the optimization unit can make the user experience cooler. For example, if the temperature around the user is low, the optimization unit can make the user experience warmer. For example, the optimization unit can dynamically adjust the user experience based on the user's environmental data. This allows the user experience to be customized based on the user's environmental data. Some or all of the above processing in the optimization unit may be performed using AI or not. For example, the optimization unit can input environmental data into the AI ​​and have the AI ​​perform the customization of the user experience.

[0061] The optimization unit can collect user feedback during optimization and improve the optimization algorithm. For example, the optimization unit adjusts the optimization algorithm based on user feedback. For example, the optimization unit extracts specific patterns from user feedback and improves the optimization algorithm. For example, the optimization unit analyzes user feedback and improves the accuracy of the optimization algorithm. This allows the optimization algorithm to be improved based on user feedback. Some or all of the above processes in the optimization unit may be performed using AI or not. For example, the optimization unit can input feedback data into AI and have the AI ​​perform improvements to the optimization algorithm.

[0062] The recognition unit can analyze the user's behavior patterns in real time during recognition and improve recognition accuracy. For example, the recognition unit improves recognition accuracy based on the user's behavior patterns. For example, the recognition unit analyzes the user's behavior patterns in real time and adjusts the recognition algorithm. For example, the recognition unit analyzes the user's behavior patterns and dynamically adjusts the recognition accuracy. This allows for improvement of recognition accuracy based on the user's behavior patterns. Some or all of the above processes in the recognition unit may be performed using AI or not. For example, the recognition unit can input behavior pattern data into the AI ​​and have the AI ​​perform the improvement of recognition accuracy.

[0063] The recognition unit can refer to the user's environmental data during recognition and customize the recognized content. For example, if the temperature around the user is high, the recognition unit will make the recognized content cooler. For example, if the temperature around the user is low, the recognition unit will make the recognized content warmer. For example, the recognition unit will dynamically adjust the recognized content based on the user's environmental data. This allows the recognized content to be customized based on the user's environmental data. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input environmental data into the AI ​​and have the AI ​​perform the customization of the recognized content.

[0064] The adjustment unit can analyze the user's biometric data in real time during adjustment and dynamically change the adjustment settings. For example, if the user's heart rate is high, the adjustment unit will lessen the adjustment settings. For example, if the user's heart rate is low, the adjustment unit will strengthen the adjustment settings. For example, the adjustment unit will dynamically adjust the adjustment settings based on the user's biometric data. This allows the adjustment settings to be dynamically changed based on the user's biometric data. Some or all of the above processing in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input biometric data into AI and have AI perform the changes to the adjustment settings.

[0065] The adjustment unit can refer to the user's environmental data during adjustment and customize the adjustment settings. For example, if the temperature around the user is high, the adjustment unit will make the settings cooler. For example, if the temperature around the user is low, the adjustment unit will make the settings warmer. For example, the adjustment unit will dynamically adjust the settings based on the user's environmental data. This allows the adjustment settings to be customized based on the user's environmental data. Some or all of the above processing in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input environmental data into the AI ​​and have the AI ​​perform the customization of the adjustment settings.

[0066] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0067] The holistic sensory experience system can also include a biometric data monitoring unit that monitors the user's biometric data in real time. This unit can acquire data such as the user's heart rate, blood pressure, and body temperature, and adjust the experience based on this data. For example, if the user's heart rate increases, the system can reduce the intensity of the experience. If the user's body temperature is high, the system can lower the temperature in the virtual space. Furthermore, if the user's blood pressure drops, the system can temporarily suspend the experience and prompt the user to rest. This allows for dynamic adjustment of the experience based on the user's biometric data, providing a safer and more comfortable experience.

[0068] The holistic sensory experience system can also include a history reference unit that customizes the experience by referring to the user's past experience history. For example, the history reference unit can provide similar experiences based on what the user has enjoyed in the past. It can also suggest different experiences based on what the user has avoided in the past. Furthermore, it can extract specific patterns from the user's past experience history and optimize the experience. This allows for the provision of a more personalized experience based on the user's past experience history.

[0069] The holistic sensory experience system can also include a preference analysis unit that analyzes user preference data and customizes the experience. For example, the preference analysis unit can analyze the user's preferred music, video, and haptic feedback patterns, and adjust the experience based on these preferences. It can also eliminate elements that the user wants to avoid. Furthermore, it can suggest new experiences based on the user's preference data. This allows for the provision of a customized experience tailored to the user's preferences.

[0070] The holistic sensory experience system can also include an eye-tracking unit that dynamically changes the focus of the visual experience using the user's eye-tracking data. For example, the eye-tracking unit can highlight an object when the user is looking at a specific object. It can also automatically focus on an object in the user's line of sight when the user moves their gaze. Furthermore, when the user's gaze is fixed, it can adjust the surrounding visual information around that viewpoint. This allows the system to change the focus of the visual experience according to the user's gaze, providing a more immersive experience.

[0071] The holistic sensory experience system can further include a temperature control unit that monitors the user's skin temperature and dynamically changes the temperature of the tactile feedback. For example, the temperature control unit can lower the temperature of the tactile feedback if the user's skin temperature is high, and raise it if the user's skin temperature is low. Furthermore, it can dynamically adjust the temperature of the tactile feedback based on fluctuations in the user's skin temperature. This allows for a more realistic tactile experience by adjusting the temperature of the tactile feedback based on the user's skin temperature.

[0072] The holistic sensory experience system can also include a food history reference unit that accesses the user's eating history to customize the simulation of smells and tastes. For example, the food history reference unit can simulate the smells and tastes of dishes the user has enjoyed eating in the past. It can also simulate the smells and tastes of specific ingredients based on the user's eating history. Furthermore, it can dynamically adjust the simulation content based on the user's eating history. This allows for the customization of smell and taste simulations based on the user's eating history, providing a more personalized experience.

[0073] The following briefly describes the processing flow for example form 1.

[0074] Step 1: The visual unit provides the visual experience. Using a high-resolution display and real-time rendering technology, the visual unit can realistically reproduce even the finest details of trees and plants, for example, when the user is walking through a virtual forest. Step 2: The audio unit provides an audio experience based on the visual experience provided by the visual unit. Using spatial audio technology, the audio unit can, for example, make the sounds of musical instruments and audience cheers sound three-dimensional if the user is in a virtual concert hall. Step 3: The haptic unit provides a haptic experience based on the acoustic experience provided by the acoustic unit. Using a haptic feedback device, the haptic unit can, for example, convey the weight and texture of a virtual stone to the user's hand when the user picks it up. Step 4: The simulation unit simulates smells and tastes based on the tactile experience provided by the haptic unit. For example, when a user drinks coffee in a virtual cafe, the simulation unit can perceive the aroma and taste. Step 5: The optimization unit optimizes the experience based on the experience provided by the simulation unit. For example, the optimization unit can optimize the experience based on user responses, and if the user is climbing a virtual mountain, it can adjust the experience according to their fatigue level and heart rate.

[0075] (Example of form 2) The total sensory experience system according to an embodiment of the present invention is a system that makes full use of sight, sound, and touch, utilizing virtual reality (VR) and augmented reality (AR) technologies and AI, while simulating smell and taste to a realistic extent. This total sensory experience system constantly optimizes the experience based on user responses and is customized to each individual user. First, the user puts on a VR or AR device and starts the experience. The visual experience is provided with clear and immersive images through a high-resolution display and real-time rendering technology. For example, if the user is walking through a virtual forest, the details of the trees and plants are realistically reproduced. Next, the acoustic experience realistically reproduces ambient sounds using spatial audio technology. For example, if the user is in a virtual concert hall, the sounds of instruments and the cheers of the audience are heard three-dimensionally, giving the user the feeling of being there. The tactile experience reproduces the feel of touching objects in the virtual space through a haptic feedback device. For example, if the user picks up a virtual stone, its weight and texture are transmitted to the hand. Furthermore, simulations of smell and taste are also performed. While currently limited to realistic simulations, this system allows users to experience the aroma and taste of coffee in a virtual cafe. This holistic sensory experience system is expected to have applications in various fields, including entertainment, education, healthcare, tourism, and rehabilitation. For example, it could be used in immersive games and virtual concerts in the entertainment industry, for recreating historical events and learning scientific concepts in education, and for telemedicine and rehabilitation in healthcare. The experience is constantly optimized based on user responses. AI recognizes user behavior and the surrounding environment, adjusting the virtual space in real time. For instance, if a user is climbing a virtual mountain, the experience is adjusted according to their fatigue level and heart rate, providing a more realistic experience. Thus, this holistic sensory experience system, utilizing virtual reality (VR) and augmented reality (AR) technologies along with AI, provides users with an immersive experience that fully engages all five senses—sight, sound, touch, smell, and taste—and is expected to have applications in various fields. This allows the holistic sensory experience system to optimize the experience based on user responses, providing a customized experience tailored to each individual user.

[0076] The total sensory experience system according to this embodiment comprises a visual unit, an acoustic unit, a haptic unit, a simulation unit, and an optimization unit. The visual unit provides a visual experience. The visual unit provides a visual experience using, for example, a high-resolution display and real-time rendering technology. For example, if the user is walking through a virtual forest, the visual unit can realistically reproduce the details of trees and plants. The acoustic unit provides an acoustic experience based on the visual experience provided by the visual unit. The acoustic unit provides an acoustic experience using, for example, spatial audio technology. For example, if the user is in a virtual concert hall, the acoustic unit can make the sounds of musical instruments and audience cheers sound three-dimensional. The haptic unit provides a haptic experience based on the acoustic experience provided by the acoustic unit. The haptic unit provides a haptic experience using, for example, a haptic feedback device. For example, if the user picks up a virtual stone, the haptic unit can convey its weight and texture to the user's hand. The simulation unit simulates smells and tastes based on the haptic experience provided by the haptic unit. For example, the simulation unit simulates smells and tastes within a realistic range. The simulation unit allows the user to experience the aroma and taste of coffee when drinking it in a virtual cafe. The optimization unit optimizes the experience based on the experience provided by the simulation unit. The optimization unit optimizes the experience based on the user's reactions, for example. If the user is climbing a virtual mountain, the optimization unit can adjust the experience according to their fatigue level and heart rate. As a result, the holistic sensory experience system according to this embodiment can provide an immersive experience that fully utilizes the five senses: sight, sound, touch, smell, and taste.

[0077] The visual unit provides a visual experience. For example, it uses high-resolution displays and real-time rendering technology to deliver a visual experience. Specifically, the visual unit uses eye-tracking technology to render visual information in real time according to the direction the user is looking. This allows the user to obtain a natural visual experience. Furthermore, the visual unit uses HDR (High Dynamic Range) technology to enhance the contrast between light and dark, providing more realistic images. For example, when a user is walking through a virtual forest, not only can the details of trees and plants be realistically reproduced, but the reflection of light and the movement of shadows are also expressed in real time. The visual unit can also provide three-dimensional images using 3D display technology. This allows the user to feel a sense of depth in the virtual space, resulting in a more immersive experience. The visual unit also dynamically updates visual information according to the user's movements, allowing the user to move freely within the virtual space. This enables the visual unit to provide the user with a highly realistic and dynamic visual experience.

[0078] The acoustic unit provides an acoustic experience based on the visual experience provided by the visual unit. For example, the acoustic unit uses spatial audio technology to deliver the acoustic experience. Specifically, the acoustic unit adjusts the direction and distance of sound in real time according to the user's position and movement. This allows the user to accurately perceive the location of the sound source and obtain a more three-dimensional acoustic experience. For example, if the user is in a virtual concert hall, the sounds of instruments and audience cheers can be heard in three dimensions. The acoustic unit also provides a more realistic acoustic environment by generating ambient sounds and sound effects in real time and synchronizing them with the visual experience. For example, when walking through a virtual forest, the sound of wind, birdsong, and rustling leaves are realistically reproduced. Furthermore, the acoustic unit can customize the acoustic experience according to the user's auditory characteristics. For example, it adjusts the acoustic profile based on the user's ear shape and hearing ability to provide an optimal acoustic experience. This allows the acoustic unit to provide the user with a highly realistic and personalized acoustic experience.

[0079] The haptic unit provides a tactile experience based on the acoustic experience provided by the acoustic unit. The haptic unit provides a tactile experience, for example, using a haptic feedback device. Specifically, the haptic unit provides tactile feedback such as vibration, pressure, and temperature through the user's device or wearable device. For example, when a user picks up a virtual stone, its weight and texture can be transmitted to the user's hand. The haptic unit also provides a more realistic tactile experience by adjusting the intensity and pattern of the haptic feedback in real time. For example, when walking through a virtual forest, the user can feel the texture of the ground under their feet and the rustling of the grass. Furthermore, the haptic unit dynamically updates the haptic feedback according to the user's movements and posture, allowing the user to feel their movements in the virtual space more realistically. In this way, the haptic unit can provide the user with a highly realistic and dynamic tactile experience.

[0080] The simulation unit simulates smells and tastes based on the tactile experience provided by the haptic unit. For example, the simulation unit simulates smells and tastes within a realistic range. Specifically, it generates specific smells and tastes through a device worn by the user. For instance, when a user drinks coffee in a virtual cafe, they can experience its aroma and taste. The simulation unit also provides a more realistic experience by adjusting the intensity and type of smells and tastes in real time. For example, when walking through a virtual forest, the user can experience the scent of flowers and the smell of damp earth. Furthermore, the simulation unit can customize the experience according to the user's olfactory and gustatory characteristics. For example, it adjusts the smell and taste profile based on the user's olfactory and gustatory sensitivity to provide an optimal experience. This allows the simulation unit to provide users with a highly realistic and personalized smell and taste experience.

[0081] The optimization unit optimizes the experience based on the experience provided by the simulation unit. For example, the optimization unit optimizes the experience based on user responses. Specifically, the optimization unit collects the user's biometric information and behavioral data in real time and dynamically adjusts the content and intensity of the experience. For example, if a user is climbing a virtual mountain, the experience can be adjusted according to their fatigue level and heart rate. The optimization unit can also customize the experience based on the user's preferences and past experience history. For example, if a user likes certain music or scenery, it will provide an experience accordingly. Furthermore, the optimization unit uses AI to analyze user responses and build a feedback loop to provide the optimal experience. This allows the optimization unit to always provide the user with the best possible experience and enhance immersion. The optimization unit can also individually optimize the experience for each user even when multiple users are sharing the experience simultaneously. This allows the optimization unit to provide users with a highly personalized experience and maximize the effectiveness of the whole-sensory experience system.

[0082] The recognition unit can recognize the user's actions and the surrounding environment. For example, the recognition unit can recognize the user's movement patterns and gestures. For example, the recognition unit can acquire the user's location information and recognize the surrounding environment. For example, the recognition unit can recognize the temperature, lighting, and acoustic environment. This allows the user experience to be adjusted based on the user's actions and environment. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input the user's movement patterns into the AI ​​and have the AI ​​perform the recognition of the movement patterns.

[0083] The adjustment unit can adjust the virtual space. For example, the adjustment unit can adjust 3D models and environment settings. For example, the adjustment unit can adjust interaction elements. This allows the virtual space to be adjusted in real time. Some or all of the above-described processes in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input a 3D model of the virtual space into the AI ​​and have the AI ​​perform adjustments to the 3D model.

[0084] The visual unit can provide a visual experience using a high-resolution display and real-time rendering technology. For example, the visual unit can provide clear images using a high-resolution display. For example, the visual unit can generate images in real time using real-time rendering technology. For example, if the user is walking through a virtual forest, the visual unit can realistically reproduce even the smallest details of trees and plants. This enables the provision of a high-resolution, real-time visual experience. Some or all of the above-described processes in the visual unit may be performed using AI or not. For example, the visual unit can have AI perform the generation of images.

[0085] The audio unit can provide an audio experience using spatial audio technology. For example, the audio unit can calculate the location of sound sources to provide a three-dimensional audio experience. For example, the audio unit can enhance the realism of sound using sound effects. For example, if the user is in a virtual concert hall, the audio unit can make the sounds of musical instruments and audience cheers sound three-dimensional. This provides a three-dimensional audio experience. Some or all of the above processing in the audio unit may be performed using AI or not. For example, the audio unit can have AI perform the calculation of the location of sound sources.

[0086] The haptic unit can provide a tactile experience using haptic feedback devices. For example, the haptic unit can provide haptic feedback using a vibration device. For example, the haptic unit can provide haptic feedback using a pressure device. For example, the haptic unit can provide haptic feedback using a temperature device. For example, when a user picks up a virtual stone, the haptic unit can convey its weight and texture to the user's hand. This can provide a realistic tactile experience. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can have AI perform the control of haptic feedback.

[0087] The simulation unit can simulate smells and tastes within a realistic range. For example, the simulation unit can adjust the intensity and duration of smells. For example, the simulation unit can adjust the intensity and duration of tastes. For example, when a user drinks coffee in a virtual cafe, the simulation unit can experience its aroma and taste. This allows for the provision of smell and taste simulations. Some or all of the above-described processes in the simulation unit may be performed using AI or not. For example, the simulation unit can have AI perform smell and taste simulations.

[0088] The optimization unit can optimize the experience based on user responses. For example, the optimization unit can recognize the user's facial expressions and movements and adjust the experience. For example, the optimization unit can monitor the user's vital signs and adjust the experience. For example, if the user is climbing a virtual mountain, the optimization unit can adjust the experience according to the user's fatigue level and heart rate. This allows the experience to be optimized based on user responses. Some or all of the above-described processes in the optimization unit may be performed using AI or not. For example, the optimization unit can input the user's vital signs into the AI ​​and have the AI ​​perform the optimization of the experience.

[0089] The visual unit can estimate the user's emotions and adjust the color tone and brightness of the visual experience based on the estimated emotions. For example, if the user is relaxed, the visual unit provides warm colors and soft brightness. For example, if the user is excited, the visual unit provides vibrant colors and high brightness. For example, if the user is tired, the visual unit provides calm colors and low brightness. This allows the visual experience to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input user emotion data into AI and have the AI ​​perform the adjustment of the visual experience.

[0090] The visual unit can dynamically change the focus of the visual experience using the user's eye-tracking data. For example, if the user is looking at a specific object, the visual unit will highlight that object. For example, if the user moves their gaze, the visual unit will automatically focus on the object in their line of sight. For example, if the user's gaze is fixed, the visual unit will adjust the surrounding visual information around that viewpoint. This allows the focus of the visual experience to change according to the user's gaze. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input eye-tracking data into AI and have the AI ​​perform the change in the focus of the visual experience.

[0091] The visual unit can monitor the user's pupillary response during the visual experience and adjust the intensity of the visual stimulus. For example, if the user's pupils dilate, the visual unit may decrease the intensity of the visual stimulus. For example, if the user's pupils constrict, the visual unit may increase the intensity of the visual stimulus. For example, if the user's pupillary response is not constant, the visual unit may dynamically adjust the intensity of the visual stimulus. This allows the intensity of the visual stimulus to be adjusted based on the user's pupillary response. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input pupillary response data into the AI ​​and have the AI ​​perform the adjustment of the intensity of the visual stimulus.

[0092] The visual unit can estimate the user's emotions and switch scenes in the visual experience based on the estimated emotions. For example, if the user is relaxed, the visual unit will switch to a calm scene. For example, if the user is excited, the visual unit will switch to an active scene. For example, if the user is tired, the visual unit will switch to a quiet scene. This allows the visual experience to switch scenes according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input user emotion data into an AI and have the AI ​​perform the scene switching.

[0093] The visual unit can detect the user's head movements during the visual experience and change the viewpoint of the visual content in real time. For example, if the user moves their head to the left, the visual unit will move the viewpoint to the left. For example, if the user moves their head up and down, the visual unit will move the viewpoint up and down. For example, if the user rotates their head, the visual unit will rotate the viewpoint. This allows the viewpoint of the visual content to be changed in response to the user's head movements. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input head movement data into the AI ​​and have the AI ​​perform the viewpoint change.

[0094] The visual unit can analyze the user's blinking frequency during the visual experience and adjust the display speed of the visual content. For example, if the user blinks frequently, the visual unit will slow down the display speed. For example, if the user blinks infrequently, the visual unit will speed up the display speed. For example, the visual unit can dynamically adjust the display speed based on the user's blinking pattern. This allows the display speed of the visual content to be adjusted based on the user's blinking frequency. Some or all of the above processing in the visual unit may be performed using AI or not. For example, the visual unit can input blinking frequency data into the AI ​​and have the AI ​​perform the adjustment of the display speed.

[0095] The acoustic unit can estimate the user's emotions and adjust the volume and sound quality of the audio experience based on the estimated emotions. For example, if the user is relaxed, the acoustic unit will lower the volume and soften the sound quality. If the user is excited, for example, the acoustic unit will increase the volume and sharpen the sound quality. If the user is tired, for example, the acoustic unit will moderate the volume and calm the sound quality. This allows the volume and sound quality of the audio experience to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the acoustic unit may be performed using AI or not. For example, the acoustic unit can input user emotion data into an AI and have the AI ​​perform volume and sound quality adjustments.

[0096] The acoustic unit can track the user's ear position during the audio experience and dynamically change the sound source position. For example, if the user moves their head to the left, the acoustic unit will move the sound source to the left. For example, if the user moves their head up and down, the acoustic unit will move the sound source up and down. For example, if the user rotates their head, the acoustic unit will rotate the sound source. This allows the sound source position to be changed according to the user's ear position. Some or all of the above processing in the acoustic unit may be performed using AI or not. For example, the acoustic unit can input ear position data into the AI ​​and have the AI ​​perform the sound source position change.

[0097] The acoustic unit can monitor the user's heart rate during the acoustic experience and adjust the rhythm of the acoustic stimulation. For example, the acoustic unit can speed up the rhythm if the user's heart rate is high, or slow down if the user's heart rate is low. The acoustic unit can also dynamically adjust the rhythm based on fluctuations in the user's heart rate. This allows the rhythm of the acoustic stimulation to be adjusted based on the user's heart rate. Some or all of the above processing in the acoustic unit may be performed using AI or not. For example, the acoustic unit can input heart rate data into the AI ​​and have the AI ​​perform the rhythm adjustment.

[0098] The sound unit can estimate the user's emotions and change the music genre of the sound experience based on the estimated emotions. For example, if the user is relaxed, the sound unit can provide classical music. If the user is excited, the sound unit can provide rock music. If the user is tired, the sound unit can provide ambient music. This allows the music genre to be changed according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the sound unit may be performed using AI or not. For example, the sound unit can input user emotion data into an AI and have the AI ​​change the music genre.

[0099] The audio unit can detect the user's breathing pattern during the audio experience and adjust the tempo of the audio content. For example, if the user's breathing is fast, the audio unit will speed up the tempo. For example, if the user's breathing is slow, the audio unit will slow down the tempo. For example, the audio unit can dynamically adjust the tempo based on the user's breathing pattern. This allows the tempo of the audio content to be adjusted based on the user's breathing pattern. Some or all of the above processing in the audio unit may be performed using AI or not. For example, the audio unit can input breathing pattern data into the AI ​​and have the AI ​​perform the tempo adjustment.

[0100] The acoustics unit can analyze the user's voice tone during the audio experience and provide acoustic feedback. For example, if the user's voice is high-pitched, the acoustics unit provides bright acoustic feedback. For example, if the user's voice is low-pitched, the acoustics unit provides calm acoustic feedback. The acoustics unit can dynamically adjust the acoustic feedback based on the user's voice tone, for example. This allows the acoustics unit to provide acoustic feedback based on the user's voice tone. Some or all of the above processing in the acoustics unit may be performed using AI or not. For example, the acoustics unit can input voice tone data into an AI and have the AI ​​perform the task of providing acoustic feedback.

[0101] The haptic unit can estimate the user's emotions and adjust the intensity of haptic feedback based on the estimated emotions. For example, if the user is relaxed, the haptic unit will lower the intensity of the haptic feedback. If the user is excited, for example, the haptic unit will increase the intensity of the haptic feedback. If the user is tired, for example, the haptic unit will set the intensity of the haptic feedback to a moderate level. This allows the intensity of haptic feedback to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input user emotion data into the AI ​​and have the AI ​​adjust the intensity of the haptic feedback.

[0102] The haptic unit can monitor the user's skin temperature during the haptic experience and dynamically change the temperature of the haptic stimulus. For example, if the user's skin temperature is high, the haptic unit will lower the temperature of the haptic stimulus. For example, if the user's skin temperature is low, the haptic unit will raise the temperature of the haptic stimulus. For example, the haptic unit can dynamically adjust the temperature of the haptic stimulus based on fluctuations in the user's skin temperature. This allows the temperature of the haptic stimulus to be adjusted based on the user's skin temperature. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input skin temperature data into the AI ​​and have the AI ​​perform the adjustment of the haptic stimulus temperature.

[0103] The haptic unit can detect the user's muscle responses during the haptic experience and adjust the haptic feedback pattern. For example, if the user's muscles are tense, the haptic unit will soften the haptic feedback pattern. For example, if the user's muscles are relaxed, the haptic unit will strengthen the haptic feedback pattern. The haptic unit can dynamically adjust the haptic feedback pattern based on the user's muscle responses. This allows the haptic feedback pattern to be adjusted based on the user's muscle responses. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input muscle response data into the AI ​​and have the AI ​​perform the adjustment of the haptic feedback pattern.

[0104] The haptic unit can estimate the user's emotions and change the type of haptic feedback based on the estimated emotions. For example, if the user is relaxed, the haptic unit provides soft haptic feedback. For example, if the user is excited, the haptic unit provides strong haptic feedback. For example, if the user is tired, the haptic unit provides gentle haptic feedback. This allows the type of haptic feedback to be changed according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input user emotion data into the AI ​​and have the AI ​​change the type of haptic feedback.

[0105] The haptic unit can track the user's hand movements during the haptic experience and dynamically change the position of the haptic feedback. For example, if the user moves their hand to the left, the haptic unit will move the position of the haptic feedback to the left. For example, if the user moves their hand up and down, the haptic unit will move the position of the haptic feedback up and down. For example, if the user rotates their hand, the haptic unit will rotate the position of the haptic feedback. This allows the position of the haptic feedback to change according to the user's hand movements. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input hand movement data into the AI ​​and have the AI ​​perform the change in the position of the haptic feedback.

[0106] The haptic unit can analyze the user's pressure sensation during the haptic experience and adjust the pressure of the haptic feedback. For example, if the user feels strong pressure, the haptic unit will lower the pressure of the haptic feedback. For example, if the user feels weak pressure, the haptic unit will increase the pressure of the haptic feedback. The haptic unit can dynamically adjust the pressure of the haptic feedback based on the user's pressure sensation. This allows the pressure of the haptic feedback to be adjusted based on the user's pressure sensation. Some or all of the above processing in the haptic unit may be performed using AI or not. For example, the haptic unit can input pressure sensation data into the AI ​​and have the AI ​​perform the adjustment of the haptic feedback pressure.

[0107] The simulation unit can estimate the user's emotions and adjust the simulation intensity of smells and tastes based on the estimated emotions. For example, if the user is relaxed, the simulation unit will lower the simulation intensity of smells and tastes. For example, if the user is excited, the simulation unit will increase the simulation intensity of smells and tastes. For example, if the user is tired, the simulation unit will set the simulation intensity of smells and tastes to a moderate level. This allows the simulation intensity of smells and tastes to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the simulation unit may be performed using AI or not. For example, the simulation unit can input user emotion data into an AI and have the AI ​​adjust the simulation intensity of smells and tastes.

[0108] The simulation unit can monitor the user's olfactory response during smell and taste simulations and dynamically change the simulation content. For example, if the user's olfactory response is strong, the simulation unit may weaken the simulation content. For example, if the user's olfactory response is weak, the simulation unit may strengthen the simulation content. For example, the simulation unit may dynamically adjust the simulation content based on the user's olfactory response. This allows the simulation content to be adjusted based on the user's olfactory response. Some or all of the above processing in the simulation unit may be performed using AI or not. For example, the simulation unit can input olfactory response data into the AI ​​and have the AI ​​execute changes to the simulation content.

[0109] The simulation unit can detect the user's saliva secretion during smell and taste simulations to improve the realism of the simulation. For example, the simulation unit increases the realism of the simulation if the user's saliva secretion is high. For example, the simulation unit decreases the realism of the simulation if the user's saliva secretion is low. For example, the simulation unit dynamically adjusts the realism of the simulation based on the user's saliva secretion. This allows the realism of the simulation to be adjusted based on the user's saliva secretion. Some or all of the above processing in the simulation unit may be performed using AI or not. For example, the simulation unit can input saliva secretion data into the AI ​​and have the AI ​​perform the adjustment of the simulation realism.

[0110] The simulation unit can estimate the user's emotions and modify the simulated scents and tastes based on the estimated emotions. For example, if the user is relaxed, the simulation unit provides calming scents and tastes. If the user is excited, the simulation unit provides stimulating scents and tastes. If the user is tired, the simulation unit provides calming scents and tastes. This allows the simulated scents and tastes to be modified according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processes in the simulation unit may be performed using AI or not. For example, the simulation unit can input user emotion data into an AI and have the AI ​​modify the simulated scents and tastes.

[0111] The simulation unit can refer to the user's eating history during the simulation of smells and tastes and customize the simulation content. For example, the simulation unit can simulate the smells and tastes of dishes that the user has enjoyed eating in the past. For example, the simulation unit can simulate the smells and tastes of specific ingredients from the user's eating history. For example, the simulation unit can dynamically adjust the simulation content based on the user's eating history. This allows the simulation content to be customized based on the user's eating history. Some or all of the above processes in the simulation unit may be performed using AI or not. For example, the simulation unit can input eating history data into AI and have the AI ​​perform the customization of the simulation content.

[0112] The simulation unit can analyze user preference data during smell and taste simulations and adjust the type of simulation. For example, the simulation unit simulates preferred smells and tastes based on the user preference data. For example, the simulation unit emphasizes specific flavors based on the user preference data. For example, the simulation unit dynamically adjusts the type of simulation based on the user preference data. This allows the type of simulation to be adjusted based on the user preference data. Some or all of the above processing in the simulation unit may be performed using AI or not. For example, the simulation unit can input preference data into AI and have the AI ​​perform the adjustment of the type of simulation.

[0113] The optimization unit can estimate the user's emotions and adjust the experience optimization algorithm based on the estimated user emotions. For example, if the user is relaxed, the optimization unit will soften the experience optimization algorithm. For example, if the user is excited, the optimization unit will strengthen the experience optimization algorithm. For example, if the user is tired, the optimization unit will moderate the experience optimization algorithm. This allows the experience optimization algorithm to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the optimization unit may be performed using AI or not. For example, the optimization unit can input user emotion data into an AI and have the AI ​​perform the adjustment of the optimization algorithm.

[0114] The optimization unit can analyze the user's biometric data in real time during optimization and dynamically change the experience content. For example, if the user's heart rate is high, the optimization unit will slow down the experience content. For example, if the user's heart rate is low, the optimization unit will intensify the experience content. For example, the optimization unit will dynamically adjust the experience content based on the user's biometric data. This allows the experience content to be dynamically changed based on the user's biometric data. Some or all of the above processing in the optimization unit may be performed using AI or not. For example, the optimization unit can input biometric data into AI and have the AI ​​perform the changes to the experience content.

[0115] The optimization unit can refer to the user's past experience history during optimization to improve the accuracy of the optimization. For example, the optimization unit provides the optimal experience content based on the user's past experience history. For example, the optimization unit extracts specific patterns from the user's past experience history to improve the accuracy of the optimization. For example, the optimization unit analyzes the user's past experience history and improves the optimization algorithm. This allows the accuracy of the optimization to be improved based on the user's past experience history. Some or all of the above processes in the optimization unit may be performed using AI or not. For example, the optimization unit can input past experience history data into AI and have the AI ​​perform the optimization accuracy improvement.

[0116] The optimization unit can estimate the user's emotions and adjust the optimization frequency of the experience based on the estimated user emotions. For example, the optimization unit may lower the optimization frequency if the user is relaxed, increase the optimization frequency if the user is excited, or set the optimization frequency to a moderate level if the user is tired. This allows the optimization frequency of the experience to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the optimization unit may be performed using AI or not. For example, the optimization unit may input user emotion data into an AI and have the AI ​​adjust the optimization frequency.

[0117] The optimization unit can refer to the user's environmental data during optimization and customize the user experience. For example, if the temperature around the user is high, the optimization unit can make the user experience cooler. For example, if the temperature around the user is low, the optimization unit can make the user experience warmer. For example, the optimization unit can dynamically adjust the user experience based on the user's environmental data. This allows the user experience to be customized based on the user's environmental data. Some or all of the above processing in the optimization unit may be performed using AI or not. For example, the optimization unit can input environmental data into the AI ​​and have the AI ​​perform the customization of the user experience.

[0118] The optimization unit can collect user feedback during optimization and improve the optimization algorithm. For example, the optimization unit adjusts the optimization algorithm based on user feedback. For example, the optimization unit extracts specific patterns from user feedback and improves the optimization algorithm. For example, the optimization unit analyzes user feedback and improves the accuracy of the optimization algorithm. This allows the optimization algorithm to be improved based on user feedback. Some or all of the above processes in the optimization unit may be performed using AI or not. For example, the optimization unit can input feedback data into AI and have the AI ​​perform improvements to the optimization algorithm.

[0119] The recognition unit can estimate the user's emotions and adjust the recognition algorithm based on the estimated emotions. For example, if the user is relaxed, the recognition unit will soften the recognition algorithm. For example, if the user is excited, the recognition unit will strengthen the recognition algorithm. For example, if the user is tired, the recognition unit will moderate the recognition algorithm. This allows the recognition algorithm to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input user emotion data into an AI and have the AI ​​perform the adjustment of the recognition algorithm.

[0120] The recognition unit can analyze the user's behavior patterns in real time during recognition and improve recognition accuracy. For example, the recognition unit improves recognition accuracy based on the user's behavior patterns. For example, the recognition unit analyzes the user's behavior patterns in real time and adjusts the recognition algorithm. For example, the recognition unit analyzes the user's behavior patterns and dynamically adjusts the recognition accuracy. This allows for improvement of recognition accuracy based on the user's behavior patterns. Some or all of the above processes in the recognition unit may be performed using AI or not. For example, the recognition unit can input behavior pattern data into the AI ​​and have the AI ​​perform the improvement of recognition accuracy.

[0121] The recognition unit can estimate the user's emotions and determine recognition priorities based on the estimated emotions. For example, if the user is relaxed, the recognition unit will lower the recognition priority. For example, if the user is excited, the recognition unit will raise the recognition priority. For example, if the user is tired, the recognition unit will set the recognition priority to a medium level. In this way, the recognition priority can be determined according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input user emotion data into an AI and have the AI ​​perform the determination of recognition priorities.

[0122] The recognition unit can refer to the user's environmental data during recognition and customize the recognized content. For example, if the temperature around the user is high, the recognition unit will make the recognized content cooler. For example, if the temperature around the user is low, the recognition unit will make the recognized content warmer. For example, the recognition unit will dynamically adjust the recognized content based on the user's environmental data. This allows the recognized content to be customized based on the user's environmental data. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input environmental data into the AI ​​and have the AI ​​perform the customization of the recognized content.

[0123] The adjustment unit can estimate the user's emotions and adjust the adjustment algorithm based on the estimated user emotions. For example, if the user is relaxed, the adjustment unit will soften the adjustment algorithm. For example, if the user is excited, the adjustment unit will strengthen the adjustment algorithm. For example, if the user is tired, the adjustment unit will moderate the adjustment algorithm. In this way, the adjustment algorithm can be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input user emotion data into an AI and have the AI ​​perform the adjustment of the adjustment algorithm.

[0124] The adjustment unit can analyze the user's biometric data in real time during adjustment and dynamically change the adjustment settings. For example, if the user's heart rate is high, the adjustment unit will lessen the adjustment settings. For example, if the user's heart rate is low, the adjustment unit will strengthen the adjustment settings. For example, the adjustment unit will dynamically adjust the adjustment settings based on the user's biometric data. This allows the adjustment settings to be dynamically changed based on the user's biometric data. Some or all of the above processing in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input biometric data into AI and have AI perform the changes to the adjustment settings.

[0125] The adjustment unit can estimate the user's emotions and adjust the frequency of adjustments based on the estimated emotions. For example, the adjustment unit may lower the adjustment frequency if the user is relaxed, increase the adjustment frequency if the user is excited, or set the adjustment frequency to a moderate level if the user is tired. This allows the adjustment frequency to be adjusted according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processes in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input user emotion data into an AI and have the AI ​​perform the adjustment of the adjustment frequency.

[0126] The adjustment unit can refer to the user's environmental data during adjustment and customize the adjustment settings. For example, if the temperature around the user is high, the adjustment unit will make the settings cooler. For example, if the temperature around the user is low, the adjustment unit will make the settings warmer. For example, the adjustment unit will dynamically adjust the settings based on the user's environmental data. This allows the adjustment settings to be customized based on the user's environmental data. Some or all of the above processing in the adjustment unit may be performed using AI or not. For example, the adjustment unit can input environmental data into the AI ​​and have the AI ​​perform the customization of the adjustment settings.

[0127] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0128] The holistic sensory experience system can also include a biometric data monitoring unit that monitors the user's biometric data in real time. This unit can acquire data such as the user's heart rate, blood pressure, and body temperature, and adjust the experience based on this data. For example, if the user's heart rate increases, the system can reduce the intensity of the experience. If the user's body temperature is high, the system can lower the temperature in the virtual space. Furthermore, if the user's blood pressure drops, the system can temporarily suspend the experience and prompt the user to rest. This allows for dynamic adjustment of the experience based on the user's biometric data, providing a safer and more comfortable experience.

[0129] The holistic sensory experience system can further include an emotion adaptation unit that estimates the user's emotions and dynamically changes the content of the experience based on those emotions. For example, if the user is feeling stressed, the emotion adaptation unit can switch to a relaxing experience. If the user is excited, it can change to a more stimulating experience. Furthermore, if the user is sad, it can provide a comforting experience. This allows for flexible changes to the experience content in response to the user's emotions, providing a more personalized experience.

[0130] The holistic sensory experience system can also include a history reference unit that customizes the experience by referring to the user's past experience history. For example, the history reference unit can provide similar experiences based on what the user has enjoyed in the past. It can also suggest different experiences based on what the user has avoided in the past. Furthermore, it can extract specific patterns from the user's past experience history and optimize the experience. This allows for the provision of a more personalized experience based on the user's past experience history.

[0131] The holistic sensory experience system can also include a preference analysis unit that analyzes user preference data and customizes the experience. For example, the preference analysis unit can analyze the user's preferred music, video, and haptic feedback patterns, and adjust the experience based on these preferences. It can also eliminate elements that the user wants to avoid. Furthermore, it can suggest new experiences based on the user's preference data. This allows for the provision of a customized experience tailored to the user's preferences.

[0132] The holistic sensory experience system can further include a music adaptation unit that estimates the user's emotions and changes the music genre of the sound experience based on those emotions. For example, the music adaptation unit can provide classical music when the user is relaxed, rock music when the user is excited, and ambient music when the user is tired. This allows the system to change the music genre according to the user's emotions, providing a more appropriate sound experience.

[0133] The holistic sensory experience system can also include an eye-tracking unit that dynamically changes the focus of the visual experience using the user's eye-tracking data. For example, the eye-tracking unit can highlight an object when the user is looking at a specific object. It can also automatically focus on an object in the user's line of sight when the user moves their gaze. Furthermore, when the user's gaze is fixed, it can adjust the surrounding visual information around that viewpoint. This allows the system to change the focus of the visual experience according to the user's gaze, providing a more immersive experience.

[0134] The holistic sensory experience system may further include a haptic adaptation unit that estimates the user's emotions and adjusts the intensity of haptic feedback based on those emotions. For example, the haptic adaptation unit can lower the intensity of haptic feedback when the user is relaxed, increase it when the user is excited, and set it to a moderate level when the user is tired. This allows the system to adjust the intensity of haptic feedback according to the user's emotions, providing a more appropriate haptic experience.

[0135] The holistic sensory experience system can further include a temperature control unit that monitors the user's skin temperature and dynamically changes the temperature of the tactile feedback. For example, the temperature control unit can lower the temperature of the tactile feedback if the user's skin temperature is high, and raise it if the user's skin temperature is low. Furthermore, it can dynamically adjust the temperature of the tactile feedback based on fluctuations in the user's skin temperature. This allows for a more realistic tactile experience by adjusting the temperature of the tactile feedback based on the user's skin temperature.

[0136] The holistic sensory experience system can further include an emotion-adaptive simulation unit that estimates the user's emotions and modifies the simulated scents and tastes based on those emotions. For example, the emotion-adaptive simulation unit can provide calming scents and tastes when the user is relaxed, stimulating scents and tastes when the user is excited, and calming scents and tastes when the user is tired. This allows the system to change the simulated scents and tastes according to the user's emotions, providing a more appropriate experience.

[0137] The holistic sensory experience system can also include a food history reference unit that accesses the user's eating history to customize the simulation of smells and tastes. For example, the food history reference unit can simulate the smells and tastes of dishes the user has enjoyed eating in the past. It can also simulate the smells and tastes of specific ingredients based on the user's eating history. Furthermore, it can dynamically adjust the simulation content based on the user's eating history. This allows for the customization of smell and taste simulations based on the user's eating history, providing a more personalized experience.

[0138] The following briefly describes the processing flow for example form 2.

[0139] Step 1: The visual unit provides the visual experience. Using a high-resolution display and real-time rendering technology, the visual unit can realistically reproduce even the finest details of trees and plants, for example, when the user is walking through a virtual forest. Step 2: The audio unit provides an audio experience based on the visual experience provided by the visual unit. Using spatial audio technology, the audio unit can, for example, make the sounds of musical instruments and audience cheers sound three-dimensional if the user is in a virtual concert hall. Step 3: The haptic unit provides a haptic experience based on the acoustic experience provided by the acoustic unit. Using a haptic feedback device, the haptic unit can, for example, convey the weight and texture of a virtual stone to the user's hand when the user picks it up. Step 4: The simulation unit simulates smells and tastes based on the tactile experience provided by the haptic unit. For example, when a user drinks coffee in a virtual cafe, the simulation unit can perceive the aroma and taste. Step 5: The optimization unit optimizes the experience based on the experience provided by the simulation unit. For example, the optimization unit can optimize the experience based on user responses, and if the user is climbing a virtual mountain, it can adjust the experience according to their fatigue level and heart rate.

[0140] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0141] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0142] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0143] Each of the multiple elements described above, including the visual, acoustic, haptic, simulation, and optimization units, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the visual unit provides a visual experience using the high-resolution display and real-time rendering technology of the smart device 14. The acoustic unit provides an acoustic experience using the spatial audio technology of the smart device 14. The haptic unit provides a haptic experience using the haptic feedback device of the smart device 14. The simulation unit simulates smells and tastes using the specific processing unit 290 of the data processing unit 12. The optimization unit optimizes the experience based on user responses using the specific processing unit 290 of the data processing unit 12. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0144] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0145] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0146] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0147] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0148] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0149] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0150] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0151] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0152] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0153] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0154] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0155] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0156] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0157] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0158] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0159] Each of the multiple elements described above, including the visual, acoustic, haptic, simulation, and optimization units, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the visual unit provides a visual experience using the high-resolution display and real-time rendering technology of the smart glasses 214. The acoustic unit provides an acoustic experience using the spatial audio technology of the smart glasses 214. The haptic unit provides a haptic experience using the haptic feedback device of the smart glasses 214. The simulation unit simulates smells and tastes using the specific processing unit 290 of the data processing unit 12. The optimization unit optimizes the experience based on user responses using the specific processing unit 290 of the data processing unit 12. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0160] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0161] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0162] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0163] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0164] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0165] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0166] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0167] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0168] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0169] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0170] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0171] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0172] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0173] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0174] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0175] Each of the multiple elements described above, including the visual, acoustic, tactile, simulation, and optimization units, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the visual unit provides a visual experience using the high-resolution display and real-time rendering technology of the headset terminal 314. The acoustic unit provides an acoustic experience using the spatial audio technology of the headset terminal 314. The tactile unit provides a tactile experience using the tactile feedback device of the headset terminal 314. The simulation unit simulates smells and tastes using the specific processing unit 290 of the data processing unit 12. The optimization unit optimizes the experience based on user responses using the specific processing unit 290 of the data processing unit 12. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0176] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0177] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0178] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0179] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0180] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0181] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0182] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0183] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0184] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0185] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0186] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0187] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0188] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0189] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0190] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0191] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0192] Each of the multiple elements described above, including the visual, acoustic, tactile, simulation, and optimization units, is implemented in at least one of the robot 414 and the data processing unit 12. For example, the visual unit provides a visual experience using the robot 414's high-resolution display and real-time rendering technology. The acoustic unit provides an acoustic experience using the robot 414's spatial audio technology. The tactile unit provides a tactile experience using the robot 414's tactile feedback device. The simulation unit simulates smells and tastes using the specific processing unit 290 of the data processing unit 12. The optimization unit optimizes the experience based on user responses using the specific processing unit 290 of the data processing unit 12. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0193] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0194] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0195] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0196] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0197] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0198] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0199] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0200] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0201] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0202] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0203] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0204] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0205] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0206] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0207] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0208] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0209] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0210] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0211] (Note 1) The Visual Department provides a visual experience, A sound unit that provides an acoustic experience based on the visual experience provided by the aforementioned visual unit, A tactile unit that provides a tactile experience based on the acoustic experience provided by the aforementioned acoustic unit, A simulation unit that simulates smells and tastes based on the tactile experience provided by the aforementioned tactile unit, The system includes an optimization unit that optimizes the experience based on the experience provided by the simulation unit. A system characterized by the following features. (Note 2) It is equipped with a recognition unit that recognizes the user's actions and the surrounding environment. The system described in Appendix 1, characterized by the features described herein. (Note 3) It is equipped with an adjustment unit for adjusting the virtual space. The system described in Appendix 1, characterized by the features described herein. (Note 4) The aforementioned visual unit is Providing a visual experience using high-resolution displays and real-time rendering technology. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned acoustic unit is Providing an audio experience using spatial audio technology. The system described in Appendix 1, characterized by the features described herein. (Note 6) The aforementioned tactile part is, Providing a haptic experience using haptic feedback devices. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned simulation unit, To simulate smells and tastes within a realistic range. The system described in Appendix 1, characterized by the features described herein. (Note 8) The optimization unit, Optimize the experience based on user feedback. The system described in Appendix 1, characterized by the features described herein. (Note 9) The aforementioned visual unit is It estimates the user's emotions and adjusts the color tone and brightness of the visual experience based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 10) The aforementioned visual unit is Dynamically change the focus of the visual experience using user eye-tracking data. The system described in Appendix 1, characterized by the features described herein. (Note 11) The aforementioned visual unit is The system monitors the user's pupillary response during the visual experience and adjusts the intensity of the visual stimuli accordingly. The system described in Appendix 1, characterized by the features described herein. (Note 12) The aforementioned visual unit is It estimates the user's emotions and switches the visual experience scene based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 13) The aforementioned visual unit is Detects the user's head movements during the visual experience and changes the viewpoint of the visual content in real time. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned visual unit is Analyze the user's blinking frequency during the visual experience and adjust the display speed of the visual content. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned acoustic unit is It estimates the user's emotions and adjusts the volume and sound quality of the audio experience based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned acoustic unit is The system tracks the user's ear position during the audio experience and dynamically changes the sound source's location. The system described in Appendix 1, characterized by the features described herein. (Note 17) The aforementioned acoustic unit is The system monitors the user's heart rate during the audio experience and adjusts the rhythm of the acoustic stimulation accordingly. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned acoustic unit is It estimates the user's emotions and changes the music genre of the sound experience based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 19) The aforementioned acoustic unit is The system detects the user's breathing patterns during the audio experience and adjusts the tempo of the audio content accordingly. The system described in Appendix 1, characterized by the features described herein. (Note 20) The aforementioned acoustic unit is It analyzes the user's voice tone during the audio experience and provides acoustic feedback. The system described in Appendix 1, characterized by the features described herein. (Note 21) The aforementioned tactile part is, It estimates the user's emotions and adjusts the intensity of haptic feedback based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 22) The aforementioned tactile part is, The system monitors the user's skin temperature during the haptic experience and dynamically changes the temperature of the haptic stimuli. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned tactile part is, It detects the user's muscle responses during the haptic experience and adjusts the pattern of haptic feedback. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned tactile part is, It estimates the user's emotions and changes the type of haptic feedback based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned tactile part is, Track the user's hand movements during the haptic experience and dynamically change the position of the haptic feedback. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned tactile part is, The system analyzes the user's pressure sensations during the haptic experience and adjusts the pressure of the haptic feedback. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned simulation unit, It estimates the user's emotions and adjusts the intensity of the scent and taste simulations based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned simulation unit, The system monitors the user's olfactory responses during smell and taste simulations and dynamically modifies the simulation content. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned simulation unit, The system detects the user's saliva production during smell and taste simulations to improve the realism of the simulations. The system described in Appendix 1, characterized by the features described herein. (Note 30) The aforementioned simulation unit, The system estimates the user's emotions and modifies the simulation of smells and tastes based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 31) The aforementioned simulation unit, During smell and taste simulations, the system references the user's eating history to customize the simulation content. The system described in Appendix 1, characterized by the features described herein. (Note 32) The aforementioned simulation unit, During the simulation of smells and tastes, user preference data is analyzed, and the type of simulation is adjusted accordingly. The system described in Appendix 1, characterized by the features described herein. (Note 33) The optimization unit, It estimates the user's emotions and adjusts the experience optimization algorithm based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 34) The optimization unit, Analyze user biometric data in real time during optimization and dynamically change the user experience. The system described in Appendix 1, characterized by the features described herein. (Note 35) The optimization unit, During optimization, we refer to the user's past experience history to improve the accuracy of the optimization. The system described in Appendix 1, characterized by the features described herein. (Note 36) The optimization unit, It estimates the user's emotions and adjusts the frequency of experience optimization based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 37) The optimization unit, During optimization, the system references user environment data to customize the user experience. The system described in Appendix 1, characterized by the features described herein. (Note 38) The optimization unit, We collect user feedback during optimization and improve the optimization algorithm. The system described in Appendix 1, characterized by the features described herein. (Note 39) The recognition unit, It estimates the user's emotions and adjusts the recognition algorithm based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 40) The recognition unit, The system analyzes user behavior patterns in real time during recognition to improve recognition accuracy. The system described in Appendix 1, characterized by the features described herein. (Note 41) The recognition unit, It estimates the user's emotions and determines recognition priorities based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 42) The recognition unit, During recognition, the system references the user's environment data to customize the recognition process. The system described in Appendix 1, characterized by the features described herein. (Note 43) The adjustment unit is, The system estimates the user's emotions and adjusts the adjustment algorithm based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 44) The adjustment unit is, The system analyzes the user's biometric data in real time during adjustment and dynamically modifies the adjustment settings. The system described in Appendix 1, characterized by the features described herein. (Note 45) The adjustment unit is, It estimates the user's emotions and adjusts the frequency of adjustments based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 46) The adjustment unit is, During the adjustment process, the system references the user's environment data and customizes the adjustment settings. The system described in Appendix 1, characterized by the features described herein. [Explanation of symbols]

[0212] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. The Visual Department provides a visual experience, A sound unit that provides an acoustic experience based on the visual experience provided by the aforementioned visual unit, A tactile unit that provides a tactile experience based on the acoustic experience provided by the aforementioned acoustic unit, A simulation unit that simulates smells and tastes based on the tactile experience provided by the aforementioned tactile unit, The system includes an optimization unit that optimizes the experience based on the experience provided by the simulation unit. A system characterized by the following features.

2. It is equipped with a recognition unit that recognizes the user's actions and the surrounding environment. The system according to feature 1.

3. It is equipped with an adjustment unit for adjusting the virtual space. The system according to feature 1.

4. The aforementioned visual unit is Providing a visual experience using high-resolution displays and real-time rendering technology. The system according to feature 1.

5. The aforementioned acoustic unit is Providing an audio experience using spatial audio technology. The system according to feature 1.

6. The aforementioned tactile part is, Providing a haptic experience using haptic feedback devices. The system according to feature 1.

7. The aforementioned simulation unit, To simulate smells and tastes within a realistic range. The system according to feature 1.

8. The optimization unit, Optimize the experience based on user feedback. The system according to feature 1.

Images