System

Abstract

A system is provided.SOLUTION: A system comprising: generation means for generating a landscape; display means for displaying the generated landscape; sound reproduction means for reproducing a sound corresponding to the landscape; smell generation means for generating a smell corresponding to the landscape; wind generation means for generating wind corresponding to the landscape; acquisition means for acquiring user information; and control means for generating the landscape based on the acquired user information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People who are bedridden due to illness or injury, or who have difficulty going out on their own, are unable to experience the changing of the seasons or the scenery outside, which can be a major source of stress. For hospitalized patients and the elderly in particular, seeing the same scenery every day can increase their mental burden, potentially delaying recovery and reducing their quality of life. Conventional digital photo frames and video playback devices lack the functionality to alleviate this stress. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including a generation means for generating a scenery, a display means for displaying the generated scenery, a sound reproduction means for reproducing a sound corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating a wind corresponding to the scenery, an acquisition means for acquiring user information, and a control means for generating a scenery based on the acquired user information. Furthermore, by including a feedback acquisition means for acquiring feedback from the user and a feedback analysis means for analyzing the feedback and reflecting it in the next scenery generation, customization according to the individual needs and preferences of the user is possible. Furthermore, by including a time / season change generation means for generating changes based on the season, weather, and time of day, even people who cannot go out can feel the change of seasons and time.

[0006] "Generation means" refers to the techniques and devices for generating a scene.

[0007] "Display means" refers to a display or screen for displaying the generated scenery.

[0008] "Sound reproduction means" refers to a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[0009] "Odor generating means" refers to a device such as an aroma diffuser that generates an odor that corresponds to the scenery.

[0010] The "wind generating means" refers to a fan or blower for generating wind according to the scenery.

[0011] "Acquisition means" refers to the technology and devices for acquiring user information.

[0012] "Control means" refers to control devices and techniques for generating scenery based on acquired user information.

[0013] "Feedback acquisition means" refers to techniques and devices for acquiring feedback from users.

[0014] "Feedback analysis means" refers to the technology and device for analyzing the acquired feedback and reflecting it in the next landscape generation.

[0015] "Time and season change generation means" refers to the technology and devices for generating changes in scenery based on season, weather, and time of day. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system that generates scenery and provides users with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, and a time and season change generation means.

[0038] System configuration

[0039] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[0040] 2. The display means is a display or screen for displaying the generated scenery.

[0041] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[0042] 4. The scent generating means is a device such as an aroma diffuser that generates scents that correspond to the scenery.

[0043] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[0044] 6. Acquisition means is a technology and device for acquiring user information, such as the user's age, gender, favorite scenery, least favorite smell, and desired display time period.

[0045] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[0046] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[0047] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[0048] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[0049] Program processing flow

[0050] Collecting user information

[0051] The user accesses the settings screen of the digital photo frame or digital window and enters the following information:

[0052] Age, gender

[0053] Favorite scenery (e.g., ocean, mountains, forest)

[0054] Disliked smells (e.g. pollen, smoke)

[0055] Desired display time period (e.g. 3pm)

[0056] Send data to the server

[0057] The terminal transmits the input user information to the server.

[0058] Scenery generation using generative AI

[0059] The server uses AI to generate the requested scenery based on the received user information, including changes according to the season and time of day.

[0060] Video and audio streaming

[0061] The server streams the generated landscape video and sound data to the terminal.

[0062] Smell and wind control

[0063] The server also transmits information about the smell and wind corresponding to the scenery to the device, which then controls the aroma diffuser and fan to recreate the smell and wind in real time.

[0064] Specific examples

[0065] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user gives feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound.

[0066] This system allows people who are unable to go outside to feel the change in seasons and time, thereby reducing stress.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] Users enter information such as personal information (e.g., age, gender), preferred landscape type (e.g., ocean, mountain, forest), disliked smells (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.) on the settings screen of a digital photo frame or digital window.

[0070] Step 2:

[0071] The terminal acquires the entered user information and transmits it to the server, properly formatting the data and implementing the necessary security measures for transmission.

[0072] Step 3:

[0073] The server analyzes the received user information and extracts necessary parameters, such as favorite scenery, least favorite smells, and desired display time, and passes these to the generation AI as parameters.

[0074] Step 4:

[0075] The server's AI generates the best scenery for the user based on the parameters passed in. For example, if a user selects "sea" at 3pm in the summer, the scenery will be generated to show blue skies and rippling seas.

[0076] Step 5:

[0077] The server streams the generated landscape video and corresponding audio data to the device. The server encodes the video and audio data and prepares them for distribution in real time.

[0078] Step 6:

[0079] The device decodes the video and audio data received from the server and plays the audio from the speaker while displaying it on the screen. Specifically, you can hear the sound of ocean waves and see an image of the ocean lapping on the screen.

[0080] Step 7:

[0081] The server also sends the smell and wind data corresponding to the generated scenery to the device, including, for example, the "smell of the sea" and a "light sea breeze."

[0082] Step 8:

[0083] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. Specifically, the aroma diffuser emits the scent of the sea and the fan recreates a light breeze.

[0084] Step 9:

[0085] After the experience, users input their feedback into the device, including their impressions of the scenery, smells, and wind strength, as well as suggestions for improvement.

[0086] Step 10:

[0087] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[0088] Step 11:

[0089] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation, such as slightly weakening the wind strength.

[0090] This series of steps allows users to have a customized real-time scenery experience, allowing them to enjoy the changes in nature and the beauty of the scenery even when it is difficult to go outside.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Conventional visual experience systems lack the ability to coordinate various senses, such as displaying scenery, reproducing sound, generating smells, and generating wind, making it difficult to provide users with a comprehensive and immersive experience. It is also difficult to customize the experience based on individual user preferences and feedback. Furthermore, they lack the functionality to reflect changes in real time according to the season, weather, and time of day. This has created technical challenges for improving user satisfaction.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound playback means for playing sounds corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a transmission means for streaming the generated scenery and sound, and an odor and wind control means for generating odor and wind control information corresponding to the scenery and transmitting it to the terminal. This enables a comprehensive, immersive experience that reflects the user's individual preferences and real-time changes in season and time zone.

[0096] The "means for generating a scene" refers to a device and technology for generating a visual scene based on user information.

[0097] The "display means for displaying the generated scenery" refers to a display or screen for displaying the generated scenery in a form that can be visually recognized by the user, and a control device for the display or screen.

[0098] "Sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or sound reproduction device for reproducing sounds that match the generated scenery.

[0099] The "odor generating means for generating an odor corresponding to a scenery" refers to an aroma diffuser or a fragrance generating device for generating various odors corresponding to a scenery.

[0100] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind with strength and direction according to the scenery.

[0101] "Means for acquiring user information" refers to devices and technologies for collecting information such as the user's age, gender, preferences, disliked smells, and desired display time period.

[0102] The "control means for generating a landscape based on acquired user information" refers to a control device and technology for controlling the generation means based on acquired user information and generating an appropriate landscape.

[0103] The "transmission means for streaming the generated scenery and sound" refers to a technique and device for transmitting the generated scenery video and sound data to a terminal in real time.

[0104] "Smell and wind control means for generating smell and wind control information corresponding to a landscape and transmitting it to a terminal" refers to technology and devices for transmitting control signals to a terminal based on smell and wind information corresponding to the generated landscape.

[0105] The "feedback acquisition means for acquiring feedback from users" refers to devices and techniques for collecting evaluations and comments entered by users after their experiences.

[0106] The "feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation" refers to a device and technology for analyzing the collected feedback and adjusting the next generation parameters.

[0107] "Generative AI means for generating scenery using a generative AI model" refers to technology and devices for generating appropriate scenery using a generative AI model based on user information.

[0108] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and equipment for changing the landscape while reflecting real-time information such as season, weather, and time of day.

[0109] The present invention relates to a system that generates scenery and provides a user with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, a transmission means, and a scent and wind control means.

[0110] Collecting user information

[0111] The user accesses the settings screen of the digital photo frame or digital window and enters information such as age, gender, favorite scenery (e.g., ocean, mountain, forest), least favorite smell (e.g., pollen, smoke), desired display time (e.g., 3:00 p.m.), etc. The entered information is packaged in JSON format by the device and sent to the server.

[0112] Scenery generation using generative AI

[0113] The server uses a generation AI to generate scenery based on the received user information. Specifically, it sends a request to the generation AI model to generate scenery based on the user's preferences, season, and time of day. The generated scenery is saved on the server as a video file.

[0114] Prompt Sentence Examples

[0115] "Generate a 'seascape and wave sounds at 3pm in the summer' based on user information. Adjust the settings to avoid unpleasant smells and provide a satisfying experience for the user."

[0116] Streaming video and audio

[0117] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and streams it to the device in real time. The device then displays the received data on its display and plays the sound through its speakers.

[0118] Smell and wind control

[0119] The server also generates information about the scent and wind corresponding to the scenery and sends it to the device. The device analyzes the received information and controls the aroma diffuser and fan. The diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[0120] Collecting and analyzing feedback

[0121] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. The device then sends this feedback in JSON format to the server. The server analyzes the feedback and makes adjustments to reflect it in the next landscape generation. The results of the feedback analysis are saved in the user profile and used again.

[0122] Specific examples

[0123] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user can give feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound. This system allows people who are unable to go outside to feel the change of seasons and time, thereby reducing stress.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, favorite scenery, least favorite smell, and desired viewing time. This input is done using a touch panel or keyboard and is confirmed by pressing the "Submit" button. The input information is structured as JSON-formatted data, as it will be used in subsequent processes to generate scenery.

[0127] (Input) User information (age, gender, favorite scenery, disliked smell, display time zone)

[0128] (Output) User information data in JSON format

[0129] Step 2:

[0130] The terminal sends the entered user information to the server. Specifically, it transfers the user information structured in JSON format to the server as an HTTP request. The server receives this request and saves the user information in a database.

[0131] (Input) JSON format user information data

[0132] (Output) The HTTP request sent to the server

[0133] Step 3:

[0134] The server uses a generative AI to generate scenery based on the received user information. Specifically, it generates prompts based on the user's preferences, season, and time of day, and inputs these into the generative AI model. The generative AI model generates related scenery data based on these prompts and returns the results to the server. The server saves this as a video file.

[0135] (Input) JSON format user information data

[0136] (Output) Generated landscape video file

[0137] Step 4:

[0138] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and sends it to the device as an HTTP stream. The device receives this in real time, displays it on the display, and plays the sound through the speaker.

[0139] (Input) Generated landscape video file

[0140] (Output) Video displayed on the display and sound played from the speakers

[0141] Step 5:

[0142] The server generates scent and wind information corresponding to the scenery and sends it to the terminal. Specifically, it generates scent and wind control information corresponding to the scenery in XML format and sends it to the terminal. The terminal analyzes this information and controls the aroma diffuser and fan. The aroma diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[0143] (Input) Generated scenery, smell, and wind control information

[0144] (Output) Emitted scent and generated wind

[0145] Step 6:

[0146] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. Feedback is given using a touch panel or keyboard and is confirmed by pressing the "Send" button. The input feedback is structured in JSON format and sent from the device to the server.

[0147] (Input) Feedback information (evaluation of scenery, sounds, smells, and wind)

[0148] (Output) Feedback data in JSON format

[0149] Step 7:

[0150] The server analyzes the user's feedback and reflects it in the next landscape generation. Specifically, it uses an analytical algorithm to analyze the feedback data and adjust the next generation parameters. The adjustment results are saved in the user's profile and used again.

[0151] (Input) Feedback data in JSON format

[0152] (Output) Adjusted generation parameter data

[0153] (Application example 1)

[0154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] While conventional landscape generation systems can provide users with visual, auditory, olfactory, and tactile experiences, they lack the ability to dynamically present information and provide feedback based on the user's real-time interests and preferences. In particular, in brick-and-mortar stores, when a user shows interest in a particular product, it is necessary to detect that interest in real time and provide related information and experiences. Current systems struggle to meet this demand. Furthermore, their ability to reflect feedback in the next experience is limited, lacking the flexibility to improve the quality of the user experience.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0157] In this invention, the server includes a visual display means, which is a display device for providing a user with a real-time scenery experience, an information display means for displaying related information during the experience, and sensor technology for tracking the user's gaze or detecting their interest. This allows scenery and related information based on the user's interests and preferences to be provided in real time, and feedback can be reflected in the scenery generation. This allows users to enjoy a realistic experience while interacting with products in a physical store, and further improves the quality of the experience through feedback.

[0158] "Generation means for generating landscapes" refers to technology and devices that generate landscapes in real time using generative AI models based on user information.

[0159] The "display means for displaying the generated scenery" is a device including a display or screen for visually presenting the generated scenery to the user.

[0160] The "sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or an audio reproduction device for providing the user with sounds related to the generated scenery.

[0161] The "odor generating means for generating an odor corresponding to the scenery" is a device such as an aroma diffuser for reproducing an odor associated with the generated scenery.

[0162] The "wind generating means for generating wind according to the scenery" is a fan or air blowing device for providing the user with wind related to the generated scenery.

[0163] The "means for acquiring user information" refers to the technology and devices for obtaining information such as the user's age, sex, and preferences.

[0164] The "control means for generating a landscape based on acquired user information" refers to a technique and device for managing the process of generating a landscape based on acquired user information.

[0165] The "visual display means, which is a display device for providing a user with a real-time scenery experience" is a device that displays the generated scenery in real time using a wearable device such as smart glasses.

[0166] "Information display means for displaying relevant information during an experience" refers to a device including a display or screen for providing additional information to the user while experiencing a scene.

[0167] "Sensor technology for tracking a user's gaze or detecting their interest" refers to a sensor technology for detecting the direction of a user's gaze or interest in real time.

[0168] The "feedback acquisition means for acquiring feedback from users" refers to techniques and devices for collecting impressions and opinions from users after the experience.

[0169] The "feedback analysis means for analyzing the feedback and reflecting it in the next scenery generation" refers to a technique and device for analyzing the acquired user feedback and reflecting it in the next scenery generation.

[0170] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and devices for changing the content of a landscape according to season, weather, and time of day in landscape generation.

[0171] The present invention relates to a system that generates scenery in real time based on user information and provides visual, auditory, olfactory, and tactile experiences. This system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, a user information acquisition means, a control means based on the acquired information, a visual display means, an information display means, sensor technology, a feedback acquisition means, a feedback analysis means, and a time / season change generation means.

[0172] System hardware and software configuration

[0173] Scenery generation method: A generative AI model (e.g., DALL-E or GPT-3) is used to generate scenery based on user information in real time.

[0174] Display means: Smart glasses or a display are used to display the generated scenery.

[0175] Sound reproduction means: sound reproducers and speakers are used to reproduce sounds associated with the scenery.

[0176] Scent generation method: An aroma diffuser is used to recreate the scent.

[0177] Wind generation means: A fan is used to recreate the wind.

[0178] Acquisition means: A sensor or input device is used to acquire user information.

[0179] Control means: Software and a control device for controlling each device based on the acquired user information.

[0180] Visual display means: Displayed in real time using smart glasses.

[0181] Information display means: A display for showing additional information.

[0182] Sensor technology: Eye-tracking sensors to detect user gaze and interest.

[0183] Feedback acquisition means: A feedback system for collecting user feedback.

[0184] Feedback analysis method: Technology that analyzes feedback and reflects it in the next landscape generation.

[0185] Time and season change generation method: Technology that generates scenery changes according to the season and time of day.

[0186] Processing flow and specific examples

[0187] 1. Collecting user information: The user enters information such as their age, gender, and favorite scenery through the smart glasses interface. For example, the user might enter, "I'm a 30-year-old woman, I like forest scenery, I dislike the smell of smoke, and I'd like to see the display in the afternoon."

[0188] 2. Scenery Generation: The server generates a scene using the generative AI model based on the acquired user information. It uses the following prompts:

[0189] The user is a 30-year-old woman who likes forest scenery. Generate a scenery that is displayed in the afternoon. The user dislikes the smell of smoke.

[0190] The generative AI model generates a scene based on these prompts and displays it in real time.

[0191] 3. Display of scenery and related information: The smart glasses display the generated scenery in real time, the sound player plays sounds related to the scenery (e.g., birdsong in the forest), the scent generator reproduces the natural scent of the forest, and the fan provides a gentle breeze.

[0192] 4. Feedback collection and analysis: The eye-tracking sensor tracks the user's gaze and identifies the object of interest. At the same time, the feedback acquisition means collects feedback from the user, and the feedback analysis means analyzes it to reflect it in the next experience. For example, if the feedback is "The wind is a little too strong," the wind setting will be adjusted for the next experience.

[0193] This system will enable users to enjoy scenery and related information that matches their preferences in real time at specific locations such as brick-and-mortar stores, improving the quality of their experience.

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] Collection of User Information

[0197] Through the smart glasses' interface, users input information such as their age, gender, favorite scenery, disliked smells, and desired viewing time. This input information is saved on the device and sent to the server. The input data includes age, gender, favorite scenery type (e.g., forest, ocean, mountain, etc.), disliked smell (e.g., smoke, pollen, etc.), and desired viewing time. This prepares the system to generate scenery tailored to the user's preferences and characteristics.

[0198] Step 2:

[0199] Prompt creation for landscape generation

[0200] The server takes user information and creates a prompt to input that information into the generative AI model. The input data includes the user's age, gender, favorite type of scenery, least favorite smell, and desired viewing time, and is formatted as the prompt sentence as follows:

[0201] "The user is a 30-year-old woman who likes forest scenery. Generate a scenery that will be displayed in the afternoon. The user dislikes the smell of smoke."

[0202] This prompt is passed to the generative AI model and serves as the basis for generating the landscape.

[0203] Step 3:

[0204] Scenery generation

[0205] The server inputs the prompt sentence into a generative AI model, which generates a scene in real time. The generative AI model (e.g., DALL-E or GPT-3) generates image data, sound data, smell information, and wind information of the scene based on the prompt sentence. The input data is the prompt sentence, and the output is an image of the generated scene, sounds related to the scene, smell information, and wind information. This allows a scene to be generated in real time according to the user's preferences and wishes.

[0206] Step 4:

[0207] Display of scenery and related information

[0208] The device receives the generated scenery data and displays it on the smart glasses. At the same time, it sends sound data to the audio player to play sounds related to the scenery. It also sends scent information to the aroma diffuser to generate the associated scent, and sends wind information to the fan to recreate the wind. The input data are the scenery image, sound data, scent information, and wind information, and the output is the scenery displayed on the smart glasses, the sound played from the audio player, the scent emitted from the aroma diffuser, and the wind blowing from the fan. This allows the user to enjoy an immersive experience.

[0209] Step 5:

[0210] Gathering feedback

[0211] After the experience, users input their impressions and opinions through a feedback screen. This feedback information is saved on the device and sent to the server. The input data includes the user's evaluation of the appropriateness of the scenery, the quality of the sound, the strength of the smell, and the strength of the wind. This allows feedback based on the user's experience to be collected.

[0212] Step 6:

[0213] Analyzing and incorporating feedback

[0214] The server analyzes the collected feedback information and reflects it in the next landscape generation. The analyzed data is feedback information from the user and is reflected in the generation AI model. For example, if there is feedback that "the wind is a little too strong," the wind strength will be adjusted the next time wind is generated. The output is an improved landscape generation process, which improves the user's experience from the next time onwards.

[0215] Through the input and output data and specific actions at each step, users can have a more personalized and immersive experience.

[0216] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0217] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, an emotion engine, an emotion-based adjustment means, and an emotion data analysis means.

[0218] System configuration

[0219] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[0220] 2. The display means is a display or screen for displaying the generated scenery.

[0221] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[0222] 4. The scent generating means is a device such as an aroma diffuser that generates a scent that corresponds to the scenery.

[0223] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[0224] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[0225] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[0226] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[0227] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[0228] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[0229] 11. Emotion engines are technologies and devices for recognizing user emotions, for example, by using facial recognition technology or sensors.

[0230] 12. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[0231] 13. Emotion data analysis means is a technique and device for acquiring and analyzing the user's emotion data recognized by the emotion engine.

[0232] Program processing flow

[0233] Collecting user information

[0234] The user inputs information such as age, sex, favorite scenery, least favorite smell, and desired display time period on the setting screen of the digital photo frame or digital window.

[0235] Send data to the server

[0236] The terminal sends the entered user information to the server, where the data is formatted appropriately and transmitted via secure communication.

[0237] Scenery generation using generative AI

[0238] The server generates the requested scenery using the received user information and generation AI. For example, if you generate a seascape at 3 pm in the summer, a blue sky and rippling seascape will be generated.

[0239] Streaming video and audio

[0240] The server then streams the generated landscape video and corresponding sound data to the device. The data is encoded and delivered in real time.

[0241] Smell and wind control

[0242] The server also sends the device smell and wind data corresponding to the generated scenery, including, for example, the "smell of the sea" and "light sea breeze."

[0243] Reproduction of video, sound, smell, and wind

[0244] The device decodes the video and audio data received from the server, displays it on the display, plays the audio from the speaker, and also emits scents from an aroma diffuser and generates wind with a fan.

[0245] User Emotion Recognition

[0246] The emotion engine uses sensors and cameras to recognize the user's emotions, for example by analyzing the user's facial expressions to identify emotions.

[0247] Dynamic landscape adjustment

[0248] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's recognized emotion, for example, changing the scenery to a calming one if the user is not relaxed.

[0249] Specific examples

[0250] The user inputs settings such as "I like ocean views, I don't like the smell of pollen, and I want it to start at 3 p.m.", and the device sends these to the server. The server uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it sends data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more soothing ones.

[0251] This system allows users who are unable to go outside to experience a real-time landscape that is customized to their emotions, which is expected to reduce stress.

[0252] The processing flow will be explained below.

[0253] Step 1:

[0254] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, preferred scenery (e.g., ocean, mountain, forest), least preferred smell (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.).

[0255] Step 2:

[0256] The device takes the entered user information, formats it, and sends it to the server, where it is encrypted to ensure data privacy and security.

[0257] Step 3:

[0258] The server analyzes the received user information and extracts necessary parameters (such as type of scenery, scent preferences, and display time period).

[0259] Step 4:

[0260] The server uses generative AI to generate scenery. For example, for a user who wants a seascape at 3 pm on a summer day, it generates a video with blue skies and rippling sea.

[0261] Step 5:

[0262] The server encodes the generated landscape video and corresponding sound data (e.g., the sound of waves) and streams them to the terminal.

[0263] Step 6:

[0264] The device decodes the received video and audio data and plays the audio from the speaker while displaying it on the screen. Specifically, the device displays an ocean scene on the screen and plays the sound of lapping waves.

[0265] Step 7:

[0266] The server also simultaneously transmits data on the smell (e.g., the smell of the sea) and wind (e.g., a light sea breeze) corresponding to the generated scenery to the terminal.

[0267] Step 8:

[0268] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. For example, the aroma diffuser can emit the scent of the sea and the fan can reproduce a light breeze.

[0269] Step 9:

[0270] The emotion engine uses cameras and sensors to analyze the user's facial expressions and biometric data to recognize the user's emotions, for example, detecting stress signals from the user's facial expressions.

[0271] Step 10:

[0272] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's emotion data recognized by the emotion engine, for example, if the user is not relaxed, the scenery is changed to be more calm and relaxing.

[0273] Step 11:

[0274] After the experience, users input their feedback into the device, including their impressions of the scenery, sounds, smells, and wind strength, as well as suggestions for improvement.

[0275] Step 12:

[0276] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[0277] Step 13:

[0278] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation (e.g., adjusting wind strength or scent intensity).

[0279] This series of steps allows users to enjoy a customized, real-time landscape experience, allowing them to enjoy the beauty of nature and its changing scenery even when it is difficult to go outside. Furthermore, the system dynamically adjusts according to the user's emotional state, providing a more personalized relaxation experience.

[0280] Example 2

[0281] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0282] Many users today are facing difficulties in going outside, resulting in fewer opportunities to experience nature and scenery. Furthermore, existing systems lack the ability to customize experiences based on the user's emotional state, making it difficult to provide a comprehensive experience that is tailored in real time based on the user's individual preferences and emotions.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0284] In this invention, the server includes a generating means for generating a scenery, a display means for displaying the generated scenery, a sound reproducing means for reproducing a sound corresponding to the scenery, an odor generating means for generating an odor corresponding to the scenery, a wind generating means for generating a wind corresponding to the scenery, a means for acquiring user information, a control means for generating a scenery based on the acquired user information, an emotion engine for recognizing the user's emotion, an emotion-based adjustment means for dynamically adjusting the scenery, sound, odor, and wind content based on the recognized user emotion, and a technology for generating a scenery based on the user information using a generative AI model. This enables a scenery experience customized in real time according to the user's emotions and preferences.

[0285] The "means for generating scenery" refers to a technique and device for generating scenery in real time based on user information.

[0286] The "display means for displaying the generated scenery" refers to a display or screen for visually presenting the generated scenery to the user.

[0287] The "sound reproduction means for reproducing sounds corresponding to the scenery" refers to a speaker or an audio reproduction device for reproducing sounds corresponding to the generated scenery.

[0288] The "odor generating means for generating an odor corresponding to a scenery" is a device such as an aroma diffuser for generating an odor corresponding to the generated scenery.

[0289] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind corresponding to the generated scenery.

[0290] The "means for acquiring user information" refers to technology and devices for collecting user information such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[0291] The "control means for generating a scene based on acquired user information" refers to a control device and technology for generating a scene based on acquired user information.

[0292] An "emotion engine that recognizes user emotions" is a technology and device that analyzes the user's facial expressions and data from sensors to recognize the user's emotions.

[0293] "Emotion-based adjustment means for dynamically adjusting the content of scenery, sounds, smells, and wind based on the recognized user emotions" refers to technology and devices that adjust the type and strength of scenery, sounds, smells, and wind in real time according to the user emotions recognized by the emotion engine.

[0294] "Technology for generating scenery based on user information using a generative AI model" is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[0295] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, an emotion engine, an emotion-based adjustment means, and a technology that uses a generative AI model.

[0296] System configuration

[0297] 1. The landscape generation means is a technology and device that uses generative AI to generate landscapes in real time based on user information. For example, the generative AI model generates a seascape of summer at 3:00 p.m. based on the user's preferences and desired time of day.

[0298] 2. The display means is a display or screen for displaying the generated scenery. It is installed in a position that is easy for the user to see and provides high-resolution images.

[0299] 3. The sound reproduction means is a speaker or sound reproduction device that reproduces sounds that match the scenery, such as the sound of waves or birds chirping.

[0300] 4. The scent generating means is a device such as an aroma diffuser that generates scents according to the scenery. For example, it emits scents related to the scenery, such as the "scent of the sea" or the "scent of flowers."

[0301] 5. The wind generating means is a fan or blower for generating wind according to the scenery. For example, a fan for reproducing a light sea breeze is used.

[0302] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[0303] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[0304] 8. Emotion engines are technologies and devices for recognizing user emotions, for example, using facial recognition technology and sensors.

[0305] 9. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[0306] 10. The technology for generating scenery based on user information using a generative AI model is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[0307] Specific examples

[0308] The user inputs their preferences, such as "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m.", and the device sends these to the server. The server then uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams this to the device. At the same time, it transmits data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more gentle ones. This allows the user to experience a personalized, relaxing experience.

[0309] Example prompt for a generative AI model:

[0310] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[0311] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0312] Step 1:

[0313] The user inputs information such as age, gender, favorite scenery, least favorite smell, desired display time, etc. on the touch screen of the digital photo frame. The input information is in the following format:

[0314] Input: Age, gender, favorite scenery, disliked smell, desired display time

[0315] Output: User information data (json format)

[0316] Step 2:

[0317] The device converts the information entered by the user into JSON format and sends it to the server in a secure HTTP request using the SSL / TLS protocol, encrypting the data during the process.

[0318] Input: User information data

[0319] Output: Encrypted user information request

[0320] Step 3:

[0321] The server decodes the received user information and generates a prompt for the AI ​​model, which includes specific instructions for generating scenery based on the user's preferences and desired time period.

[0322] Input: Encrypted user information request

[0323] Output: prompt statement

[0324] Example prompt sentence:

[0325] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[0326] Step 4:

[0327] The server uses a generative AI model to generate corresponding scenery images and sound data based on the prompt sentence, which uses a large-scale neural network to generate scenery data in real time.

[0328] Input: prompt statement

[0329] Output: Landscape image data, sound data

[0330] Step 5:

[0331] The server encodes the generated scenery video and audio data as an H.264 video stream and streams it to the device. The audio data is encoded in AAC format and synchronized with the video stream.

[0332] Input: landscape image data, sound data

[0333] Output: Encoded video stream, audio stream

[0334] Step 6:

[0335] The server generates smell and wind setting data corresponding to the landscape in JSON format and sends it to the terminal.

[0336] Input: User information, landscape image data, sound data

[0337] Output: Smell data, wind data

[0338] Example of scent data (JSON format):

[0339] json

[0340] {

[0341] "scent": "sea_breeze",

[0342] "wind_level": 2

[0343] }

[0344] Step 7:

[0345] The device decodes the received video stream and displays it on the display. At the same time, it decodes the sound data and plays it from the speaker. It also sends the command "sea_breeze" to the aroma diffuser, causing the fan to operate at wind level "2". Input: Encoded video stream, sound stream, smell data, wind data

[0346] Output: Display, audio playback, scent emission, wind generation

[0347] Step 8:

[0348] The emotion engine uses a camera and facial expression recognition algorithms (e.g., OpenCV and Dlib) to obtain emotion data from the user's facial expressions, and analyzes the obtained emotion data to determine whether the user is relaxed or not.

[0349] Input: Camera video data

[0350] Output: Emotion data

[0351] Step 9:

[0352] The emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind parameters based on the analysis results of the emotion engine. For example, if the user is not relaxed, the scenery will be changed to a "calm lake" and the music will be changed to a "quiet piano melody."

[0353] Input: Emotion data, scenery, sound, smell, wind parameters

[0354] Output: Adjusted scenery, sound, smell, and wind parameters

[0355] (Application example 2)

[0356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] When seeking relaxation and peace of mind, modern users seek customized, holistic experiences based not only on visual elements but also on hearing, smell, touch, and even emotional state. However, existing systems face challenges in integrating these multiple sensory elements and dynamically adjusting them based on individual users' emotions and feedback. In particular, there is a need for systems that can analyze a user's emotional state in real time and make appropriate adjustments in virtual experiences aimed at stress reduction and relaxation.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0359] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound reproduction means for reproducing a sound corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating a wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a distribution means for streaming the generated scenery, an emotion analysis means for analyzing the user's emotions, and an emotion-based adjustment means for dynamically adjusting the scenery based on the emotion analysis. This makes it possible to provide a comprehensive and dynamically customized scenery experience for each individual user, thereby realizing user relaxation and stress reduction.

[0360] "Generation means" refers to the devices and techniques used to generate a landscape.

[0361] "Display means" refers to a display or screen for displaying the generated scenery.

[0362] The "sound reproduction means" refers to a speaker or an audio reproduction device for reproducing sounds that correspond to the scenery.

[0363] The "scent generating means" is a device such as an aroma diffuser that generates a scent that matches the scenery.

[0364] The "wind generating means" refers to a fan or air blower for generating wind that matches the scenery.

[0365] The "acquisition means" refers to a sensor or input device for acquiring user information.

[0366] The "control means" is a computer or software for controlling the generation of scenery based on the acquired user information.

[0367] "Delivery means" refers to the network technology and servers used to stream the generated scenery in real time.

[0368] "Emotion analysis means" refers to facial recognition technology and sensors for analyzing the user's emotions.

[0369] An "emotion-based adjustment" is a system or algorithm for dynamically adjusting a scene based on emotion analysis.

[0370] The "feedback acquisition means" is a device or interface for collecting feedback from users.

[0371] "Feedback analysis means" refers to technology or software that analyzes the obtained feedback and reflects it in the next landscape generation.

[0372] A "generative AI model" is an artificial intelligence model that generates landscapes in real time based on user information and input data.

[0373] A "prompt sentence" is an input sentence or command used to give instructions to a generative AI model.

[0374] To implement the present invention, the system is composed of the following elements: a server, a terminal, and a user play key roles.

[0375] 1. Collection of User Information

[0376] Users use their smartphones to input information such as their preferred scenery, scent preferences, and desired viewing times, etc. Based on this information, basic data is generated to provide an individually customized experience.

[0377] 2. Data processing on the server

[0378] The terminal transmits the entered user information over the Internet to a server, which receives the information, formats it appropriately, and stores it.

[0379] The server is equipped with a generative AI model that generates scenery in real time based on user information.

[0380] Example: If a user selects "seascapes at 3pm" as their preference, the generative AI model generates "seascapes at 3pm in the summer."

[0381] During this process, the AI ​​model is instructed using prompts, such as "User preferences: beach scenery at 3 PM, dislikes floral scents. Generate scenery and analyze emotions in real-time."

[0382] 3. Streaming content

[0383] The server encodes the generated scenery and corresponding sound, smell, and wind data in real time and streams them to the device.

[0384] A distribution means is used to ensure uninterrupted distribution of data.

[0385] 4. Playing data on the device

[0386] The device decodes the received data in real time, displays the scenery on the display, plays sounds using the sound playback means, emits appropriate smells using the smell generation means, and reproduces wind using the wind generation means.

[0387] Example: At 3:00 PM, as set by the user, an ocean scene is displayed on the smartphone screen, the sound of waves is played from the speaker, the aroma diffuser emits the scent of the sea, and a fan simulates a light breeze.

[0388] 5. Sentiment Analysis and Dynamic Adjustment

[0389] The device captures the user's facial expressions and movements using a camera or sensor and sends them to a server.

[0390] The emotion analysis means of the server analyzes the emotion of the user, and the emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind to suit the emotion of the user.

[0391] Example: If the user is not relaxed, the scenery or music changes to a calming one.

[0392] As a result, this system is able to provide a comprehensive relaxation experience that is individually customized for each user and adapted to their emotional state.

[0393] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0394] Step 1:

[0395] Entering user information

[0396] Users launch the smartphone app and enter information such as their favorite scenery, least favorite smells, and desired display time.

[0397] Input: User preferences and settings (e.g., "I want to see ocean views at 3pm").

[0398] Output: User information is retrieved and passed to the next processing step.

[0399] Step 2:

[0400] Sending user information

[0401] The terminal transmits the acquired user information to a server via the Internet.

[0402] Input: User information obtained in step 1.

[0403] Output: User information is sent to the server and stored on the server.

[0404] Step 3:

[0405] Prompt generation for landscape generation

[0406] The server creates a prompt sentence that gives instructions to the generative AI model based on the received user information.

[0407] Input: User information (e.g. "I want to see a seascape at 3pm").

[0408] Output: A prompt statement is generated (e.g., "User preferences: beach scenery at 3 PM, dislikes floral scents.").

[0409] Step 4:

[0410] Scenery generation

[0411] The server's generative AI model generates scenery in real time based on the generated prompt sentence.

[0412] Input: The prompt statement.

[0413] Output: Generated landscape data and associated sound, smell, and wind data.

[0414] Step 5:

[0415] Encoding and Streaming Data

[0416] The server encodes the generated landscape data in real time and streams it to the terminal.

[0417] Input: Generated landscape data, sound data, smell data, and wind data.

[0418] Output: The encoded data is delivered to the device.

[0419] Step 6:

[0420] Data decoding and playback

[0421] The device decodes the received data, displays the scenery on the screen, plays sound from the speaker, and controls an aroma diffuser or fan to recreate scents and breezes.

[0422] Input: Encoded scenery data, sound data, smell data, and wind data.

[0423] Output: Scenery displayed on the screen, audio played from the speaker, scent emitted from the aroma diffuser, and wind generated by the fan.

[0424] Step 7:

[0425] User Emotion Recognition

[0426] The device uses a camera and sensors to capture the user's facial expressions and movements, and sends the data to a server.

[0427] Input: User facial expression and movement data.

[0428] Output: The acquired emotion data is sent to the server.

[0429] Step 8:

[0430] Sentiment analysis and landscape adjustment

[0431] The server analyzes the user's emotional state using an emotion analysis means, and dynamically adjusts the scenery, sounds, smells, and wind using an emotion-based adjustment means.

[0432] Input: The submitted emotion data.

[0433] Output: New scenery, sound, smell, and wind data adjusted based on the user's emotions.

[0434] Step 9:

[0435] Get feedback and incorporate it into the next generation

[0436] The device receives feedback from the user and sends it to the server, which analyzes the feedback and reflects it in the next landscape generation.

[0437] Input: User feedback information (e.g. satisfaction level, text comments).

[0438] Output: The analyzed feedback data is reflected in the next prompt sentence of the generative AI model.

[0439] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0440] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0441] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0442] [Second embodiment]

[0443] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0444] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0445] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0446] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0447] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0448] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0449] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0450] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0451] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0452] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0453] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0454] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0455] The present invention relates to a system that generates scenery and provides users with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, and a time and season change generation means.

[0456] System configuration

[0457] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[0458] 2. The display means is a display or screen for displaying the generated scenery.

[0459] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[0460] 4. The scent generating means is a device such as an aroma diffuser that generates scents that correspond to the scenery.

[0461] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[0462] 6. Acquisition means is a technology and device for acquiring user information, such as the user's age, gender, favorite scenery, least favorite smell, and desired display time period.

[0463] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[0464] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[0465] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[0466] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[0467] Program processing flow

[0468] Collecting user information

[0469] The user accesses the settings screen of the digital photo frame or digital window and enters the following information:

[0470] Age, gender

[0471] Favorite scenery (e.g., ocean, mountains, forest)

[0472] Disliked smells (e.g. pollen, smoke)

[0473] Desired display time period (e.g. 3pm)

[0474] Send data to the server

[0475] The terminal transmits the input user information to the server.

[0476] Scenery generation using generative AI

[0477] The server uses AI to generate the requested scenery based on the received user information, including changes according to the season and time of day.

[0478] Streaming video and audio

[0479] The server streams the generated landscape video and sound data to the terminal.

[0480] Smell and wind control

[0481] The server also transmits information about the smell and wind corresponding to the scenery to the device, which then controls the aroma diffuser and fan to recreate the smell and wind in real time.

[0482] Specific examples

[0483] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user gives feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound.

[0484] This system allows people who are unable to go outside to feel the change in seasons and time, thereby reducing stress.

[0485] The processing flow will be explained below.

[0486] Step 1:

[0487] Users enter information such as personal information (e.g., age, gender), preferred landscape type (e.g., ocean, mountain, forest), disliked smells (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.) on the settings screen of a digital photo frame or digital window.

[0488] Step 2:

[0489] The terminal acquires the entered user information and transmits it to the server, properly formatting the data and implementing the necessary security measures for transmission.

[0490] Step 3:

[0491] The server analyzes the received user information and extracts necessary parameters, such as favorite scenery, least favorite smells, and desired display time, and passes these to the generation AI as parameters.

[0492] Step 4:

[0493] The server's AI generates the best scenery for the user based on the parameters passed in. For example, if a user selects "sea" at 3 pm in the summer, the scenery will be generated to show blue skies and rippling seas.

[0494] Step 5:

[0495] The server streams the generated landscape video and corresponding audio data to the device. The server encodes the video and audio data and prepares them for distribution in real time.

[0496] Step 6:

[0497] The device decodes the video and audio data received from the server and plays the audio from the speaker while displaying it on the screen. Specifically, you can hear the sound of ocean waves and see an image of the ocean lapping on the screen.

[0498] Step 7:

[0499] The server also sends the smell and wind data corresponding to the generated scenery to the device, including, for example, the "smell of the sea" and a "light sea breeze."

[0500] Step 8:

[0501] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. Specifically, the aroma diffuser emits the scent of the sea and the fan recreates a light breeze.

[0502] Step 9:

[0503] After the experience, users input their feedback into the device, including their impressions of the scenery, smells, and wind strength, as well as suggestions for improvement.

[0504] Step 10:

[0505] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[0506] Step 11:

[0507] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation, such as slightly weakening the wind strength.

[0508] This series of steps allows users to have a customized real-time scenery experience, allowing them to enjoy the changes in nature and the beauty of the scenery even when it is difficult to go outside.

[0509] Example 1

[0510] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0511] Conventional visual experience systems lack the ability to coordinate various senses, such as displaying scenery, reproducing sound, generating smells, and generating wind, making it difficult to provide users with a comprehensive and immersive experience. It is also difficult to customize the experience based on individual user preferences and feedback. Furthermore, they lack the functionality to reflect changes in real time according to the season, weather, and time of day. This has created technical challenges for improving user satisfaction.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0513] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound playback means for playing sounds corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a transmission means for streaming the generated scenery and sound, and an odor and wind control means for generating odor and wind control information corresponding to the scenery and transmitting it to the terminal. This enables a comprehensive, immersive experience that reflects the user's individual preferences and real-time changes in season and time zone.

[0514] The "means for generating a scene" refers to a device and technology for generating a visual scene based on user information.

[0515] The "display means for displaying the generated scenery" refers to a display or screen for displaying the generated scenery in a form that can be visually recognized by the user, and a control device for the display or screen.

[0516] "Sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or sound reproduction device for reproducing sounds that match the generated scenery.

[0517] The "odor generating means for generating an odor corresponding to a scenery" refers to an aroma diffuser or a fragrance generating device for generating various odors corresponding to a scenery.

[0518] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind with strength and direction according to the scenery.

[0519] "Means for acquiring user information" refers to devices and technologies for collecting information such as the user's age, gender, preferences, disliked smells, and desired display time period.

[0520] The "control means for generating a landscape based on acquired user information" refers to a control device and technology for controlling the generation means based on acquired user information and generating an appropriate landscape.

[0521] The "transmission means for streaming the generated scenery and sound" refers to a technique and device for transmitting the generated scenery video and sound data to a terminal in real time.

[0522] "Smell and wind control means for generating smell and wind control information corresponding to a landscape and transmitting it to a terminal" refers to technology and devices for transmitting control signals to a terminal based on smell and wind information corresponding to the generated landscape.

[0523] The "feedback acquisition means for acquiring feedback from users" refers to devices and techniques for collecting evaluations and comments entered by users after their experiences.

[0524] The "feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation" refers to a device and technology for analyzing the collected feedback and adjusting the next generation parameters.

[0525] "Generative AI means for generating scenery using a generative AI model" refers to technology and devices for generating appropriate scenery using a generative AI model based on user information.

[0526] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and equipment for changing the landscape while reflecting real-time information such as season, weather, and time of day.

[0527] The present invention relates to a system that generates scenery and provides a user with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, a transmission means, and a scent and wind control means.

[0528] Collecting user information

[0529] The user accesses the settings screen of the digital photo frame or digital window and enters information such as age, gender, favorite scenery (e.g., ocean, mountain, forest), least favorite smell (e.g., pollen, smoke), desired display time (e.g., 3:00 p.m.), etc. The entered information is packaged in JSON format by the device and sent to the server.

[0530] Scenery generation using generative AI

[0531] The server uses a generation AI to generate scenery based on the received user information. Specifically, it sends a request to the generation AI model to generate scenery based on the user's preferences, season, and time of day. The generated scenery is saved on the server as a video file.

[0532] Prompt Sentence Examples

[0533] "Generate a 'seascape and wave sounds at 3pm in the summer' based on user information. Adjust the settings to avoid unpleasant smells and provide a satisfying experience for the user."

[0534] Streaming video and audio

[0535] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and streams it to the device in real time. The device then displays the received data on its display and plays the sound through its speakers.

[0536] Smell and wind control

[0537] The server also generates information about the scent and wind corresponding to the scenery and sends it to the device. The device analyzes the received information and controls the aroma diffuser and fan. The diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[0538] Collecting and analyzing feedback

[0539] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. The device then sends this feedback in JSON format to the server. The server analyzes the feedback and makes adjustments to reflect it in the next landscape generation. The results of the feedback analysis are saved in the user profile and used again.

[0540] Specific examples

[0541] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user can give feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound. This system allows people who are unable to go outside to feel the change of seasons and time, thereby reducing stress.

[0542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0543] Step 1:

[0544] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, favorite scenery, least favorite smell, and desired viewing time. This input is done using a touch panel or keyboard and is confirmed by pressing the "Submit" button. The input information is structured as JSON-formatted data, as it will be used in subsequent processes to generate scenery.

[0545] (Input) User information (age, gender, favorite scenery, disliked smell, display time zone)

[0546] (Output) User information data in JSON format

[0547] Step 2:

[0548] The terminal sends the entered user information to the server. Specifically, it transfers the user information structured in JSON format to the server as an HTTP request. The server receives this request and saves the user information in a database.

[0549] (Input) JSON format user information data

[0550] (Output) The HTTP request sent to the server

[0551] Step 3:

[0552] The server uses a generative AI to generate scenery based on the received user information. Specifically, it generates prompts based on the user's preferences, season, and time of day, and inputs these into the generative AI model. The generative AI model generates related scenery data based on these prompts and returns the results to the server. The server saves this as a video file.

[0553] (Input) JSON format user information data

[0554] (Output) Generated landscape video file

[0555] Step 4:

[0556] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and sends it to the device as an HTTP stream. The device receives this in real time, displays it on the display, and plays the sound through the speaker.

[0557] (Input) Generated landscape video file

[0558] (Output) Video displayed on the display and sound played from the speakers

[0559] Step 5:

[0560] The server generates scent and wind information corresponding to the scenery and sends it to the terminal. Specifically, it generates scent and wind control information corresponding to the scenery in XML format and sends it to the terminal. The terminal analyzes this information and controls the aroma diffuser and fan. The aroma diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[0561] (Input) Generated scenery, smell, and wind control information

[0562] (Output) Emitted scent and generated wind

[0563] Step 6:

[0564] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. Feedback is given using a touch panel or keyboard and is confirmed by pressing the "Send" button. The input feedback is structured in JSON format and sent from the device to the server.

[0565] (Input) Feedback information (evaluation of scenery, sounds, smells, and wind)

[0566] (Output) Feedback data in JSON format

[0567] Step 7:

[0568] The server analyzes the user's feedback and reflects it in the next landscape generation. Specifically, it uses an analytical algorithm to analyze the feedback data and adjust the next generation parameters. The adjustment results are saved in the user's profile and used again.

[0569] (Input) Feedback data in JSON format

[0570] (Output) Adjusted generation parameter data

[0571] (Application example 1)

[0572] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] While conventional landscape generation systems can provide users with visual, auditory, olfactory, and tactile experiences, they lack the ability to dynamically present information and provide feedback based on the user's real-time interests and preferences. In particular, in brick-and-mortar stores, when a user shows interest in a particular product, it is necessary to detect that interest in real time and provide related information and experiences. Current systems struggle to meet this demand. Furthermore, their ability to reflect feedback in the next experience is limited, lacking the flexibility to improve the quality of the user experience.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0575] In this invention, the server includes a visual display means, which is a display device for providing a user with a real-time scenery experience, an information display means for displaying related information during the experience, and sensor technology for tracking the user's gaze or detecting their interest. This allows scenery and related information based on the user's interests and preferences to be provided in real time, and feedback can be reflected in the scenery generation. This allows users to enjoy a realistic experience while interacting with products in a physical store, and further improves the quality of the experience through feedback.

[0576] "Generation means for generating landscapes" refers to technology and devices that generate landscapes in real time using generative AI models based on user information.

[0577] The "display means for displaying the generated scenery" is a device including a display or screen for visually presenting the generated scenery to the user.

[0578] The "sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or an audio reproduction device for providing the user with sounds related to the generated scenery.

[0579] The "odor generating means for generating an odor corresponding to the scenery" is a device such as an aroma diffuser for reproducing an odor associated with the generated scenery.

[0580] The "wind generating means for generating wind according to the scenery" is a fan or air blowing device for providing the user with wind related to the generated scenery.

[0581] The "means for acquiring user information" refers to the technology and devices for obtaining information such as the user's age, sex, and preferences.

[0582] The "control means for generating a landscape based on acquired user information" refers to a technique and device for managing the process of generating a landscape based on acquired user information.

[0583] The "visual display means, which is a display device for providing a user with a real-time scenery experience" is a device that displays the generated scenery in real time using a wearable device such as smart glasses.

[0584] "Information display means for displaying relevant information during an experience" refers to a device including a display or screen for providing additional information to the user while experiencing a scene.

[0585] "Sensor technology for tracking a user's gaze or detecting their interest" refers to a sensor technology for detecting the direction of a user's gaze or interest in real time.

[0586] The "feedback acquisition means for acquiring feedback from users" refers to techniques and devices for collecting impressions and opinions from users after the experience.

[0587] The "feedback analysis means for analyzing the feedback and reflecting it in the next scenery generation" refers to a technique and device for analyzing the acquired user feedback and reflecting it in the next scenery generation.

[0588] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and devices for changing the content of a landscape according to season, weather, and time of day in landscape generation.

[0589] The present invention relates to a system that generates scenery in real time based on user information and provides visual, auditory, olfactory, and tactile experiences. This system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, a user information acquisition means, a control means based on the acquired information, a visual display means, an information display means, sensor technology, a feedback acquisition means, a feedback analysis means, and a time / season change generation means.

[0590] System hardware and software configuration

[0591] Scenery generation method: A generative AI model (e.g., DALL-E or GPT-3) is used to generate scenery based on user information in real time.

[0592] Display means: Smart glasses or a display are used to display the generated scenery.

[0593] Sound reproduction means: sound reproducers and speakers are used to reproduce sounds associated with the scenery.

[0594] Scent generation method: An aroma diffuser is used to recreate the scent.

[0595] Wind generation means: A fan is used to recreate the wind.

[0596] Acquisition means: A sensor or input device is used to acquire user information.

[0597] Control means: Software and a control device for controlling each device based on the acquired user information.

[0598] Visual display means: Displayed in real time using smart glasses.

[0599] Information display means: A display for showing additional information.

[0600] Sensor technology: Eye-tracking sensors to detect user gaze and interest.

[0601] Feedback acquisition means: A feedback system for collecting user feedback.

[0602] Feedback analysis method: Technology that analyzes feedback and reflects it in the next landscape generation.

[0603] Time and season change generation method: Technology that generates scenery changes according to the season and time of day.

[0604] Processing flow and specific examples

[0605] 1. Collecting user information: The user enters information such as their age, gender, and favorite scenery through the smart glasses interface. For example, the user might enter, "I'm a 30-year-old woman, I like forest scenery, I dislike the smell of smoke, and I'd like to see the display in the afternoon."

[0606] 2. Scenery Generation: The server generates a scene using the generative AI model based on the acquired user information. It uses the following prompts:

[0607] The user is a 30-year-old woman who likes forest scenery. Generate a scenery that is displayed in the afternoon. The user dislikes the smell of smoke.

[0608] The generative AI model generates a scene based on these prompts and displays it in real time.

[0609] 3. Display of scenery and related information: The smart glasses display the generated scenery in real time, the sound player plays sounds related to the scenery (e.g., birdsong in the forest), the scent generator reproduces the natural scent of the forest, and the fan provides a gentle breeze.

[0610] 4. Feedback collection and analysis: The eye-tracking sensor tracks the user's gaze and identifies the object of interest. At the same time, the feedback acquisition means collects feedback from the user, and the feedback analysis means analyzes it to reflect it in the next experience. For example, if the feedback is "The wind is a little too strong," the wind setting will be adjusted for the next experience.

[0611] This system will enable users to enjoy scenery and related information that matches their preferences in real time at specific locations such as brick-and-mortar stores, improving the quality of their experience.

[0612] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0613] Step 1:

[0614] Collection of User Information

[0615] Through the smart glasses' interface, users input information such as their age, gender, favorite scenery, disliked smells, and desired viewing time. This input information is saved on the device and sent to the server. The input data includes age, gender, favorite scenery type (e.g., forest, ocean, mountain, etc.), disliked smell (e.g., smoke, pollen, etc.), and desired viewing time. This prepares the system to generate scenery tailored to the user's preferences and characteristics.

[0616] Step 2:

[0617] Prompt creation for landscape generation

[0618] The server takes user information and creates a prompt to input that information into the generative AI model. The input data includes the user's age, gender, favorite type of scenery, least favorite smell, and desired viewing time, and is formatted as the prompt sentence as follows:

[0619] "The user is a 30-year-old woman who likes forest scenery. Generate a scenery that will be displayed in the afternoon. The user dislikes the smell of smoke."

[0620] This prompt is passed to the generative AI model and serves as the basis for generating the landscape.

[0621] Step 3:

[0622] Scenery generation

[0623] The server inputs the prompt sentence into a generative AI model, which generates a scene in real time. The generative AI model (e.g., DALL-E or GPT-3) generates image data, sound data, smell information, and wind information of the scene based on the prompt sentence. The input data is the prompt sentence, and the output is an image of the generated scene, sounds related to the scene, smell information, and wind information. This allows a scene to be generated in real time according to the user's preferences and wishes.

[0624] Step 4:

[0625] Display of scenery and related information

[0626] The device receives the generated scenery data and displays it on the smart glasses. At the same time, it sends sound data to the audio player to play sounds related to the scenery. It also sends scent information to the aroma diffuser to generate the associated scent, and sends wind information to the fan to recreate the wind. The input data are the scenery image, sound data, scent information, and wind information, and the output is the scenery displayed on the smart glasses, the sound played from the audio player, the scent emitted from the aroma diffuser, and the wind blowing from the fan. This allows the user to enjoy an immersive experience.

[0627] Step 5:

[0628] Gathering feedback

[0629] After the experience, users input their impressions and opinions through a feedback screen. This feedback information is saved on the device and sent to the server. The input data includes the user's evaluation of the appropriateness of the scenery, the quality of the sound, the strength of the smell, and the strength of the wind. This allows feedback based on the user's experience to be collected.

[0630] Step 6:

[0631] Analyzing and incorporating feedback

[0632] The server analyzes the collected feedback information and reflects it in the next landscape generation. The analyzed data is feedback information from the user and is reflected in the generation AI model. For example, if there is feedback that "the wind is a little too strong," the wind strength will be adjusted the next time wind is generated. The output is an improved landscape generation process, which improves the user's experience from the next time onwards.

[0633] Through the input and output data and specific actions at each step, users can have a more personalized and immersive experience.

[0634] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0635] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, an emotion engine, an emotion-based adjustment means, and an emotion data analysis means.

[0636] System configuration

[0637] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[0638] 2. The display means is a display or screen for displaying the generated scenery.

[0639] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[0640] 4. The scent generating means is a device such as an aroma diffuser that generates a scent that corresponds to the scenery.

[0641] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[0642] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[0643] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[0644] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[0645] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[0646] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[0647] 11. Emotion engines are technologies and devices for recognizing user emotions, for example, by using facial recognition technology or sensors.

[0648] 12. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[0649] 13. Emotion data analysis means is a technique and device for acquiring and analyzing the user's emotion data recognized by the emotion engine.

[0650] Program processing flow

[0651] Collecting user information

[0652] The user inputs information such as age, sex, favorite scenery, least favorite smell, and desired display time period on the setting screen of the digital photo frame or digital window.

[0653] Send data to the server

[0654] The terminal sends the entered user information to the server, where the data is formatted appropriately and transmitted via secure communication.

[0655] Scenery generation using generative AI

[0656] The server generates the requested scenery using the received user information and generation AI. For example, if you generate a seascape at 3 pm in the summer, a blue sky and rippling seascape will be generated.

[0657] Streaming video and audio

[0658] The server then streams the generated landscape video and corresponding sound data to the device. The data is encoded and delivered in real time.

[0659] Smell and wind control

[0660] The server also sends the device smell and wind data corresponding to the generated scenery, including, for example, the "smell of the sea" and "light sea breeze."

[0661] Reproduction of video, sound, smell, and wind

[0662] The device decodes the video and audio data received from the server, displays it on the display, plays the audio from the speaker, and also emits scents from an aroma diffuser and generates wind with a fan.

[0663] User Emotion Recognition

[0664] The emotion engine uses sensors and cameras to recognize the user's emotions, for example by analyzing the user's facial expressions to identify emotions.

[0665] Dynamic landscape adjustment

[0666] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's recognized emotion, for example, changing the scenery to a calming one if the user is not relaxed.

[0667] Specific examples

[0668] The user inputs settings such as "I like ocean views, I don't like the smell of pollen, and I want it to start at 3 p.m.", and the device sends these to the server. The server uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it sends data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more soothing ones.

[0669] This system allows users who are unable to go outside to experience a real-time landscape that is customized to their emotions, which is expected to reduce stress.

[0670] The processing flow will be explained below.

[0671] Step 1:

[0672] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, preferred scenery (e.g., ocean, mountain, forest), least preferred smell (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.).

[0673] Step 2:

[0674] The device takes the entered user information, formats it, and sends it to the server, where it is encrypted to ensure data privacy and security.

[0675] Step 3:

[0676] The server analyzes the received user information and extracts necessary parameters (such as type of scenery, scent preferences, and display time period).

[0677] Step 4:

[0678] The server uses generative AI to generate scenery. For example, for a user who wants a seascape at 3 pm on a summer day, it generates a video with blue skies and rippling sea.

[0679] Step 5:

[0680] The server encodes the generated landscape video and corresponding sound data (e.g., the sound of waves) and streams them to the terminal.

[0681] Step 6:

[0682] The device decodes the received video and audio data and plays the audio from the speaker while displaying it on the screen. Specifically, the device displays an ocean scene on the screen and plays the sound of lapping waves.

[0683] Step 7:

[0684] The server also simultaneously transmits data on the smell (e.g., the smell of the sea) and wind (e.g., a light sea breeze) corresponding to the generated scenery to the terminal.

[0685] Step 8:

[0686] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. For example, the aroma diffuser can emit the scent of the sea and the fan can reproduce a light breeze.

[0687] Step 9:

[0688] The emotion engine uses cameras and sensors to analyze the user's facial expressions and biometric data to recognize the user's emotions, for example, detecting stress signals from the user's facial expressions.

[0689] Step 10:

[0690] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's emotion data recognized by the emotion engine, for example, if the user is not relaxed, the scenery is changed to be more calm and relaxing.

[0691] Step 11:

[0692] After the experience, users input their feedback into the device, including their impressions of the scenery, sounds, smells, and wind strength, as well as suggestions for improvement.

[0693] Step 12:

[0694] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[0695] Step 13:

[0696] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation (e.g., adjusting wind strength or scent intensity).

[0697] This series of steps allows users to enjoy a customized, real-time landscape experience, allowing them to enjoy the beauty of nature and its changing scenery even when it is difficult to go outside. Furthermore, the system dynamically adjusts according to the user's emotional state, providing a more personalized relaxation experience.

[0698] Example 2

[0699] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0700] Many users today are facing difficulties in going outside, resulting in fewer opportunities to experience nature and scenery. Furthermore, existing systems lack the ability to customize experiences based on the user's emotional state, making it difficult to provide a comprehensive experience that is tailored in real time based on the user's individual preferences and emotions.

[0701] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0702] In this invention, the server includes a generating means for generating a scenery, a display means for displaying the generated scenery, a sound reproducing means for reproducing a sound corresponding to the scenery, an odor generating means for generating an odor corresponding to the scenery, a wind generating means for generating a wind corresponding to the scenery, a means for acquiring user information, a control means for generating a scenery based on the acquired user information, an emotion engine for recognizing the user's emotion, an emotion-based adjustment means for dynamically adjusting the scenery, sound, odor, and wind content based on the recognized user emotion, and a technology for generating a scenery based on the user information using a generative AI model. This enables a scenery experience customized in real time according to the user's emotions and preferences.

[0703] The "means for generating scenery" refers to a technique and device for generating scenery in real time based on user information.

[0704] The "display means for displaying the generated scenery" refers to a display or screen for visually presenting the generated scenery to the user.

[0705] The "sound reproduction means for reproducing sounds corresponding to the scenery" refers to a speaker or an audio reproduction device for reproducing sounds corresponding to the generated scenery.

[0706] The "odor generating means for generating an odor corresponding to a scenery" is a device such as an aroma diffuser for generating an odor corresponding to the generated scenery.

[0707] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind corresponding to the generated scenery.

[0708] The "means for acquiring user information" refers to technology and devices for collecting user information such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[0709] The "control means for generating a scene based on acquired user information" refers to a control device and technology for generating a scene based on acquired user information.

[0710] An "emotion engine that recognizes user emotions" is a technology and device that analyzes the user's facial expressions and data from sensors to recognize the user's emotions.

[0711] "Emotion-based adjustment means for dynamically adjusting the content of scenery, sounds, smells, and wind based on the recognized user emotions" refers to technology and devices that adjust the type and strength of scenery, sounds, smells, and wind in real time according to the user emotions recognized by the emotion engine.

[0712] "Technology for generating scenery based on user information using a generative AI model" is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[0713] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, an emotion engine, an emotion-based adjustment means, and a technology that uses a generative AI model.

[0714] System configuration

[0715] 1. The landscape generation means is a technology and device that uses generative AI to generate landscapes in real time based on user information. For example, the generative AI model generates a seascape of summer at 3:00 p.m. based on the user's preferences and desired time of day.

[0716] 2. The display means is a display or screen for displaying the generated scenery. It is installed in a position that is easy for the user to see and provides high-resolution images.

[0717] 3. The sound reproduction means is a speaker or sound reproduction device that reproduces sounds that match the scenery, such as the sound of waves or birds chirping.

[0718] 4. The scent generating means is a device such as an aroma diffuser that generates scents according to the scenery. For example, it emits scents related to the scenery, such as the "scent of the sea" or the "scent of flowers."

[0719] 5. The wind generating means is a fan or blower for generating wind according to the scenery. For example, a fan for reproducing a light sea breeze is used.

[0720] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[0721] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[0722] 8. Emotion engines are technologies and devices for recognizing user emotions, for example, using facial recognition technology and sensors.

[0723] 9. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[0724] 10. The technology for generating scenery based on user information using a generative AI model is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[0725] Specific examples

[0726] The user inputs their preferences, such as "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m.", and the device sends these to the server. The server then uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams this to the device. At the same time, it transmits data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more gentle ones. This allows the user to experience a personalized, relaxing experience.

[0727] Example prompt for a generative AI model:

[0728] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[0729] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0730] Step 1:

[0731] The user inputs information such as age, gender, favorite scenery, least favorite smell, desired display time, etc. on the touch screen of the digital photo frame. The input information is in the following format:

[0732] Input: Age, gender, favorite scenery, disliked smell, desired display time

[0733] Output: User information data (json format)

[0734] Step 2:

[0735] The device converts the information entered by the user into JSON format and sends it to the server in a secure HTTP request using the SSL / TLS protocol, encrypting the data during the process.

[0736] Input: User information data

[0737] Output: Encrypted user information request

[0738] Step 3:

[0739] The server decodes the received user information and generates a prompt for the AI ​​model, which includes specific instructions for generating scenery based on the user's preferences and desired time period.

[0740] Input: Encrypted user information request

[0741] Output: prompt statement

[0742] Example prompt sentence:

[0743] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[0744] Step 4:

[0745] The server uses a generative AI model to generate corresponding scenery images and sound data based on the prompt sentence, which uses a large-scale neural network to generate scenery data in real time.

[0746] Input: prompt statement

[0747] Output: Landscape image data, sound data

[0748] Step 5:

[0749] The server encodes the generated scenery video and audio data as an H.264 video stream and streams it to the device. The audio data is encoded in AAC format and synchronized with the video stream.

[0750] Input: landscape image data, sound data

[0751] Output: Encoded video stream, audio stream

[0752] Step 6:

[0753] The server generates smell and wind setting data corresponding to the landscape in JSON format and sends it to the terminal.

[0754] Input: User information, landscape image data, sound data

[0755] Output: Smell data, wind data

[0756] Example of scent data (JSON format):

[0757] json

[0758] {

[0759] "scent": "sea_breeze",

[0760] "wind_level": 2

[0761] }

[0762] Step 7:

[0763] The device decodes the received video stream and displays it on the display. At the same time, it decodes the sound data and plays it from the speaker. It also sends the command "sea_breeze" to the aroma diffuser, causing the fan to operate at wind level "2". Input: Encoded video stream, sound stream, smell data, wind data

[0764] Output: Display, audio playback, scent emission, wind generation

[0765] Step 8:

[0766] The emotion engine uses a camera and facial expression recognition algorithms (e.g., OpenCV and Dlib) to obtain emotion data from the user's facial expressions, and analyzes the obtained emotion data to determine whether the user is relaxed or not.

[0767] Input: Camera video data

[0768] Output: Emotion data

[0769] Step 9:

[0770] The emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind parameters based on the analysis results of the emotion engine. For example, if the user is not relaxed, the scenery will be changed to a "calm lake" and the music will be changed to a "quiet piano melody."

[0771] Input: Emotion data, scenery, sound, smell, wind parameters

[0772] Output: Adjusted scenery, sound, smell, and wind parameters

[0773] (Application example 2)

[0774] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0775] When seeking relaxation and peace of mind, modern users seek customized, holistic experiences based not only on visual elements but also on hearing, smell, touch, and even emotional state. However, existing systems face challenges in integrating these multiple sensory elements and dynamically adjusting them based on individual users' emotions and feedback. In particular, there is a need for systems that can analyze a user's emotional state in real time and make appropriate adjustments in virtual experiences aimed at stress reduction and relaxation.

[0776] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0777] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound reproduction means for reproducing a sound corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating a wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a distribution means for streaming the generated scenery, an emotion analysis means for analyzing the user's emotions, and an emotion-based adjustment means for dynamically adjusting the scenery based on the emotion analysis. This makes it possible to provide a comprehensive and dynamically customized scenery experience for each individual user, thereby realizing user relaxation and stress reduction.

[0778] "Generation means" refers to the devices and techniques used to generate a landscape.

[0779] "Display means" refers to a display or screen for displaying the generated scenery.

[0780] The "sound reproduction means" refers to a speaker or an audio reproduction device for reproducing sounds that correspond to the scenery.

[0781] The "scent generating means" is a device such as an aroma diffuser that generates a scent that matches the scenery.

[0782] The "wind generating means" refers to a fan or air blower for generating wind that matches the scenery.

[0783] The "acquisition means" refers to a sensor or input device for acquiring user information.

[0784] The "control means" is a computer or software for controlling the generation of scenery based on the acquired user information.

[0785] "Delivery means" refers to the network technology and servers used to stream the generated scenery in real time.

[0786] "Emotion analysis means" refers to facial recognition technology and sensors for analyzing the user's emotions.

[0787] An "emotion-based adjustment" is a system or algorithm for dynamically adjusting a scene based on emotion analysis.

[0788] The "feedback acquisition means" is a device or interface for collecting feedback from users.

[0789] "Feedback analysis means" refers to technology or software that analyzes the obtained feedback and reflects it in the next landscape generation.

[0790] A "generative AI model" is an artificial intelligence model that generates landscapes in real time based on user information and input data.

[0791] A "prompt sentence" is an input sentence or command used to give instructions to a generative AI model.

[0792] To implement the present invention, the system is composed of the following elements: a server, a terminal, and a user play key roles.

[0793] 1. Collection of User Information

[0794] Users use their smartphones to input information such as their preferred scenery, scent preferences, and desired viewing times, etc. Based on this information, basic data is generated to provide an individually customized experience.

[0795] 2. Data processing on the server

[0796] The terminal transmits the entered user information over the Internet to a server, which receives the information, formats it appropriately, and stores it.

[0797] The server is equipped with a generative AI model that generates scenery in real time based on user information.

[0798] Example: If a user selects "seascapes at 3pm" as their preference, the generative AI model generates "seascapes at 3pm in the summer."

[0799] During this process, the AI ​​model is instructed using prompts, such as "User preferences: beach scenery at 3 PM, dislikes floral scents. Generate scenery and analyze emotions in real-time."

[0800] 3. Streaming content

[0801] The server encodes the generated scenery and corresponding sound, smell, and wind data in real time and streams them to the device.

[0802] A distribution means is used to ensure uninterrupted distribution of data.

[0803] 4. Playing data on the device

[0804] The device decodes the received data in real time, displays the scenery on the display, plays sounds using the sound playback means, emits appropriate smells using the smell generation means, and reproduces wind using the wind generation means.

[0805] Example: At 3:00 PM, as set by the user, an ocean scene is displayed on the smartphone screen, the sound of waves is played from the speaker, the aroma diffuser emits the scent of the sea, and a fan simulates a light breeze.

[0806] 5. Sentiment Analysis and Dynamic Adjustment

[0807] The device captures the user's facial expressions and movements using a camera or sensor and sends them to a server.

[0808] The emotion analysis means of the server analyzes the emotion of the user, and the emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind to suit the emotion of the user.

[0809] Example: If the user is not relaxed, the scenery or music changes to a calming one.

[0810] As a result, this system is able to provide a comprehensive relaxation experience that is individually customized for each user and adapted to their emotional state.

[0811] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0812] Step 1:

[0813] Entering user information

[0814] Users launch the smartphone app and enter information such as their favorite scenery, least favorite smells, and desired display time.

[0815] Input: User preferences and settings (e.g., "I want to see ocean views at 3pm").

[0816] Output: User information is retrieved and passed to the next processing step.

[0817] Step 2:

[0818] Sending user information

[0819] The terminal transmits the acquired user information to a server via the Internet.

[0820] Input: User information obtained in step 1.

[0821] Output: User information is sent to the server and stored on the server.

[0822] Step 3:

[0823] Prompt generation for landscape generation

[0824] The server creates a prompt sentence that gives instructions to the generative AI model based on the received user information.

[0825] Input: User information (e.g. "I want to see a seascape at 3pm").

[0826] Output: A prompt statement is generated (e.g., "User preferences: beach scenery at 3 PM, dislikes floral scents.").

[0827] Step 4:

[0828] Scenery generation

[0829] The server's generative AI model generates scenery in real time based on the generated prompt sentence.

[0830] Input: The prompt statement.

[0831] Output: Generated landscape data and associated sound, smell, and wind data.

[0832] Step 5:

[0833] Encoding and Streaming Data

[0834] The server encodes the generated landscape data in real time and streams it to the terminal.

[0835] Input: Generated landscape data, sound data, smell data, and wind data.

[0836] Output: The encoded data is delivered to the device.

[0837] Step 6:

[0838] Data decoding and playback

[0839] The device decodes the received data, displays the scenery on the screen, plays sound from the speaker, and controls an aroma diffuser or fan to recreate scents and breezes.

[0840] Input: Encoded scenery data, sound data, smell data, and wind data.

[0841] Output: Scenery displayed on the screen, audio played from the speaker, scent emitted from the aroma diffuser, and wind generated by the fan.

[0842] Step 7:

[0843] User Emotion Recognition

[0844] The device uses a camera and sensors to capture the user's facial expressions and movements, and sends the data to a server.

[0845] Input: User facial expression and movement data.

[0846] Output: The acquired emotion data is sent to the server.

[0847] Step 8:

[0848] Sentiment analysis and landscape adjustment

[0849] The server analyzes the user's emotional state using an emotion analysis means, and dynamically adjusts the scenery, sounds, smells, and wind using an emotion-based adjustment means.

[0850] Input: The submitted emotion data.

[0851] Output: New scenery, sound, smell, and wind data adjusted based on the user's emotions.

[0852] Step 9:

[0853] Get feedback and incorporate it into the next generation

[0854] The device receives feedback from the user and sends it to the server, which analyzes the feedback and reflects it in the next landscape generation.

[0855] Input: User feedback information (e.g. satisfaction level, text comments).

[0856] Output: The analyzed feedback data is reflected in the next prompt sentence of the generative AI model.

[0857] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0858] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0859] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0860] [Third embodiment]

[0861] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0862] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0863] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0864] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0865] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0866] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0867] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0868] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0869] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0870] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0871] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0872] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0873] The present invention relates to a system that generates scenery and provides users with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, and a time and season change generation means.

[0874] System configuration

[0875] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[0876] 2. The display means is a display or screen for displaying the generated scenery.

[0877] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[0878] 4. The scent generating means is a device such as an aroma diffuser that generates scents that correspond to the scenery.

[0879] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[0880] 6. Acquisition means is a technology and device for acquiring user information, such as the user's age, gender, favorite scenery, least favorite smell, and desired display time period.

[0881] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[0882] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[0883] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[0884] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[0885] Program processing flow

[0886] Collecting user information

[0887] The user accesses the settings screen of the digital photo frame or digital window and enters the following information:

[0888] Age, gender

[0889] Favorite scenery (e.g., ocean, mountains, forest)

[0890] Disliked smells (e.g. pollen, smoke)

[0891] Desired display time period (e.g. 3pm)

[0892] Send data to the server

[0893] The terminal transmits the input user information to the server.

[0894] Scenery generation using generative AI

[0895] The server uses AI to generate the requested scenery based on the received user information, including changes according to the season and time of day.

[0896] Streaming video and audio

[0897] The server streams the generated landscape video and sound data to the terminal.

[0898] Smell and wind control

[0899] The server also transmits information about the smell and wind corresponding to the scenery to the device, which then controls the aroma diffuser and fan to recreate the smell and wind in real time.

[0900] Specific examples

[0901] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user gives feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound.

[0902] This system allows people who are unable to go outside to feel the change in seasons and time, thereby reducing stress.

[0903] The processing flow will be explained below.

[0904] Step 1:

[0905] Users enter information such as personal information (e.g., age, gender), preferred landscape type (e.g., ocean, mountain, forest), disliked smells (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.) on the settings screen of a digital photo frame or digital window.

[0906] Step 2:

[0907] The terminal acquires the entered user information and transmits it to the server, properly formatting the data and implementing the necessary security measures for transmission.

[0908] Step 3:

[0909] The server analyzes the received user information and extracts necessary parameters, such as favorite scenery, least favorite smells, and desired display time, and passes these to the generation AI as parameters.

[0910] Step 4:

[0911] The server's AI generates the best scenery for the user based on the parameters passed in. For example, if a user selects "sea" at 3pm in the summer, the scenery will be generated to show blue skies and rippling seas.

[0912] Step 5:

[0913] The server streams the generated landscape video and corresponding audio data to the device. The server encodes the video and audio data and prepares them for distribution in real time.

[0914] Step 6:

[0915] The device decodes the video and audio data received from the server and plays the audio from the speaker while displaying it on the screen. Specifically, you can hear the sound of ocean waves and see an image of the ocean lapping on the screen.

[0916] Step 7:

[0917] The server also sends the smell and wind data corresponding to the generated scenery to the device, including, for example, the "smell of the sea" and a "light sea breeze."

[0918] Step 8:

[0919] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. Specifically, the aroma diffuser emits the scent of the sea and the fan recreates a light breeze.

[0920] Step 9:

[0921] After the experience, users input their feedback into the device, including their impressions of the scenery, smells, and wind strength, as well as suggestions for improvement.

[0922] Step 10:

[0923] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[0924] Step 11:

[0925] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation, such as slightly weakening the wind strength.

[0926] This series of steps allows users to have a customized real-time scenery experience, allowing them to enjoy the changes in nature and the beauty of the scenery even when it is difficult to go outside.

[0927] Example 1

[0928] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0929] Conventional visual experience systems lack the ability to coordinate various senses, such as displaying scenery, reproducing sound, generating smells, and generating wind, making it difficult to provide users with a comprehensive and immersive experience. It is also difficult to customize the experience based on individual user preferences and feedback. Furthermore, they lack the functionality to reflect changes in real time according to the season, weather, and time of day. This has created technical challenges for improving user satisfaction.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0931] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound playback means for playing sounds corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a transmission means for streaming the generated scenery and sound, and an odor and wind control means for generating odor and wind control information corresponding to the scenery and transmitting it to the terminal. This enables a comprehensive, immersive experience that reflects the user's individual preferences and real-time changes in season and time zone.

[0932] The "means for generating a scene" refers to a device and technology for generating a visual scene based on user information.

[0933] The "display means for displaying the generated scenery" refers to a display or screen for displaying the generated scenery in a form that can be visually recognized by the user, and a control device for the display or screen.

[0934] "Sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or sound reproduction device for reproducing sounds that match the generated scenery.

[0935] The "odor generating means for generating an odor corresponding to a scenery" refers to an aroma diffuser or a fragrance generating device for generating various odors corresponding to a scenery.

[0936] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind with strength and direction according to the scenery.

[0937] "Means for acquiring user information" refers to devices and technologies for collecting information such as the user's age, gender, preferences, disliked smells, and desired display time period.

[0938] The "control means for generating a landscape based on acquired user information" refers to a control device and technology for controlling the generation means based on acquired user information and generating an appropriate landscape.

[0939] The "transmission means for streaming the generated scenery and sound" refers to a technique and device for transmitting the generated scenery video and sound data to a terminal in real time.

[0940] "Smell and wind control means for generating smell and wind control information corresponding to a landscape and transmitting it to a terminal" refers to technology and devices for transmitting control signals to a terminal based on smell and wind information corresponding to the generated landscape.

[0941] The "feedback acquisition means for acquiring feedback from users" refers to devices and techniques for collecting evaluations and comments entered by users after their experiences.

[0942] The "feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation" refers to a device and technology for analyzing the collected feedback and adjusting the next generation parameters.

[0943] "Generative AI means for generating scenery using a generative AI model" refers to technology and devices for generating appropriate scenery using a generative AI model based on user information.

[0944] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and equipment for changing the landscape while reflecting real-time information such as season, weather, and time of day.

[0945] The present invention relates to a system that generates scenery and provides a user with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, a transmission means, and a scent and wind control means.

[0946] Collecting user information

[0947] The user accesses the settings screen of the digital photo frame or digital window and enters information such as age, gender, favorite scenery (e.g., ocean, mountain, forest), least favorite smell (e.g., pollen, smoke), desired display time (e.g., 3:00 p.m.), etc. The entered information is packaged in JSON format by the device and sent to the server.

[0948] Scenery generation using generative AI

[0949] The server uses a generation AI to generate scenery based on the received user information. Specifically, it sends a request to the generation AI model to generate scenery based on the user's preferences, season, and time of day. The generated scenery is saved on the server as a video file.

[0950] Prompt Sentence Examples

[0951] "Generate a 'seascape and wave sounds at 3pm in the summer' based on user information. Adjust the settings to avoid unpleasant smells and provide a satisfying experience for the user."

[0952] Streaming video and audio

[0953] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and streams it to the device in real time. The device then displays the received data on its display and plays the sound through its speakers.

[0954] Smell and wind control

[0955] The server also generates information about the scent and wind corresponding to the scenery and sends it to the device. The device analyzes the received information and controls the aroma diffuser and fan. The diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[0956] Collecting and analyzing feedback

[0957] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. The device then sends this feedback in JSON format to the server. The server analyzes the feedback and makes adjustments to reflect it in the next landscape generation. The results of the feedback analysis are saved in the user profile and used again.

[0958] Specific examples

[0959] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user can give feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound. This system allows people who are unable to go outside to feel the change of seasons and time, thereby reducing stress.

[0960] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0961] Step 1:

[0962] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, favorite scenery, least favorite smell, and desired viewing time. This input is done using a touch panel or keyboard and is confirmed by pressing the "Submit" button. The input information is structured as JSON-formatted data, as it will be used in subsequent processes to generate scenery.

[0963] (Input) User information (age, gender, favorite scenery, disliked smell, display time zone)

[0964] (Output) User information data in JSON format

[0965] Step 2:

[0966] The terminal sends the entered user information to the server. Specifically, it transfers the user information structured in JSON format to the server as an HTTP request. The server receives this request and saves the user information in a database.

[0967] (Input) JSON format user information data

[0968] (Output) The HTTP request sent to the server

[0969] Step 3:

[0970] The server uses a generative AI to generate scenery based on the received user information. Specifically, it generates prompts based on the user's preferences, season, and time of day, and inputs these into the generative AI model. The generative AI model generates related scenery data based on these prompts and returns the results to the server. The server saves this as a video file.

[0971] (Input) JSON format user information data

[0972] (Output) Generated landscape video file

[0973] Step 4:

[0974] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and sends it to the device as an HTTP stream. The device receives this in real time, displays it on the display, and plays the sound through the speaker.

[0975] (Input) Generated landscape video file

[0976] (Output) Video displayed on the display and sound played from the speakers

[0977] Step 5:

[0978] The server generates scent and wind information corresponding to the scenery and sends it to the terminal. Specifically, it generates scent and wind control information corresponding to the scenery in XML format and sends it to the terminal. The terminal analyzes this information and controls the aroma diffuser and fan. The aroma diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[0979] (Input) Generated scenery, smell, and wind control information

[0980] (Output) Emitted scent and generated wind

[0981] Step 6:

[0982] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. Feedback is given using a touch panel or keyboard and is confirmed by pressing the "Send" button. The input feedback is structured in JSON format and sent from the device to the server.

[0983] (Input) Feedback information (evaluation of scenery, sounds, smells, and wind)

[0984] (Output) Feedback data in JSON format

[0985] Step 7:

[0986] The server analyzes the user's feedback and reflects it in the next landscape generation. Specifically, it uses an analytical algorithm to analyze the feedback data and adjust the next generation parameters. The adjustment results are saved in the user's profile and used again.

[0987] (Input) Feedback data in JSON format

[0988] (Output) Adjusted generation parameter data

[0989] (Application example 1)

[0990] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0991] While conventional landscape generation systems can provide users with visual, auditory, olfactory, and tactile experiences, they lack the ability to dynamically present information and provide feedback based on the user's real-time interests and preferences. In particular, in brick-and-mortar stores, when a user shows interest in a particular product, it is necessary to detect that interest in real time and provide related information and experiences. Current systems struggle to meet this demand. Furthermore, their ability to reflect feedback in the next experience is limited, lacking the flexibility to improve the quality of the user experience.

[0992] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0993] In this invention, the server includes a visual display means, which is a display device for providing a user with a real-time scenery experience, an information display means for displaying related information during the experience, and sensor technology for tracking the user's gaze or detecting their interest. This allows scenery and related information based on the user's interests and preferences to be provided in real time, and feedback can be reflected in the scenery generation. This allows users to enjoy a realistic experience while interacting with products in a physical store, and further improves the quality of the experience through feedback.

[0994] "Generation means for generating landscapes" refers to technology and devices that generate landscapes in real time using generative AI models based on user information.

[0995] The "display means for displaying the generated scenery" is a device including a display or screen for visually presenting the generated scenery to the user.

[0996] The "sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or an audio reproduction device for providing the user with sounds related to the generated scenery.

[0997] The "odor generating means for generating an odor corresponding to the scenery" is a device such as an aroma diffuser for reproducing an odor associated with the generated scenery.

[0998] The "wind generating means for generating wind according to the scenery" is a fan or air blowing device for providing the user with wind related to the generated scenery.

[0999] The "means for acquiring user information" refers to the technology and devices for obtaining information such as the user's age, sex, and preferences.

[1000] The "control means for generating a landscape based on acquired user information" refers to a technique and device for managing the process of generating a landscape based on acquired user information.

[1001] The "visual display means, which is a display device for providing a user with a real-time scenery experience" is a device that displays the generated scenery in real time using a wearable device such as smart glasses.

[1002] "Information display means for displaying relevant information during an experience" refers to a device including a display or screen for providing additional information to the user while experiencing a scene.

[1003] "Sensor technology for tracking a user's gaze or detecting their interest" refers to a sensor technology for detecting the direction of a user's gaze or interest in real time.

[1004] The "feedback acquisition means for acquiring feedback from users" refers to techniques and devices for collecting impressions and opinions from users after the experience.

[1005] The "feedback analysis means for analyzing the feedback and reflecting it in the next scenery generation" refers to a technique and device for analyzing the acquired user feedback and reflecting it in the next scenery generation.

[1006] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and devices for changing the content of a landscape according to season, weather, and time of day in landscape generation.

[1007] The present invention relates to a system that generates scenery in real time based on user information and provides visual, auditory, olfactory, and tactile experiences. This system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, a user information acquisition means, a control means based on the acquired information, a visual display means, an information display means, sensor technology, a feedback acquisition means, a feedback analysis means, and a time / season change generation means.

[1008] System hardware and software configuration

[1009] Scenery generation method: A generative AI model (e.g., DALL-E or GPT-3) is used to generate scenery based on user information in real time.

[1010] Display means: Smart glasses or a display are used to display the generated scenery.

[1011] Sound reproduction means: sound reproducers and speakers are used to reproduce sounds associated with the scenery.

[1012] Scent generation method: An aroma diffuser is used to recreate the scent.

[1013] Wind generation means: A fan is used to recreate the wind.

[1014] Acquisition means: A sensor or input device is used to acquire user information.

[1015] Control means: Software and a control device for controlling each device based on the acquired user information.

[1016] Visual display means: Displayed in real time using smart glasses.

[1017] Information display means: A display for showing additional information.

[1018] Sensor technology: Eye-tracking sensors to detect user gaze and interest.

[1019] Feedback acquisition means: A feedback system for collecting user feedback.

[1020] Feedback analysis method: Technology that analyzes feedback and reflects it in the next landscape generation.

[1021] Time and season change generation method: Technology that generates scenery changes according to the season and time of day.

[1022] Processing flow and specific examples

[1023] 1. Collecting user information: The user enters information such as their age, gender, and favorite scenery through the smart glasses interface. For example, the user might enter, "I'm a 30-year-old woman, I like forest scenery, I dislike the smell of smoke, and I'd like to see the display in the afternoon."

[1024] 2. Scenery Generation: The server generates a scene using the generative AI model based on the acquired user information. It uses the following prompts:

[1025] The user is a 30-year-old woman who likes forest scenery. Generate a scenery that is displayed in the afternoon. The user dislikes the smell of smoke.

[1026] The generative AI model generates a scene based on these prompts and displays it in real time.

[1027] 3. Display of scenery and related information: The smart glasses display the generated scenery in real time, the sound player plays sounds related to the scenery (e.g., birdsong in the forest), the scent generator reproduces the natural scent of the forest, and the fan provides a gentle breeze.

[1028] 4. Feedback collection and analysis: The eye-tracking sensor tracks the user's gaze and identifies the object of interest. At the same time, the feedback acquisition means collects feedback from the user, and the feedback analysis means analyzes it to reflect it in the next experience. For example, if the feedback is "The wind is a little too strong," the wind setting will be adjusted for the next experience.

[1029] This system will enable users to enjoy scenery and related information that matches their preferences in real time at specific locations such as brick-and-mortar stores, improving the quality of their experience.

[1030] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1031] Step 1:

[1032] Collection of User Information

[1033] Through the smart glasses' interface, users input information such as their age, gender, favorite scenery, disliked smells, and desired viewing time. This input information is saved on the device and sent to the server. The input data includes age, gender, favorite scenery type (e.g., forest, ocean, mountain, etc.), disliked smell (e.g., smoke, pollen, etc.), and desired viewing time. This prepares the system to generate scenery tailored to the user's preferences and characteristics.

[1034] Step 2:

[1035] Prompt creation for landscape generation

[1036] The server takes user information and creates a prompt to input that information into the generative AI model. The input data includes the user's age, gender, favorite type of scenery, least favorite smell, and desired viewing time, and is formatted as the prompt sentence as follows:

[1037] "The user is a 30-year-old woman who likes forest scenery. Generate a scenery that will be displayed in the afternoon. The user dislikes the smell of smoke."

[1038] This prompt is passed to the generative AI model and serves as the basis for generating the landscape.

[1039] Step 3:

[1040] Scenery generation

[1041] The server inputs the prompt sentence into a generative AI model, which generates a scene in real time. The generative AI model (e.g., DALL-E or GPT-3) generates image data, sound data, smell information, and wind information of the scene based on the prompt sentence. The input data is the prompt sentence, and the output is an image of the generated scene, sounds related to the scene, smell information, and wind information. This allows a scene to be generated in real time according to the user's preferences and wishes.

[1042] Step 4:

[1043] Display of scenery and related information

[1044] The device receives the generated scenery data and displays it on the smart glasses. At the same time, it sends sound data to the audio player to play sounds related to the scenery. It also sends scent information to the aroma diffuser to generate the associated scent, and sends wind information to the fan to recreate the wind. The input data are the scenery image, sound data, scent information, and wind information, and the output is the scenery displayed on the smart glasses, the sound played from the audio player, the scent emitted from the aroma diffuser, and the wind blowing from the fan. This allows the user to enjoy an immersive experience.

[1045] Step 5:

[1046] Gathering feedback

[1047] After the experience, users input their impressions and opinions through a feedback screen. This feedback information is saved on the device and sent to the server. The input data includes the user's evaluation of the appropriateness of the scenery, the quality of the sound, the strength of the smell, and the strength of the wind. This allows feedback based on the user's experience to be collected.

[1048] Step 6:

[1049] Analyzing and incorporating feedback

[1050] The server analyzes the collected feedback information and reflects it in the next landscape generation. The analyzed data is feedback information from the user and is reflected in the generation AI model. For example, if there is feedback that "the wind is a little too strong," the wind strength will be adjusted the next time wind is generated. The output is an improved landscape generation process, which improves the user's experience from the next time onwards.

[1051] Through the input and output data and specific actions at each step, users can have a more personalized and immersive experience.

[1052] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1053] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, an emotion engine, an emotion-based adjustment means, and an emotion data analysis means.

[1054] System configuration

[1055] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[1056] 2. The display means is a display or screen for displaying the generated scenery.

[1057] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[1058] 4. The scent generating means is a device such as an aroma diffuser that generates a scent that corresponds to the scenery.

[1059] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[1060] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[1061] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[1062] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[1063] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[1064] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[1065] 11. Emotion engines are technologies and devices for recognizing user emotions, for example, by using facial recognition technology or sensors.

[1066] 12. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[1067] 13. Emotion data analysis means is a technique and device for acquiring and analyzing the user's emotion data recognized by the emotion engine.

[1068] Program processing flow

[1069] Collecting user information

[1070] The user inputs information such as age, sex, favorite scenery, least favorite smell, and desired display time period on the setting screen of the digital photo frame or digital window.

[1071] Send data to the server

[1072] The terminal sends the entered user information to the server, where the data is formatted appropriately and transmitted via secure communication.

[1073] Scenery generation using generative AI

[1074] The server generates the requested scenery using the received user information and generation AI. For example, if you generate a seascape at 3 pm in the summer, a blue sky and rippling seascape will be generated.

[1075] Streaming video and audio

[1076] The server then streams the generated landscape video and corresponding sound data to the device. The data is encoded and delivered in real time.

[1077] Smell and wind control

[1078] The server also sends the device smell and wind data corresponding to the generated scenery, including, for example, the "smell of the sea" and "light sea breeze."

[1079] Reproduction of video, sound, smell, and wind

[1080] The device decodes the video and audio data received from the server, displays it on the display, plays the audio from the speaker, and also emits scents from an aroma diffuser and generates wind with a fan.

[1081] User Emotion Recognition

[1082] The emotion engine uses sensors and cameras to recognize the user's emotions, for example by analyzing the user's facial expressions to identify emotions.

[1083] Dynamic landscape adjustment

[1084] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's recognized emotion, for example, changing the scenery to a calming one if the user is not relaxed.

[1085] Specific examples

[1086] The user inputs settings such as "I like ocean views, I don't like the smell of pollen, and I want it to start at 3 p.m.", and the device sends these to the server. The server uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it sends data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more soothing ones.

[1087] This system allows users who are unable to go outside to experience a real-time landscape that is customized to their emotions, which is expected to reduce stress.

[1088] The processing flow will be explained below.

[1089] Step 1:

[1090] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, preferred scenery (e.g., ocean, mountain, forest), least preferred smell (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.).

[1091] Step 2:

[1092] The device takes the entered user information, formats it, and sends it to the server, where it is encrypted to ensure data privacy and security.

[1093] Step 3:

[1094] The server analyzes the received user information and extracts necessary parameters (such as type of scenery, scent preferences, and display time period).

[1095] Step 4:

[1096] The server uses generative AI to generate scenery. For example, for a user who wants a seascape at 3 pm on a summer day, it generates a video with blue skies and rippling sea.

[1097] Step 5:

[1098] The server encodes the generated landscape video and corresponding sound data (e.g., the sound of waves) and streams them to the terminal.

[1099] Step 6:

[1100] The device decodes the received video and audio data and plays the audio from the speaker while displaying it on the screen. Specifically, the device displays an ocean scene on the screen and plays the sound of lapping waves.

[1101] Step 7:

[1102] The server also simultaneously transmits data on the smell (e.g., the smell of the sea) and wind (e.g., a light sea breeze) corresponding to the generated scenery to the terminal.

[1103] Step 8:

[1104] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. For example, the aroma diffuser can emit the scent of the sea and the fan can reproduce a light breeze.

[1105] Step 9:

[1106] The emotion engine uses cameras and sensors to analyze the user's facial expressions and biometric data to recognize the user's emotions, for example, detecting stress signals from the user's facial expressions.

[1107] Step 10:

[1108] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's emotion data recognized by the emotion engine, for example, if the user is not relaxed, the scenery is changed to be more calm and relaxing.

[1109] Step 11:

[1110] After the experience, users input their feedback into the device, including their impressions of the scenery, sounds, smells, and wind strength, as well as suggestions for improvement.

[1111] Step 12:

[1112] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[1113] Step 13:

[1114] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation (e.g., adjusting wind strength or scent intensity).

[1115] This series of steps allows users to enjoy a customized, real-time landscape experience, allowing them to enjoy the beauty of nature and its changing scenery even when it is difficult to go outside. Furthermore, the system dynamically adjusts according to the user's emotional state, providing a more personalized relaxation experience.

[1116] Example 2

[1117] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1118] Many users today are facing difficulties in going outside, resulting in fewer opportunities to experience nature and scenery. Furthermore, existing systems lack the ability to customize experiences based on the user's emotional state, making it difficult to provide a comprehensive experience that is tailored in real time based on the user's individual preferences and emotions.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1120] In this invention, the server includes a generating means for generating a scenery, a display means for displaying the generated scenery, a sound reproducing means for reproducing a sound corresponding to the scenery, an odor generating means for generating an odor corresponding to the scenery, a wind generating means for generating a wind corresponding to the scenery, a means for acquiring user information, a control means for generating a scenery based on the acquired user information, an emotion engine for recognizing the user's emotion, an emotion-based adjustment means for dynamically adjusting the scenery, sound, odor, and wind content based on the recognized user emotion, and a technology for generating a scenery based on the user information using a generative AI model. This enables a scenery experience customized in real time according to the user's emotions and preferences.

[1121] The "means for generating scenery" refers to a technique and device for generating scenery in real time based on user information.

[1122] The "display means for displaying the generated scenery" refers to a display or screen for visually presenting the generated scenery to the user.

[1123] The "sound reproduction means for reproducing sounds corresponding to the scenery" refers to a speaker or an audio reproduction device for reproducing sounds corresponding to the generated scenery.

[1124] The "odor generating means for generating an odor corresponding to a scenery" is a device such as an aroma diffuser for generating an odor corresponding to the generated scenery.

[1125] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind corresponding to the generated scenery.

[1126] The "means for acquiring user information" refers to technology and devices for collecting user information such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[1127] The "control means for generating a scene based on acquired user information" refers to a control device and technology for generating a scene based on acquired user information.

[1128] An "emotion engine that recognizes user emotions" is a technology and device that analyzes the user's facial expressions and data from sensors to recognize the user's emotions.

[1129] "Emotion-based adjustment means for dynamically adjusting the content of scenery, sounds, smells, and wind based on the recognized user emotions" refers to technology and devices that adjust the type and strength of scenery, sounds, smells, and wind in real time according to the user emotions recognized by the emotion engine.

[1130] "Technology for generating scenery based on user information using a generative AI model" is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[1131] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, an emotion engine, an emotion-based adjustment means, and a technology that uses a generative AI model.

[1132] System configuration

[1133] 1. The landscape generation means is a technology and device that uses generative AI to generate landscapes in real time based on user information. For example, the generative AI model generates a seascape of summer at 3:00 p.m. based on the user's preferences and desired time of day.

[1134] 2. The display means is a display or screen for displaying the generated scenery. It is installed in a position that is easy for the user to see and provides high-resolution images.

[1135] 3. The sound reproduction means is a speaker or sound reproduction device that reproduces sounds that match the scenery, such as the sound of waves or birds chirping.

[1136] 4. The scent generating means is a device such as an aroma diffuser that generates scents according to the scenery. For example, it emits scents related to the scenery, such as the "scent of the sea" or the "scent of flowers."

[1137] 5. The wind generating means is a fan or blower for generating wind according to the scenery. For example, a fan for reproducing a light sea breeze is used.

[1138] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[1139] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[1140] 8. Emotion engines are technologies and devices for recognizing user emotions, for example, using facial recognition technology and sensors.

[1141] 9. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[1142] 10. The technology for generating scenery based on user information using a generative AI model is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[1143] Specific examples

[1144] The user inputs their preferences, such as "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m.", and the device sends these to the server. The server then uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams this to the device. At the same time, it transmits data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more gentle ones. This allows the user to experience a personalized, relaxing experience.

[1145] Example prompt for a generative AI model:

[1146] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[1147] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1148] Step 1:

[1149] The user inputs information such as age, gender, favorite scenery, least favorite smell, desired display time, etc. on the touch screen of the digital photo frame. The input information is in the following format:

[1150] Input: Age, gender, favorite scenery, disliked smell, desired display time

[1151] Output: User information data (json format)

[1152] Step 2:

[1153] The device converts the information entered by the user into JSON format and sends it to the server in a secure HTTP request using the SSL / TLS protocol, encrypting the data during the process.

[1154] Input: User information data

[1155] Output: Encrypted user information request

[1156] Step 3:

[1157] The server decodes the received user information and generates a prompt for the AI ​​model, which includes specific instructions for generating scenery based on the user's preferences and desired time period.

[1158] Input: Encrypted user information request

[1159] Output: prompt statement

[1160] Example prompt sentence:

[1161] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[1162] Step 4:

[1163] The server uses a generative AI model to generate corresponding scenery images and sound data based on the prompt sentence, which uses a large-scale neural network to generate scenery data in real time.

[1164] Input: prompt statement

[1165] Output: Landscape image data, sound data

[1166] Step 5:

[1167] The server encodes the generated scenery video and audio data as an H.264 video stream and streams it to the device. The audio data is encoded in AAC format and synchronized with the video stream.

[1168] Input: landscape image data, sound data

[1169] Output: Encoded video stream, audio stream

[1170] Step 6:

[1171] The server generates smell and wind setting data corresponding to the landscape in JSON format and sends it to the terminal.

[1172] Input: User information, landscape image data, sound data

[1173] Output: Smell data, wind data

[1174] Example of scent data (JSON format):

[1175] json

[1176] {

[1177] "scent": "sea_breeze",

[1178] "wind_level": 2

[1179] }

[1180] Step 7:

[1181] The device decodes the received video stream and displays it on the display. At the same time, it decodes the sound data and plays it from the speaker. It also sends the command "sea_breeze" to the aroma diffuser, causing the fan to operate at wind level "2". Input: Encoded video stream, sound stream, smell data, wind data

[1182] Output: Display, audio playback, scent emission, wind generation

[1183] Step 8:

[1184] The emotion engine uses a camera and facial expression recognition algorithms (e.g., OpenCV and Dlib) to obtain emotion data from the user's facial expressions, and analyzes the obtained emotion data to determine whether the user is relaxed or not.

[1185] Input: Camera video data

[1186] Output: Emotion data

[1187] Step 9:

[1188] The emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind parameters based on the analysis results of the emotion engine. For example, if the user is not relaxed, the scenery will be changed to a "calm lake" and the music will be changed to a "quiet piano melody."

[1189] Input: Emotion data, scenery, sound, smell, wind parameters

[1190] Output: Adjusted scenery, sound, smell, and wind parameters

[1191] (Application example 2)

[1192] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1193] When seeking relaxation and peace of mind, modern users seek customized, holistic experiences based not only on visual elements but also on hearing, smell, touch, and even emotional state. However, existing systems face challenges in integrating these multiple sensory elements and dynamically adjusting them based on individual users' emotions and feedback. In particular, there is a need for systems that can analyze a user's emotional state in real time and make appropriate adjustments in virtual experiences aimed at stress reduction and relaxation.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1195] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound reproduction means for reproducing a sound corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating a wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a distribution means for streaming the generated scenery, an emotion analysis means for analyzing the user's emotions, and an emotion-based adjustment means for dynamically adjusting the scenery based on the emotion analysis. This makes it possible to provide a comprehensive and dynamically customized scenery experience for each individual user, thereby realizing user relaxation and stress reduction.

[1196] "Generation means" refers to the devices and techniques used to generate a landscape.

[1197] "Display means" refers to a display or screen for displaying the generated scenery.

[1198] The "sound reproduction means" refers to a speaker or an audio reproduction device for reproducing sounds that correspond to the scenery.

[1199] The "scent generating means" is a device such as an aroma diffuser that generates a scent that matches the scenery.

[1200] The "wind generating means" refers to a fan or air blower for generating wind that matches the scenery.

[1201] The "acquisition means" refers to a sensor or input device for acquiring user information.

[1202] The "control means" is a computer or software for controlling the generation of scenery based on the acquired user information.

[1203] "Delivery means" refers to the network technology and servers used to stream the generated scenery in real time.

[1204] "Emotion analysis means" refers to facial recognition technology and sensors for analyzing the user's emotions.

[1205] An "emotion-based adjustment" is a system or algorithm for dynamically adjusting a scene based on emotion analysis.

[1206] The "feedback acquisition means" is a device or interface for collecting feedback from users.

[1207] "Feedback analysis means" refers to technology or software that analyzes the obtained feedback and reflects it in the next landscape generation.

[1208] A "generative AI model" is an artificial intelligence model that generates landscapes in real time based on user information and input data.

[1209] A "prompt sentence" is an input sentence or command used to give instructions to a generative AI model.

[1210] To implement the present invention, the system is composed of the following elements: a server, a terminal, and a user play key roles.

[1211] 1. Collection of User Information

[1212] Users use their smartphones to input information such as their preferred scenery, scent preferences, and desired viewing times, etc. Based on this information, basic data is generated to provide an individually customized experience.

[1213] 2. Data processing on the server

[1214] The terminal transmits the entered user information over the Internet to a server, which receives the information, formats it appropriately, and stores it.

[1215] The server is equipped with a generative AI model that generates scenery in real time based on user information.

[1216] Example: If a user selects "seascapes at 3pm" as their preference, the generative AI model generates "seascapes at 3pm in the summer."

[1217] During this process, the AI ​​model is instructed using prompts, such as "User preferences: beach scenery at 3 PM, dislikes floral scents. Generate scenery and analyze emotions in real-time."

[1218] 3. Streaming content

[1219] The server encodes the generated scenery and corresponding sound, smell, and wind data in real time and streams them to the device.

[1220] A distribution means is used to ensure uninterrupted distribution of data.

[1221] 4. Playing data on the device

[1222] The device decodes the received data in real time, displays the scenery on the display, plays sounds using the sound playback means, emits appropriate smells using the smell generation means, and reproduces wind using the wind generation means.

[1223] Example: At 3:00 PM, as set by the user, an ocean scene is displayed on the smartphone screen, the sound of waves is played from the speaker, the aroma diffuser emits the scent of the sea, and a fan simulates a light breeze.

[1224] 5. Sentiment Analysis and Dynamic Adjustment

[1225] The device captures the user's facial expressions and movements using a camera or sensor and sends them to a server.

[1226] The emotion analysis means of the server analyzes the emotion of the user, and the emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind to suit the emotion of the user.

[1227] Example: If the user is not relaxed, the scenery or music changes to a calming one.

[1228] As a result, this system is able to provide a comprehensive relaxation experience that is individually customized for each user and adapted to their emotional state.

[1229] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1230] Step 1:

[1231] Entering user information

[1232] Users launch the smartphone app and enter information such as their favorite scenery, least favorite smells, and desired display time.

[1233] Input: User preferences and settings (e.g., "I want to see ocean views at 3pm").

[1234] Output: User information is retrieved and passed to the next processing step.

[1235] Step 2:

[1236] Sending user information

[1237] The terminal transmits the acquired user information to a server via the Internet.

[1238] Input: User information obtained in step 1.

[1239] Output: User information is sent to the server and stored on the server.

[1240] Step 3:

[1241] Prompt generation for landscape generation

[1242] The server creates a prompt sentence that gives instructions to the generative AI model based on the received user information.

[1243] Input: User information (e.g. "I want to see a seascape at 3pm").

[1244] Output: A prompt statement is generated (e.g., "User preferences: beach scenery at 3 PM, dislikes floral scents.").

[1245] Step 4:

[1246] Scenery generation

[1247] The server's generative AI model generates scenery in real time based on the generated prompt sentence.

[1248] Input: The prompt statement.

[1249] Output: Generated landscape data and associated sound, smell, and wind data.

[1250] Step 5:

[1251] Encoding and Streaming Data

[1252] The server encodes the generated landscape data in real time and streams it to the terminal.

[1253] Input: Generated landscape data, sound data, smell data, and wind data.

[1254] Output: The encoded data is delivered to the device.

[1255] Step 6:

[1256] Data decoding and playback

[1257] The device decodes the received data, displays the scenery on the screen, plays sound from the speaker, and controls an aroma diffuser or fan to recreate scents and breezes.

[1258] Input: Encoded scenery data, sound data, smell data, and wind data.

[1259] Output: Scenery displayed on the screen, audio played from the speaker, scent emitted from the aroma diffuser, and wind generated by the fan.

[1260] Step 7:

[1261] User Emotion Recognition

[1262] The device uses a camera and sensors to capture the user's facial expressions and movements, and sends the data to a server.

[1263] Input: User facial expression and movement data.

[1264] Output: The acquired emotion data is sent to the server.

[1265] Step 8:

[1266] Sentiment analysis and landscape adjustment

[1267] The server analyzes the user's emotional state using an emotion analysis means, and dynamically adjusts the scenery, sounds, smells, and wind using an emotion-based adjustment means.

[1268] Input: The submitted emotion data.

[1269] Output: New scenery, sound, smell, and wind data adjusted based on the user's emotions.

[1270] Step 9:

[1271] Get feedback and incorporate it into the next generation

[1272] The device receives feedback from the user and sends it to the server, which analyzes the feedback and reflects it in the next landscape generation.

[1273] Input: User feedback information (e.g. satisfaction level, text comments).

[1274] Output: The analyzed feedback data is reflected in the next prompt sentence of the generative AI model.

[1275] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1276] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1277] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1278] [Fourth embodiment]

[1279] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1280] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1281] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1282] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1283] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1284] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1285] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1286] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1287] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1288] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1289] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1290] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1291] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1292] The present invention relates to a system that generates scenery and provides users with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, and a time and season change generation means.

[1293] System configuration

[1294] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[1295] 2. The display means is a display or screen for displaying the generated scenery.

[1296] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[1297] 4. The scent generating means is a device such as an aroma diffuser that generates scents that correspond to the scenery.

[1298] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[1299] 6. Acquisition means is a technology and device for acquiring user information, such as the user's age, gender, favorite scenery, least favorite smell, and desired display time period.

[1300] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[1301] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[1302] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[1303] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[1304] Program processing flow

[1305] Collecting user information

[1306] The user accesses the settings screen of the digital photo frame or digital window and enters the following information:

[1307] Age, gender

[1308] Favorite scenery (e.g., ocean, mountains, forest)

[1309] Disliked smells (e.g. pollen, smoke)

[1310] Desired display time period (e.g. 3pm)

[1311] Send data to the server

[1312] The terminal transmits the input user information to the server.

[1313] Scenery generation using generative AI

[1314] The server uses AI to generate the requested scenery based on the received user information, including changes according to the season and time of day.

[1315] Streaming video and audio

[1316] The server streams the generated landscape video and sound data to the terminal.

[1317] Smell and wind control

[1318] The server also transmits information about the smell and wind corresponding to the scenery to the device, which then controls the aroma diffuser and fan to recreate the smell and wind in real time.

[1319] Specific examples

[1320] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user gives feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound.

[1321] This system allows people who are unable to go outside to feel the change in seasons and time, thereby reducing stress.

[1322] The processing flow will be explained below.

[1323] Step 1:

[1324] Users enter information such as personal information (e.g., age, gender), preferred landscape type (e.g., ocean, mountain, forest), disliked smells (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.) on the settings screen of a digital photo frame or digital window.

[1325] Step 2:

[1326] The terminal acquires the entered user information and transmits it to the server, properly formatting the data and implementing the necessary security measures for transmission.

[1327] Step 3:

[1328] The server analyzes the received user information and extracts necessary parameters, such as favorite scenery, least favorite smells, and desired display time, and passes these to the generation AI as parameters.

[1329] Step 4:

[1330] The server's AI generates the best scenery for the user based on the parameters passed in. For example, if a user selects "sea" at 3pm in the summer, the scenery will be generated to show blue skies and rippling seas.

[1331] Step 5:

[1332] The server streams the generated landscape video and corresponding audio data to the device. The server encodes the video and audio data and prepares them for distribution in real time.

[1333] Step 6:

[1334] The device decodes the video and audio data received from the server and plays the audio from the speaker while displaying it on the screen. Specifically, you can hear the sound of ocean waves and see an image of the ocean lapping on the screen.

[1335] Step 7:

[1336] The server also sends the smell and wind data corresponding to the generated scenery to the device, including, for example, the "smell of the sea" and a "light sea breeze."

[1337] Step 8:

[1338] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. Specifically, the aroma diffuser emits the scent of the sea and the fan recreates a light breeze.

[1339] Step 9:

[1340] After the experience, users input their feedback into the device, including their impressions of the scenery, smells, and wind strength, as well as suggestions for improvement.

[1341] Step 10:

[1342] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[1343] Step 11:

[1344] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation, such as slightly weakening the wind strength.

[1345] This series of steps allows users to have a customized real-time scenery experience, allowing them to enjoy the changes in nature and the beauty of the scenery even when it is difficult to go outside.

[1346] Example 1

[1347] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1348] Conventional visual experience systems lack the ability to coordinate various senses, such as displaying scenery, reproducing sound, generating smells, and generating wind, making it difficult to provide users with a comprehensive and immersive experience. It is also difficult to customize the experience based on individual user preferences and feedback. Furthermore, they lack the functionality to reflect changes in real time according to the season, weather, and time of day. This has created technical challenges for improving user satisfaction.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1350] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound playback means for playing sounds corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a transmission means for streaming the generated scenery and sound, and an odor and wind control means for generating odor and wind control information corresponding to the scenery and transmitting it to the terminal. This enables a comprehensive, immersive experience that reflects the user's individual preferences and real-time changes in season and time zone.

[1351] The "means for generating a scene" refers to a device and technology for generating a visual scene based on user information.

[1352] The "display means for displaying the generated scenery" refers to a display or screen for displaying the generated scenery in a form that can be visually recognized by the user, and a control device for the display or screen.

[1353] "Sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or sound reproduction device for reproducing sounds that match the generated scenery.

[1354] The "odor generating means for generating an odor corresponding to a scenery" refers to an aroma diffuser or a fragrance generating device for generating various odors corresponding to a scenery.

[1355] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind with strength and direction according to the scenery.

[1356] "Means for acquiring user information" refers to devices and technologies for collecting information such as the user's age, gender, preferences, disliked smells, and desired display time period.

[1357] The "control means for generating a landscape based on acquired user information" refers to a control device and technology for controlling the generation means based on acquired user information and generating an appropriate landscape.

[1358] The "transmission means for streaming the generated scenery and sound" refers to a technique and device for transmitting the generated scenery video and sound data to a terminal in real time.

[1359] "Smell and wind control means for generating smell and wind control information corresponding to a landscape and transmitting it to a terminal" refers to technology and devices for transmitting control signals to a terminal based on smell and wind information corresponding to the generated landscape.

[1360] The "feedback acquisition means for acquiring feedback from users" refers to devices and techniques for collecting evaluations and comments entered by users after their experiences.

[1361] The "feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation" refers to a device and technology for analyzing the collected feedback and adjusting the next generation parameters.

[1362] "Generative AI means for generating scenery using a generative AI model" refers to technology and devices for generating appropriate scenery using a generative AI model based on user information.

[1363] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and equipment for changing the landscape while reflecting real-time information such as season, weather, and time of day.

[1364] The present invention relates to a system that generates scenery and provides a user with realistic visual, auditory, olfactory, and tactile experiences. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, a transmission means, and a scent and wind control means.

[1365] Collecting user information

[1366] The user accesses the settings screen of the digital photo frame or digital window and enters information such as age, gender, favorite scenery (e.g., ocean, mountain, forest), least favorite smell (e.g., pollen, smoke), desired display time (e.g., 3:00 p.m.), etc. The entered information is packaged in JSON format by the device and sent to the server.

[1367] Scenery generation using generative AI

[1368] The server uses a generation AI to generate scenery based on the received user information. Specifically, it sends a request to the generation AI model to generate scenery based on the user's preferences, season, and time of day. The generated scenery is saved on the server as a video file.

[1369] Prompt Sentence Examples

[1370] "Generate a 'seascape and wave sounds at 3pm in the summer' based on user information. Adjust the settings to avoid unpleasant smells and provide a satisfying experience for the user."

[1371] Streaming video and audio

[1372] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and streams it to the device in real time. The device then displays the received data on its display and plays the sound through its speakers.

[1373] Smell and wind control

[1374] The server also generates information about the scent and wind corresponding to the scenery and sends it to the device. The device analyzes the received information and controls the aroma diffuser and fan. The diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[1375] Collecting and analyzing feedback

[1376] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. The device then sends this feedback in JSON format to the server. The server analyzes the feedback and makes adjustments to reflect it in the next landscape generation. The results of the feedback analysis are saved in the user profile and used again.

[1377] Specific examples

[1378] Suppose a user inputs, "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m." The device sends this information to the server, which uses a generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it also sends information about the "sea scent," and the device emits the ocean scent from the aroma diffuser while adjusting the fan to create a gentle breeze. After the experience, the user can give feedback that "the wind was a little too strong," and the device sends this feedback to the server, which then adjusts the wind strength the next time it generates a sound. This system allows people who are unable to go outside to feel the change of seasons and time, thereby reducing stress.

[1379] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1380] Step 1:

[1381] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, favorite scenery, least favorite smell, and desired viewing time. This input is done using a touch panel or keyboard and is confirmed by pressing the "Submit" button. The input information is structured as JSON-formatted data, as it will be used in subsequent processes to generate scenery.

[1382] (Input) User information (age, gender, favorite scenery, disliked smell, display time zone)

[1383] (Output) User information data in JSON format

[1384] Step 2:

[1385] The terminal sends the entered user information to the server. Specifically, it transfers the user information structured in JSON format to the server as an HTTP request. The server receives this request and saves the user information in a database.

[1386] (Input) JSON format user information data

[1387] (Output) The HTTP request sent to the server

[1388] Step 3:

[1389] The server uses a generative AI to generate scenery based on the received user information. Specifically, it generates prompts based on the user's preferences, season, and time of day, and inputs these into the generative AI model. The generative AI model generates related scenery data based on these prompts and returns the results to the server. The server saves this as a video file.

[1390] (Input) JSON format user information data

[1391] (Output) Generated landscape video file

[1392] Step 4:

[1393] Once generated, the scenery video and sound data are streamed from the server to the device. The server converts the stored video file into a streaming format and sends it to the device as an HTTP stream. The device receives this in real time, displays it on the display, and plays the sound through the speaker.

[1394] (Input) Generated landscape video file

[1395] (Output) Video displayed on the display and sound played from the speakers

[1396] Step 5:

[1397] The server generates scent and wind information corresponding to the scenery and sends it to the terminal. Specifically, it generates scent and wind control information corresponding to the scenery in XML format and sends it to the terminal. The terminal analyzes this information and controls the aroma diffuser and fan. The aroma diffuser emits the specified scent, and the fan generates wind with the specified strength and direction.

[1398] (Input) Generated scenery, smell, and wind control information

[1399] (Output) Emitted scent and generated wind

[1400] Step 6:

[1401] After the experience, the user inputs feedback about the scenery, sounds, smells, and wind into the device. Feedback is given using a touch panel or keyboard and is confirmed by pressing the "Send" button. The input feedback is structured in JSON format and sent from the device to the server.

[1402] (Input) Feedback information (evaluation of scenery, sounds, smells, and wind)

[1403] (Output) Feedback data in JSON format

[1404] Step 7:

[1405] The server analyzes the user's feedback and reflects it in the next landscape generation. Specifically, it uses an analytical algorithm to analyze the feedback data and adjust the next generation parameters. The adjustment results are saved in the user's profile and used again.

[1406] (Input) Feedback data in JSON format

[1407] (Output) Adjusted generation parameter data

[1408] (Application example 1)

[1409] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1410] While conventional landscape generation systems can provide users with visual, auditory, olfactory, and tactile experiences, they lack the ability to dynamically present information and provide feedback based on the user's real-time interests and preferences. In particular, in brick-and-mortar stores, when a user shows interest in a particular product, it is necessary to detect that interest in real time and provide related information and experiences. Current systems struggle to meet this demand. Furthermore, their ability to reflect feedback in the next experience is limited, lacking the flexibility to improve the quality of the user experience.

[1411] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1412] In this invention, the server includes a visual display means, which is a display device for providing a user with a real-time scenery experience, an information display means for displaying related information during the experience, and sensor technology for tracking the user's gaze or detecting their interest. This allows scenery and related information based on the user's interests and preferences to be provided in real time, and feedback can be reflected in the scenery generation. This allows users to enjoy a realistic experience while interacting with products in a physical store, and further improves the quality of the experience through feedback.

[1413] "Generation means for generating landscapes" refers to technology and devices that generate landscapes in real time using generative AI models based on user information.

[1414] The "display means for displaying the generated scenery" is a device including a display or screen for visually presenting the generated scenery to the user.

[1415] The "sound reproduction means for reproducing sounds according to the scenery" refers to a speaker or an audio reproduction device for providing the user with sounds related to the generated scenery.

[1416] The "odor generating means for generating an odor corresponding to the scenery" is a device such as an aroma diffuser for reproducing an odor associated with the generated scenery.

[1417] The "wind generating means for generating wind according to the scenery" is a fan or air blowing device for providing the user with wind related to the generated scenery.

[1418] The "means for acquiring user information" refers to the technology and devices for obtaining information such as the user's age, sex, and preferences.

[1419] The "control means for generating a landscape based on acquired user information" refers to a technique and device for managing the process of generating a landscape based on acquired user information.

[1420] The "visual display means, which is a display device for providing a user with a real-time scenery experience" is a device that displays the generated scenery in real time using a wearable device such as smart glasses.

[1421] "Information display means for displaying relevant information during an experience" refers to a device including a display or screen for providing additional information to the user while experiencing a scene.

[1422] "Sensor technology for tracking a user's gaze or detecting their interest" refers to a sensor technology for detecting the direction of a user's gaze or interest in real time.

[1423] The "feedback acquisition means for acquiring feedback from users" refers to techniques and devices for collecting impressions and opinions from users after the experience.

[1424] The "feedback analysis means for analyzing the feedback and reflecting it in the next scenery generation" refers to a technique and device for analyzing the acquired user feedback and reflecting it in the next scenery generation.

[1425] "Time and season change generation means for generating changes based on season, weather, and time of day" refers to technology and devices for changing the content of a landscape according to season, weather, and time of day in landscape generation.

[1426] The present invention relates to a system that generates scenery in real time based on user information and provides visual, auditory, olfactory, and tactile experiences. This system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, a user information acquisition means, a control means based on the acquired information, a visual display means, an information display means, sensor technology, a feedback acquisition means, a feedback analysis means, and a time / season change generation means.

[1427] System hardware and software configuration

[1428] Scenery generation method: A generative AI model (e.g., DALL-E or GPT-3) is used to generate scenery based on user information in real time.

[1429] Display means: Smart glasses or a display are used to display the generated scenery.

[1430] Sound reproduction means: sound reproducers and speakers are used to reproduce sounds associated with the scenery.

[1431] Scent generation method: An aroma diffuser is used to recreate the scent.

[1432] Wind generation means: A fan is used to recreate the wind.

[1433] Acquisition means: A sensor or input device is used to acquire user information.

[1434] Control means: Software and a control device for controlling each device based on the acquired user information.

[1435] Visual display means: Displayed in real time using smart glasses.

[1436] Information display means: A display for showing additional information.

[1437] Sensor technology: Eye-tracking sensors to detect user gaze and interest.

[1438] Feedback acquisition means: A feedback system for collecting user feedback.

[1439] Feedback analysis method: Technology that analyzes feedback and reflects it in the next landscape generation.

[1440] Time and season change generation method: Technology that generates scenery changes according to the season and time of day.

[1441] Processing flow and specific examples

[1442] 1. Collecting user information: The user enters information such as their age, gender, and favorite scenery through the smart glasses interface. For example, the user might enter, "I'm a 30-year-old woman, I like forest scenery, I dislike the smell of smoke, and I'd like to see the display in the afternoon."

[1443] 2. Scenery Generation: The server generates a scene using the generative AI model based on the acquired user information. It uses the following prompts:

[1444] The user is a 30-year-old woman who likes forest scenery. Generate a scenery that is displayed in the afternoon. The user dislikes the smell of smoke.

[1445] The generative AI model generates a scene based on these prompts and displays it in real time.

[1446] 3. Display of scenery and related information: The smart glasses display the generated scenery in real time, the sound player plays sounds related to the scenery (e.g., birdsong in the forest), the scent generator reproduces the natural scent of the forest, and the fan provides a gentle breeze.

[1447] 4. Feedback collection and analysis: The eye-tracking sensor tracks the user's gaze and identifies the object of interest. At the same time, the feedback acquisition means collects feedback from the user, and the feedback analysis means analyzes it to reflect it in the next experience. For example, if the feedback is "The wind is a little too strong," the wind setting will be adjusted for the next experience.

[1448] This system will enable users to enjoy scenery and related information that matches their preferences in real time at specific locations such as brick-and-mortar stores, improving the quality of their experience.

[1449] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1450] Step 1:

[1451] Collection of User Information

[1452] Through the smart glasses' interface, users input information such as their age, gender, favorite scenery, disliked smells, and desired viewing time. This input information is saved on the device and sent to the server. The input data includes age, gender, favorite scenery type (e.g., forest, ocean, mountain, etc.), disliked smell (e.g., smoke, pollen, etc.), and desired viewing time. This prepares the system to generate scenery tailored to the user's preferences and characteristics.

[1453] Step 2:

[1454] Prompt creation for landscape generation

[1455] The server takes user information and creates a prompt to input that information into the generative AI model. The input data includes the user's age, gender, favorite type of scenery, least favorite smell, and desired viewing time, and is formatted as the prompt sentence as follows:

[1456] "The user is a 30-year-old woman who likes forest scenery. Generate a scenery that will be displayed in the afternoon. The user dislikes the smell of smoke."

[1457] This prompt is passed to the generative AI model and serves as the basis for generating the landscape.

[1458] Step 3:

[1459] Scenery generation

[1460] The server inputs the prompt sentence into a generative AI model, which generates a scene in real time. The generative AI model (e.g., DALL-E or GPT-3) generates image data, sound data, smell information, and wind information of the scene based on the prompt sentence. The input data is the prompt sentence, and the output is an image of the generated scene, sounds related to the scene, smell information, and wind information. This allows a scene to be generated in real time according to the user's preferences and wishes.

[1461] Step 4:

[1462] Display of scenery and related information

[1463] The device receives the generated scenery data and displays it on the smart glasses. At the same time, it sends sound data to the audio player to play sounds related to the scenery. It also sends scent information to the aroma diffuser to generate the associated scent, and sends wind information to the fan to recreate the wind. The input data are the scenery image, sound data, scent information, and wind information, and the output is the scenery displayed on the smart glasses, the sound played from the audio player, the scent emitted from the aroma diffuser, and the wind blowing from the fan. This allows the user to enjoy an immersive experience.

[1464] Step 5:

[1465] Gathering feedback

[1466] After the experience, users input their impressions and opinions through a feedback screen. This feedback information is saved on the device and sent to the server. The input data includes the user's evaluation of the appropriateness of the scenery, the quality of the sound, the strength of the smell, and the strength of the wind. This allows feedback based on the user's experience to be collected.

[1467] Step 6:

[1468] Analyzing and incorporating feedback

[1469] The server analyzes the collected feedback information and reflects it in the next landscape generation. The analyzed data is feedback information from the user and is reflected in the generation AI model. For example, if there is feedback that "the wind is a little too strong," the wind strength will be adjusted the next time wind is generated. The output is an improved landscape generation process, which improves the user's experience from the next time onwards.

[1470] Through the input and output data and specific actions at each step, users can have a more personalized and immersive experience.

[1471] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1472] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, a feedback acquisition means, a feedback analysis means, a time and season change generation means, an emotion engine, an emotion-based adjustment means, and an emotion data analysis means.

[1473] System configuration

[1474] 1. The landscape generation means is a technology and device that uses generation AI to generate landscapes in real time based on user information.

[1475] 2. The display means is a display or screen for displaying the generated scenery.

[1476] 3. The sound reproduction means is a speaker or sound reproduction device for reproducing sounds that correspond to the scenery.

[1477] 4. The scent generating means is a device such as an aroma diffuser that generates a scent that corresponds to the scenery.

[1478] 5. The wind generating means is a fan or blower for generating wind according to the scenery.

[1479] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[1480] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[1481] 8. Feedback acquisition means are techniques and devices for acquiring feedback from users.

[1482] 9. The feedback analysis means is a technique and device for analyzing the obtained feedback and reflecting it in the next landscape generation.

[1483] 10. Time and season change generation means is a technology and device for generating scenery changes based on seasons, weather, and time of day.

[1484] 11. Emotion engines are technologies and devices for recognizing user emotions, for example, by using facial recognition technology or sensors.

[1485] 12. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[1486] 13. Emotion data analysis means is a technique and device for acquiring and analyzing the user's emotion data recognized by the emotion engine.

[1487] Program processing flow

[1488] Collecting user information

[1489] The user inputs information such as age, sex, favorite scenery, least favorite smell, and desired display time period on the setting screen of the digital photo frame or digital window.

[1490] Send data to the server

[1491] The terminal sends the entered user information to the server, where the data is formatted appropriately and transmitted via secure communication.

[1492] Scenery generation using generative AI

[1493] The server generates the requested scenery using the received user information and generation AI. For example, if you generate a seascape at 3 pm in the summer, a blue sky and rippling seascape will be generated.

[1494] Streaming video and audio

[1495] The server then streams the generated landscape video and corresponding sound data to the device. The data is encoded and delivered in real time.

[1496] Smell and wind control

[1497] The server also sends the device smell and wind data corresponding to the generated scenery, including, for example, the "smell of the sea" and "light sea breeze."

[1498] Reproduction of video, sound, smell, and wind

[1499] The device decodes the video and audio data received from the server, displays it on the display, plays the audio from the speaker, and also emits scents from an aroma diffuser and generates wind with a fan.

[1500] User Emotion Recognition

[1501] The emotion engine uses sensors and cameras to recognize the user's emotions, for example by analyzing the user's facial expressions to identify emotions.

[1502] Dynamic landscape adjustment

[1503] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's recognized emotion, for example, changing the scenery to a calming one if the user is not relaxed.

[1504] Specific examples

[1505] The user inputs settings such as "I like ocean views, I don't like the smell of pollen, and I want it to start at 3 p.m.", and the device sends these to the server. The server uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams it to the device. At the same time, it sends data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more soothing ones.

[1506] This system allows users who are unable to go outside to experience a real-time landscape that is customized to their emotions, which is expected to reduce stress.

[1507] The processing flow will be explained below.

[1508] Step 1:

[1509] Users access the settings screen of their digital photo frame or digital window and enter information such as their age, gender, preferred scenery (e.g., ocean, mountain, forest), least preferred smell (e.g., pollen, smoke), and desired display time (e.g., 3:00 p.m.).

[1510] Step 2:

[1511] The device takes the entered user information, formats it, and sends it to the server, where it is encrypted to ensure data privacy and security.

[1512] Step 3:

[1513] The server analyzes the received user information and extracts necessary parameters (such as type of scenery, scent preferences, and display time period).

[1514] Step 4:

[1515] The server uses generative AI to generate scenery. For example, for a user who wants a seascape at 3 pm on a summer day, it generates a video with blue skies and rippling sea.

[1516] Step 5:

[1517] The server encodes the generated landscape video and corresponding sound data (e.g., the sound of waves) and streams them to the terminal.

[1518] Step 6:

[1519] The device decodes the received video and audio data and plays the audio from the speaker while displaying it on the screen. Specifically, the device displays an ocean scene on the screen and plays the sound of lapping waves.

[1520] Step 7:

[1521] The server also simultaneously transmits data on the smell (e.g., the smell of the sea) and wind (e.g., a light sea breeze) corresponding to the generated scenery to the terminal.

[1522] Step 8:

[1523] The device controls the aroma diffuser and fan based on the scent and wind data received from the server. For example, the aroma diffuser can emit the scent of the sea and the fan can reproduce a light breeze.

[1524] Step 9:

[1525] The emotion engine uses cameras and sensors to analyze the user's facial expressions and biometric data to recognize the user's emotions, for example, detecting stress signals from the user's facial expressions.

[1526] Step 10:

[1527] The emotion-based adjustment means adjusts the scenery, sound, smell, and wind content in real time based on the user's emotion data recognized by the emotion engine, for example, if the user is not relaxed, the scenery is changed to be more calm and relaxing.

[1528] Step 11:

[1529] After the experience, users input their feedback into the device, including their impressions of the scenery, sounds, smells, and wind strength, as well as suggestions for improvement.

[1530] Step 12:

[1531] The terminal transmits feedback data from the user to the server, which is used to reflect the feedback data in the next scenery generation.

[1532] Step 13:

[1533] The server analyzes the received feedback data and adjusts the parameters necessary for the next landscape generation (e.g., adjusting wind strength or scent intensity).

[1534] This series of steps allows users to enjoy a customized, real-time landscape experience, allowing them to enjoy the beauty of nature and its changing scenery even when it is difficult to go outside. Furthermore, the system dynamically adjusts according to the user's emotional state, providing a more personalized relaxation experience.

[1535] Example 2

[1536] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1537] Many users today are facing difficulties in going outside, resulting in fewer opportunities to experience nature and scenery. Furthermore, existing systems lack the ability to customize experiences based on the user's emotional state, making it difficult to provide a comprehensive experience that is tailored in real time based on the user's individual preferences and emotions.

[1538] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1539] In this invention, the server includes a generating means for generating a scenery, a display means for displaying the generated scenery, a sound reproducing means for reproducing a sound corresponding to the scenery, an odor generating means for generating an odor corresponding to the scenery, a wind generating means for generating a wind corresponding to the scenery, a means for acquiring user information, a control means for generating a scenery based on the acquired user information, an emotion engine for recognizing the user's emotion, an emotion-based adjustment means for dynamically adjusting the scenery, sound, odor, and wind content based on the recognized user emotion, and a technology for generating a scenery based on the user information using a generative AI model. This enables a scenery experience customized in real time according to the user's emotions and preferences.

[1540] The "means for generating scenery" refers to a technique and device for generating scenery in real time based on user information.

[1541] The "display means for displaying the generated scenery" refers to a display or screen for visually presenting the generated scenery to the user.

[1542] The "sound reproduction means for reproducing sounds corresponding to the scenery" refers to a speaker or an audio reproduction device for reproducing sounds corresponding to the generated scenery.

[1543] The "odor generating means for generating an odor corresponding to a scenery" is a device such as an aroma diffuser for generating an odor corresponding to the generated scenery.

[1544] The "wind generating means for generating wind according to the scenery" refers to a fan or air blower for generating wind corresponding to the generated scenery.

[1545] The "means for acquiring user information" refers to technology and devices for collecting user information such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[1546] The "control means for generating a scene based on acquired user information" refers to a control device and technology for generating a scene based on acquired user information.

[1547] An "emotion engine that recognizes user emotions" is a technology and device that analyzes the user's facial expressions and data from sensors to recognize the user's emotions.

[1548] "Emotion-based adjustment means for dynamically adjusting the content of scenery, sounds, smells, and wind based on the recognized user emotions" refers to technology and devices that adjust the type and strength of scenery, sounds, smells, and wind in real time according to the user emotions recognized by the emotion engine.

[1549] "Technology for generating scenery based on user information using a generative AI model" is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[1550] The present invention relates to a system that generates scenery and provides a user with an individually customized experience based on their visual, auditory, olfactory, tactile, and even emotional state. The system includes a scenery generation means, a display means, a sound reproduction means, a scent generation means, a wind generation means, an acquisition means, a control means, an emotion engine, an emotion-based adjustment means, and a technology that uses a generative AI model.

[1551] System configuration

[1552] 1. The landscape generation means is a technology and device that uses generative AI to generate landscapes in real time based on user information. For example, the generative AI model generates a seascape of summer at 3:00 p.m. based on the user's preferences and desired time of day.

[1553] 2. The display means is a display or screen for displaying the generated scenery. It is installed in a position that is easy for the user to see and provides high-resolution images.

[1554] 3. The sound reproduction means is a speaker or sound reproduction device that reproduces sounds that match the scenery, such as the sound of waves or birds chirping.

[1555] 4. The scent generating means is a device such as an aroma diffuser that generates scents according to the scenery. For example, it emits scents related to the scenery, such as the "scent of the sea" or the "scent of flowers."

[1556] 5. The wind generating means is a fan or blower for generating wind according to the scenery. For example, a fan for reproducing a light sea breeze is used.

[1557] 6. Acquisition means is a technology and device for acquiring user information, such as age, gender, favorite scenery, least favorite smell, and desired display time period.

[1558] 7. The control means is a control device and technique for generating scenery based on the acquired user information.

[1559] 8. Emotion engines are technologies and devices for recognizing user emotions, for example, using facial recognition technology and sensors.

[1560] 9. Emotion-based adjustment means are techniques and devices for dynamically adjusting the content of scenery, sounds, smells, and wind based on an emotion engine.

[1561] 10. The technology for generating scenery based on user information using a generative AI model is a technology that uses generative AI to generate scenery images and videos using user information as input parameters.

[1562] Specific examples

[1563] The user inputs their preferences, such as "I like ocean views, I don't like the smell of pollen, and I want the display to start at 3 p.m.", and the device sends these to the server. The server then uses generation AI to generate "ocean views and the sound of waves at 3 p.m. in the summer" and streams this to the device. At the same time, it transmits data on "ocean scent" and "light breeze," and the device emits the ocean scent from an aroma diffuser and recreates a light breeze with a fan. If the emotion engine determines that the user is not relaxed, it uses emotion-based adjustment methods to change the images and music to more gentle ones. This allows the user to experience a personalized, relaxing experience.

[1564] Example prompt for a generative AI model:

[1565] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[1566] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1567] Step 1:

[1568] The user inputs information such as age, gender, favorite scenery, least favorite smell, desired display time, etc. on the touch screen of the digital photo frame. The input information is in the following format:

[1569] Input: Age, gender, favorite scenery, disliked smell, desired display time

[1570] Output: User information data (json format)

[1571] Step 2:

[1572] The device converts the information entered by the user into JSON format and sends it to the server in a secure HTTP request using the SSL / TLS protocol, encrypting the data during the process.

[1573] Input: User information data

[1574] Output: Encrypted user information request

[1575] Step 3:

[1576] The server decodes the received user information and generates a prompt for the AI ​​model, which includes specific instructions for generating scenery based on the user's preferences and desired time period.

[1577] Input: Encrypted user information request

[1578] Output: prompt statement

[1579] Example prompt sentence:

[1580] "Based on the information entered by a 30-year-old male user, generate a scene of blue skies and rippling ocean at 3pm in the summer, accompanied by the scent of the sea and a light sea breeze."

[1581] Step 4:

[1582] The server uses a generative AI model to generate corresponding scenery images and sound data based on the prompt sentence, which uses a large-scale neural network to generate scenery data in real time.

[1583] Input: prompt statement

[1584] Output: Landscape image data, sound data

[1585] Step 5:

[1586] The server encodes the generated scenery video and audio data as an H.264 video stream and streams it to the device. The audio data is encoded in AAC format and synchronized with the video stream.

[1587] Input: landscape image data, sound data

[1588] Output: Encoded video stream, audio stream

[1589] Step 6:

[1590] The server generates smell and wind setting data corresponding to the landscape in JSON format and sends it to the terminal.

[1591] Input: User information, landscape image data, sound data

[1592] Output: Smell data, wind data

[1593] Example of scent data (JSON format):

[1594] json

[1595] {

[1596] "scent": "sea_breeze",

[1597] "wind_level": 2

[1598] }

[1599] Step 7:

[1600] The device decodes the received video stream and displays it on the display. At the same time, it decodes the sound data and plays it from the speaker. It also sends the command "sea_breeze" to the aroma diffuser, causing the fan to operate at wind level "2". Input: Encoded video stream, sound stream, smell data, wind data

[1601] Output: Display, audio playback, scent emission, wind generation

[1602] Step 8:

[1603] The emotion engine uses a camera and facial expression recognition algorithms (e.g., OpenCV and Dlib) to obtain emotion data from the user's facial expressions, and analyzes the obtained emotion data to determine whether the user is relaxed or not.

[1604] Input: Camera video data

[1605] Output: Emotion data

[1606] Step 9:

[1607] The emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind parameters based on the analysis results of the emotion engine. For example, if the user is not relaxed, the scenery will be changed to a "calm lake" and the music will be changed to a "quiet piano melody."

[1608] Input: Emotion data, scenery, sound, smell, wind parameters

[1609] Output: Adjusted scenery, sound, smell, and wind parameters

[1610] (Application example 2)

[1611] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1612] When seeking relaxation and peace of mind, modern users seek customized, holistic experiences based not only on visual elements but also on hearing, smell, touch, and even emotional state. However, existing systems face challenges in integrating these multiple sensory elements and dynamically adjusting them based on individual users' emotions and feedback. In particular, there is a need for systems that can analyze a user's emotional state in real time and make appropriate adjustments in virtual experiences aimed at stress reduction and relaxation.

[1613] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1614] In this invention, the server includes a generation means for generating a scenery, a display means for displaying the generated scenery, a sound reproduction means for reproducing a sound corresponding to the scenery, an odor generation means for generating an odor corresponding to the scenery, a wind generation means for generating a wind corresponding to the scenery, an acquisition means for acquiring user information, a control means for generating a scenery based on the acquired user information, a distribution means for streaming the generated scenery, an emotion analysis means for analyzing the user's emotions, and an emotion-based adjustment means for dynamically adjusting the scenery based on the emotion analysis. This makes it possible to provide a comprehensive and dynamically customized scenery experience for each individual user, thereby realizing user relaxation and stress reduction.

[1615] "Generation means" refers to the devices and techniques used to generate a landscape.

[1616] "Display means" refers to a display or screen for displaying the generated scenery.

[1617] The "sound reproduction means" refers to a speaker or an audio reproduction device for reproducing sounds that correspond to the scenery.

[1618] The "scent generating means" is a device such as an aroma diffuser that generates a scent that matches the scenery.

[1619] The "wind generating means" refers to a fan or air blower for generating wind that matches the scenery.

[1620] The "acquisition means" refers to a sensor or input device for acquiring user information.

[1621] The "control means" is a computer or software for controlling the generation of scenery based on the acquired user information.

[1622] "Delivery means" refers to the network technology and servers used to stream the generated scenery in real time.

[1623] "Emotion analysis means" refers to facial recognition technology and sensors for analyzing the user's emotions.

[1624] An "emotion-based adjustment" is a system or algorithm for dynamically adjusting a scene based on emotion analysis.

[1625] The "feedback acquisition means" is a device or interface for collecting feedback from users.

[1626] "Feedback analysis means" refers to technology or software that analyzes the obtained feedback and reflects it in the next landscape generation.

[1627] A "generative AI model" is an artificial intelligence model that generates landscapes in real time based on user information and input data.

[1628] A "prompt sentence" is an input sentence or command used to give instructions to a generative AI model.

[1629] To implement the present invention, the system is composed of the following elements: a server, a terminal, and a user play key roles.

[1630] 1. Collection of User Information

[1631] Users use their smartphones to input information such as their preferred scenery, scent preferences, and desired viewing times, etc. Based on this information, basic data is generated to provide an individually customized experience.

[1632] 2. Data processing on the server

[1633] The terminal transmits the entered user information over the Internet to a server, which receives the information, formats it appropriately, and stores it.

[1634] The server is equipped with a generative AI model that generates scenery in real time based on user information.

[1635] Example: If a user selects "seascapes at 3pm" as their preference, the generative AI model generates "seascapes at 3pm in the summer."

[1636] During this process, the AI ​​model is instructed using prompts, such as "User preferences: beach scenery at 3 PM, dislikes floral scents. Generate scenery and analyze emotions in real-time."

[1637] 3. Streaming content

[1638] The server encodes the generated scenery and corresponding sound, smell, and wind data in real time and streams them to the device.

[1639] A distribution means is used to ensure uninterrupted distribution of data.

[1640] 4. Playing data on the device

[1641] The device decodes the received data in real time, displays the scenery on the display, plays sounds using the sound playback means, emits appropriate smells using the smell generation means, and reproduces wind using the wind generation means.

[1642] Example: At 3:00 PM, as set by the user, an ocean scene is displayed on the smartphone screen, the sound of waves is played from the speaker, the aroma diffuser emits the scent of the sea, and a fan simulates a light breeze.

[1643] 5. Sentiment Analysis and Dynamic Adjustment

[1644] The device captures the user's facial expressions and movements using a camera or sensor and sends them to a server.

[1645] The emotion analysis means of the server analyzes the emotion of the user, and the emotion-based adjustment means dynamically adjusts the scenery, sound, smell, and wind to suit the emotion of the user.

[1646] Example: If the user is not relaxed, the scenery or music changes to a calming one.

[1647] As a result, this system is able to provide a comprehensive relaxation experience that is individually customized for each user and adapted to their emotional state.

[1648] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1649] Step 1:

[1650] Entering user information

[1651] Users launch the smartphone app and enter information such as their favorite scenery, least favorite smells, and desired display time.

[1652] Input: User preferences and settings (e.g., "I want to see ocean views at 3pm").

[1653] Output: User information is retrieved and passed to the next processing step.

[1654] Step 2:

[1655] Sending user information

[1656] The terminal transmits the acquired user information to a server via the Internet.

[1657] Input: User information obtained in step 1.

[1658] Output: User information is sent to the server and stored on the server.

[1659] Step 3:

[1660] Prompt generation for landscape generation

[1661] The server creates a prompt sentence that gives instructions to the generative AI model based on the received user information.

[1662] Input: User information (e.g. "I want to see a seascape at 3pm").

[1663] Output: A prompt statement is generated (e.g., "User preferences: beach scenery at 3 PM, dislikes floral scents.").

[1664] Step 4:

[1665] Scenery generation

[1666] The server's generative AI model generates scenery in real time based on the generated prompt sentence.

[1667] Input: The prompt statement.

[1668] Output: Generated landscape data and associated sound, smell, and wind data.

[1669] Step 5:

[1670] Encoding and Streaming Data

[1671] The server encodes the generated landscape data in real time and streams it to the terminal.

[1672] Input: Generated landscape data, sound data, smell data, and wind data.

[1673] Output: The encoded data is delivered to the device.

[1674] Step 6:

[1675] Data decoding and playback

[1676] The device decodes the received data, displays the scenery on the screen, plays sound from the speaker, and controls an aroma diffuser or fan to recreate scents and breezes.

[1677] Input: Encoded scenery data, sound data, smell data, and wind data.

[1678] Output: Scenery displayed on the screen, audio played from the speaker, scent emitted from the aroma diffuser, and wind generated by the fan.

[1679] Step 7:

[1680] User Emotion Recognition

[1681] The device uses a camera and sensors to capture the user's facial expressions and movements, and sends the data to a server.

[1682] Input: User facial expression and movement data.

[1683] Output: The acquired emotion data is sent to the server.

[1684] Step 8:

[1685] Sentiment analysis and landscape adjustment

[1686] The server analyzes the user's emotional state using an emotion analysis means, and dynamically adjusts the scenery, sounds, smells, and wind using an emotion-based adjustment means.

[1687] Input: The submitted emotion data.

[1688] Output: New scenery, sound, smell, and wind data adjusted based on the user's emotions.

[1689] Step 9:

[1690] Get feedback and incorporate it into the next generation

[1691] The device receives feedback from the user and sends it to the server, which analyzes the feedback and reflects it in the next landscape generation.

[1692] Input: User feedback information (e.g. satisfaction level, text comments).

[1693] Output: The analyzed feedback data is reflected in the next prompt sentence of the generative AI model.

[1694] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1695] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1696] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1697] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1698] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1699] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1700] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1701] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1702] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1703] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1704] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1705] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1706] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1707] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1708] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1709] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1710] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1711] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1712] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1713] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1714] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1715] The following is further disclosed regarding the above embodiment.

[1716] (Claim 1)

[1717] A generating means for generating a landscape;

[1718] a display means for displaying the generated scenery;

[1719] a sound reproducing means for reproducing a sound according to the scenery;

[1720] an odor generating means for generating an odor corresponding to a landscape;

[1721] a wind generating means for generating wind according to the scenery;

[1722] An acquisition means for acquiring user information;

[1723] control means for generating a scene based on the acquired user information;

[1724] A system including:

[1725] (Claim 2)

[1726] feedback acquisition means for acquiring feedback from a user;

[1727] a feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation;

[1728] The system of claim 1 further comprising:

[1729] (Claim 3)

[1730] 2. The system according to claim 1, further comprising time and season change generating means for generating changes based on seasons, weather, and time periods in generating scenery.

[1731] "Example 1"

[1732] (Claim 1)

[1733] A generating means for generating a landscape;

[1734] a display means for displaying the generated scenery;

[1735] a sound reproducing means for reproducing a sound according to the scenery;

[1736] an odor generating means for generating an odor corresponding to a landscape;

[1737] a wind generating means for generating wind according to the scenery;

[1738] An acquisition means for acquiring user information;

[1739] control means for generating a scene based on the acquired user information;

[1740] a transmitting means for streaming the generated sights and sounds;

[1741] an odor and wind control means for generating odor and wind control information corresponding to the scenery and transmitting the information to a terminal;

[1742] A system including:

[1743] (Claim 2)

[1744] feedback acquisition means for acquiring feedback from a user;

[1745] a feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation;

[1746] a generative AI means for generating a landscape using a generative AI model;

[1747] The system of claim 1 further comprising:

[1748] (Claim 3)

[1749] 2. The system according to claim 1, further comprising time and season change generating means for generating changes based on seasons, weather, and time periods in generating scenery.

[1750] "Application Example 1"

[1751] (Claim 1)

[1752] A generating means for generating a landscape;

[1753] a display means for displaying the generated scenery;

[1754] a sound reproducing means for reproducing a sound according to the scenery;

[1755] an odor generating means for generating an odor corresponding to a landscape;

[1756] a wind generating means for generating wind according to the scenery;

[1757] An acquisition means for acquiring user information;

[1758] control means for generating a scene based on the acquired user information;

[1759] a visual display means, which is a display device for providing a real-time scenery experience to a user;

[1760] an information display means for displaying related information during the experience;

[1761] sensor technology for detecting user gaze tracking or interest;

[1762] A system including:

[1763] (Claim 2)

[1764] feedback acquisition means for acquiring feedback from a user;

[1765] a feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation;

[1766] The system of claim 1 further comprising:

[1767] (Claim 3)

[1768] 2. The system according to claim 1, further comprising time and season change generating means for generating changes based on seasons, weather, and time periods in generating scenery.

[1769] "Example 2: Combining Emotion Engines"

[1770] (Claim 1)

[1771] A generating means for generating a landscape;

[1772] a display means for displaying the generated scenery;

[1773] a sound reproducing means for reproducing a sound according to the scenery;

[1774] an odor generating means for generating an odor corresponding to a landscape;

[1775] a wind generating means for generating wind according to the scenery;

[1776] An acquisition means for acquiring user information;

[1777] control means for generating a scene based on the acquired user information;

[1778] an emotion engine for recognizing a user's emotion;

[1779] emotion-based adjustment means for dynamically adjusting the content of the scenery, sounds, smells, and wind based on the recognized emotion of the user;

[1780] A technology that uses a generative AI model to generate scenery based on user information,

[1781] A system including:

[1782] (Claim 2)

[1783] feedback acquisition means for acquiring feedback from a user;

[1784] a feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation;

[1785] The system of claim 1 further comprising:

[1786] (Claim 3)

[1787] 2. The system according to claim 1, further comprising time and season change generating means for generating changes based on seasons, weather, and time periods in generating scenery.

[1788] "Application example 2 when combining emotion engines"

[1789] (Claim 1)

[1790] A generating means for generating a landscape;

[1791] a display means for displaying the generated scenery;

[1792] a sound reproducing means for reproducing a sound according to the scenery;

[1793] an odor generating means for generating an odor corresponding to a landscape;

[1794] a wind generating means for generating wind according to the scenery;

[1795] An acquisition means for acquiring user information;

[1796] control means for generating a scene based on the acquired user information;

[1797] a distribution means for streaming the generated scenery;

[1798] emotion analysis means for analyzing the emotion of a user;

[1799] emotion-based adjustment means for dynamically adjusting the scenery based on emotion analysis;

[1800] A system including:

[1801] (Claim 2)

[1802] feedback acquisition means for acquiring feedback from a user;

[1803] a feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation;

[1804] A means for dynamically adjusting the prompt sentences of the generative AI model based on the feedback results;

[1805] The system of claim 1 further comprising:

[1806] (Claim 3)

[1807] 2. The system according to claim 1, further comprising time and season change generating means for generating changes based on seasons, weather, and time periods in generating scenery. [Explanation of symbols]

[1808] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A generating means for generating a landscape; a display means for displaying the generated scenery; a sound reproducing means for reproducing a sound according to the scenery; an odor generating means for generating an odor corresponding to a landscape; a wind generating means for generating wind according to the scenery; An acquisition means for acquiring user information; control means for generating a scene based on the acquired user information; A system including:

2. feedback acquisition means for acquiring feedback from a user; A feedback analysis means for analyzing the feedback and reflecting it in the next landscape generation; The system of claim 1 further comprising:

3. 2. The system according to claim 1, further comprising time and season change generating means for generating changes based on seasons, weather, and time periods in generating scenery.

Images