system

Abstract

We provide the system. [Solution] A means of generating information based on user preferences, A means for acquiring real-time environmental and time data, means for updating the aforementioned information based on the environmental data and the time data, Means for transmitting the updated information to a display medium, A means of providing a virtual experience based on the user's choice, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, due to technological progress, the time spent indoors has increased. As a result, contact with the outside world has decreased, and the sense of mental blockage and stress among people has increased. Also, in situations where traveling is not possible, the demand for fresh scenery and environmental changes is increasing. However, technologies that provide a video experience linked to real - time and weather and corresponding to individual user preferences are still insufficient to meet these needs.

Means for Solving the Problems

[0005] The present invention provides a system for generating images based on user preferences. Specifically, it includes means for generating images customized for each user based on data acquired from the user, means for acquiring real-time weather and time data and updating the images based on this data, and means for transmitting the updated images to a display device. By using AI technology, this system enables dynamic adjustment of image elements and achieves a high degree of personalization. As a result, users can experience different places and atmospheres while maintaining a sense of reality, and reduce feelings of mental confinement.

[0006] "User preferences" refer to the choices of themes, scenery, music, time of day, etc., that individual users desire under specific conditions or environments.

[0007] "Means for generating images" refers to a technical system or process for forming visual materials and scenes based on user settings and external data.

[0008] "Real-time weather data" refers to the latest information on current weather conditions obtained from external weather information services.

[0009] "Time data" refers to information about the current time and date, and is used to change the scenes and atmosphere of a video according to that time.

[0010] A "display device" refers to hardware equipment that can physically display generated images, such as televisions and monitors.

[0011] "AI technology" refers to artificial intelligence technology, which is a technology that has the ability to automatically and dynamically adjust the elements of a video using machine learning and data analysis.

[0012] "Personalization" refers to adjusting specific experiences or services to suit the individual user's preferences and needs. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] A specific embodiment for carrying out the present invention is shown below. This system personalizes the video experience based on the user's preferences and dynamically updates the video using real-time weather and time data.

[0035] First, users use their devices to set their preferred themes and locations. These settings include visual themes such as "tropical beach" or "European cityscape." This settings data is formatted by the device and sent to the server.

[0036] The server stores the received user settings information in a database, obtains weather data from external weather information services, and also collects time data. Based on this collected data, AI technology within the server generates video optimized for the user. This AI is driven by a specific program and automatically adjusts the color tone, atmosphere, and dynamic elements of the video to match the user's preferences.

[0037] The generated video data is transmitted to the user's terminal via the network and reflected on the display device. On this display device, the video is updated in real time according to weather and time, providing the user with a realistic and immersive visual experience.

[0038] As a concrete example, consider a scenario where a user selects the themes "Hawaii beach during the day" and "San Francisco night view." In this case, the server generates images of a sunny beach or a city illuminated at night based on the weather in the respective region, providing the optimal display according to the user's time of day. Through these images, the user can experience the feeling of being in those locations, even though they cannot actually visit them.

[0039] Thus, the present invention makes it possible to obtain a mental refreshing effect by providing users with realistic and personalized images.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] Users interact with applications on their devices to select their preferred themes, locations, times of day, and music genres. This is done through the user interface and the preferences are entered into the device as data.

[0043] Step 2:

[0044] The device organizes the user-specified configuration information and sends it to the server as formatted data. This allows for personalized information based on the user's requests.

[0045] Step 3:

[0046] The server stores user configuration information received from the terminal in a database. This information will be used as a basis for future reference and video generation.

[0047] Step 4:

[0048] The server accesses external weather APIs and time data APIs to obtain the current weather and time for the location selected by the user. This allows for the acquisition of real-time environmental data.

[0049] Step 5:

[0050] The server uses AI to combine user preference information with acquired real-time data to generate optimal video content. The AI ​​generates visual elements, color schemes, and dynamic scenes according to the selected theme.

[0051] Step 6:

[0052] The server sends the generated video data to the terminal. After receiving the data, the terminal prepares to display it on the display device.

[0053] Step 7:

[0054] The terminal outputs the received video data to a display device, providing the user with a visual experience. It also quickly reflects new data when updates are periodically received from the server.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] There is a need for a system that can generate visual data tailored to individual user preferences in real time and dynamically update it in response to environmental changes. Conventional systems have the problem of not being able to effectively combine user settings information with real-time environmental data, which limits the quality of the visual experience.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes means for generating visual data based on user choices, means for acquiring real-time environmental data and time information from external sources, and means for dynamically updating the visual data based on the environmental data and time information. This makes it possible to provide a visual experience personalized to the user's individual preferences based on real-time environmental changes.

[0060] "User" refers to an individual or group that operates or configures an information processing device.

[0061] "Options" refer to settings that allow users to specify their preferences and conditions when generating visual data.

[0062] "Visual data" refers to video and image data generated to provide users with a visual experience.

[0063] "Environmental data" refers to data such as weather and geographical information obtained from external sources for use in generating and updating visual data.

[0064] "Time information" refers to information about the current time used to influence the generation and updating of visual data.

[0065] "Dynamic updating" refers to the process of adapting the content of visual data based on environmental data and time information that change in real time.

[0066] A "display device" refers to a device such as a screen or projector used to show generated visual data to the user.

[0067] An "information processing device" refers to a computer or server that performs a series of functions, such as generating visual data based on user selection and updating it according to the environment.

[0068] This system aims to provide users with personalized visual experiences, specifically by generating visual data based on themes and locations selected by the user. The server is primarily responsible for generating and dynamically updating visual data using machine learning techniques. Specifically, the server uses software platforms such as TENSORFLOW® and OpenCV to create visual data by utilizing environmental data and time information obtained from external sources.

[0069] Users input their preferred themes through their device. This device can be a smartphone, tablet, or personal computer. Themes selected by users include options such as "tropical beach" or "European cityscape," and this information is sent from the device to the server. Based on this information, the server uses a generative AI model to construct optimal visual data.

[0070] Furthermore, the server acquires real-time environmental data from external weather information services and also obtains current time information from the system. By integrating this information, the server can provide personalized visual data tailored to each user's environment. The visual data generated by the server is transmitted to the terminal via the network and displayed on the terminal's display device.

[0071] As a concrete example, consider a scenario where a user selects "a beach in Hawaii during the daytime" using their device. In this case, the server generates video footage of a sunny beach based on acquired Hawaiian weather data and inserts it at daytime. This visual data provides the user with the illusion of actually being in Hawaii.

[0072] As an example of a prompt, by sending a specific request to the generation AI model, such as "Generate a video themed around a tropical beach, and make it realistic by matching it with the current weather," a video tailored to the user's needs will be generated.

[0073] In this way, the system can provide users with a rich and personalized visual experience, resulting in a better user experience.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] Input: Themes and location information selected by the user via their device.

[0077] Specific operation: The user uses a smartphone or PC application to select a theme such as "tropical beach" or "European cityscape."

[0078] Data processing and output: The terminal formats this information in XML or JSON format and sends it to the server. The transmitted data is stored on the server side as user-specific configuration information.

[0079] Step 2:

[0080] Input: User theme information sent to the server.

[0081] Specific operation: Based on the received information, the server calls an external weather information service API to obtain real-time weather data. It also obtains the current time information from the system's internal clock.

[0082] Data processing and output: Weather data and time information are stored on the server as JSON objects.

[0083] Step 3:

[0084] Input: User theme information and weather and time information obtained from external sources.

[0085] Specific operation: Based on this information, the server activates the generative AI model and starts the process of generating visual data. During this process, TensorFlow is used to input "prompt statements" to the AI ​​model.

[0086] Data Processing and Output: The AI ​​model analyzes the input data and generates video data with adjusted color tones and effects based on the user's preferences. The generated video data is stored in memory.

[0087] Step 4:

[0088] Input: Video data generated by an AI model.

[0089] Specific operation: The server transmits the generated video data to the user's terminal via the network.

[0090] Data Processing and Output: Video data is transmitted using the HTTP protocol, and the terminal renders the received data on the display device. This allows users to view the visual data in real time.

[0091] Step 5:

[0092] Input: Displayed video data, real-time weather, and time-of-day information.

[0093] Specific operation: The user's device periodically connects to the server to obtain new weather data and time information.

[0094] Data Processing and Output: The acquired data is used to update the visual data, dynamically switching to images that match the current situation on the device. This ensures that the visual experience is always updated based on the latest real-time data.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] Currently, there are limited means of easily enjoying personalized visual information by providing virtual experiences in real time based on the environment and time that users wish to experience at that moment. The present invention aims to provide a new method for generating immersive virtual experiences that match the user's preferences by dynamically adapting to the real environment and time.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for generating information based on user preferences, means for acquiring real-time environmental and time data, and means for updating the information based on the environmental and time data. This allows the virtual environment to adapt to real-world conditions in real time according to the user's selection, enabling a more personalized experience.

[0100] A "user" refers to a person who uses the system, receives information, and enjoys virtual experiences based on their individual wishes and preferences.

[0101] "Preferences" refer to the specific tendencies and tastes that individual users possess, and are elements that should be considered when selecting and presenting visual information.

[0102] "Information" refers to the data and content that constitute virtual environments and experiences generated based on user preferences.

[0103] "Generative means" refers to the processes and technologies used to construct data according to user preferences and to embody it as visual or experiential information.

[0104] "Environmental data" refers to information about a specific location or situation collected in real time, including weather conditions and surrounding environment.

[0105] "Time data" refers to information about the time of day that is considered when the information is generated, and includes elements based on time of day, such as daytime or nighttime.

[0106] "Means of updating" refers to methods or processes for appropriately modifying or enhancing the content of previously generated information based on acquired real-time data.

[0107] "Display medium" refers to a device or screen that allows users to visually experience a generated virtual environment or information.

[0108] A "virtual experience" refers to a synthetic environment or event that allows users to obtain similar sensations and impressions without directly experiencing them in the real world.

[0109] The following system configuration is necessary to implement the present invention.

[0110] The server runs software to generate information based on user preferences and acquire real-time environmental and time data. Specifically, the server uses machine learning techniques to analyze environmental data and generate visual information tailored to the user's preferences. The hardware used would be a standard network-connected computing device, and the software could utilize Python or data processing libraries (e.g., Pandas, NumPy).

[0111] The terminal is an information display device owned by the user, which displays updated information received from the server. Portable devices such as smartphones and tablets are used for this purpose. The terminal interprets the information provided by the server and runs applications to provide the user with a visual experience. For example, libraries (e.g., PIL, OpenCV) can be used to visually represent real-time environmental changes.

[0112] Users access the system and select their desired theme and time slot. Through the interface, users input their preferences and specify a particular virtual experience. The entered data is sent to the server, and information generated based on this is fed back to the terminal, allowing users to experience environments that are difficult to encounter in reality.

[0113] For example, if a user selects "tropical beach" and requests a sunny midday, the server generates visual information of a sunny afternoon beach in Hawaii and sends it to the device. This allows the user to enjoy the feeling of being there, both visually and psychologically, even while at home.

[0114] A concrete example of a prompt message for a generative AI model would be: "Generate a dynamic image based on the theme the user has chosen, taking into account the local weather and time of day. For example, if the user selects 'tropical beach' and the local weather is sunny, create an image of a white sand beach reflecting sunlight."

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The user selects their preferred theme and desired experience using the device's interface. The user's preferences and selections are provided as input, and this is output as data sent from the device to the server.

[0118] Step 2:

[0119] The server receives selection data submitted by the user and retrieves real-time environmental and time data from external environmental data sources. In this step, the input consists of user preference data and retrieved environmental data, and these data are used to create the conditions necessary for information generation.

[0120] Step 3:

[0121] The server uses an AI model based on acquired environmental and time data to generate virtual information tailored to the user's preferences. Specifically, preference data and environmental data are input to the AI ​​model via prompts, and visual virtual information is obtained as output.

[0122] Step 4:

[0123] The generated information is sent from the server to the terminal. Here, virtual information is the input, and the data transfer to the terminal is the output. This process takes place over a network.

[0124] Step 5:

[0125] The terminal displays virtual information received from the server. It displays the virtual information as input on the display device and performs specific actions to provide the user with a preferred experience.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] A specific embodiment for carrying out the present invention is shown below. This system provides personalized video that responds to the user's emotions by combining an emotion engine that recognizes the user's emotions.

[0128] The user selects their preferred theme through the device and collects emotional data using a camera and microphone for emotion recognition. The device immediately passes this emotional data to an emotion engine, which analyzes the user's current emotional state. The emotion engine classifies emotions such as joy, sadness, and surprise based on the user's voice tone and facial expressions.

[0129] The analyzed emotional data is sent from the device to the server, which uses this data to generate video. AI technology is particularly used to dynamically adjust the video's color tone, music, and environmental elements based on the emotional data and user preferences. For example, when a user wants to relax, the video is generated with calming colors and soothing music.

[0130] Once this generation process is complete, the server sends the generated video data to the terminal. The terminal projects the latest video data onto a display device, providing the user with the most suitable video experience. This system also takes real-time weather and time data into consideration, so for example, if a user is feeling gloomy on a rainy day, it can provide a warm indoor scene.

[0131] As a concrete example, suppose a user is in a cheerful mood and has selected the theme "Caribbean beach." In this case, the emotion engine detects the user's smile, and the server generates a video showing bright sunshine and a clear ocean, mixed with upbeat music. Through this process, the user is provided with a video that matches their emotions, leading to a deeper sense of satisfaction.

[0132] The introduction of this technology will enable users to not only obtain visual information but also enjoy new experiences that are tailored to their emotions.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] The user launches the application on their device and enables their preferred video theme and the on-device camera and microphone. This prepares the system for inputting the basic data for video generation.

[0136] Step 2:

[0137] The device collects user emotion data by capturing the user's facial expressions with a camera and recording their voice tone with a microphone. This data is then passed to the emotion engine in real time.

[0138] Step 3:

[0139] The emotion engine analyzes the received facial expression and voice data to identify the user's emotional state. For example, it recognizes emotions such as "joy" when there are many smiles and "calmness" when there is prolonged silence.

[0140] Step 4:

[0141] The device sends the analyzed emotional data to the server. Along with this, information about the user's preferred themes and locations is also sent to the server.

[0142] Step 5:

[0143] The server retrieves real-time weather and time data from external APIs. Based on this data, it integrates all the information necessary for generating video.

[0144] Step 6:

[0145] The server's AI technology uses user preferences, emotional states, weather data, and time data to generate optimal video. This process adjusts elements such as the video's color tone, music, and motion.

[0146] Step 7:

[0147] The server sends the generated video data to the terminal. The terminal then displays the video optimized for the monitor based on this data.

[0148] Step 8:

[0149] The device continuously collects emotional data while displaying video to the user, and updates the video by sending new data to the server as needed. This ensures that the video is always appropriate to the user's state.

[0150] (Example 2)

[0151] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0152] Conventional video generation systems generate videos without considering the user's emotional state, making it difficult to provide a video experience tailored to the user's individual emotions. Furthermore, they were insufficient in generating personalized videos that responded to real-time environmental changes. This resulted in decreased user satisfaction.

[0153] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0154] In this invention, the server includes means for analyzing the user's emotions, means for generating video based on the user's preferences, and means for acquiring real-time environmental data. This makes it possible to provide personalized video based on the user's emotions and environmental conditions.

[0155] A "user" is an individual who uses the system to receive personalized, emotion-based videos.

[0156] "Methods for analyzing emotions" refer to a technical process that determines emotions based on a user's facial expressions and voice, and provides this information as data.

[0157] "Means of generating video" refers to technologies used to create unique videos based on the user's emotions and preferences.

[0158] "Real-time environmental data" refers to information that can instantly obtain current weather, time, and other environmental conditions.

[0159] "Updated footage" refers to the latest footage generated to reflect user sentiment data and real-time environmental data.

[0160] A "display device" is a device used to visually present generated video to the user.

[0161] "Generative artificial intelligence" is an AI technology used to dynamically adjust each element of a video based on user input data.

[0162] This invention relates to a system that generates personalized videos based on the user's emotions. The system consists of a terminal, a server, an emotion analysis engine, and a generative AI model.

[0163] First, the user selects the theme of the video they want to watch through their device. The device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice in real time. This emotional data is immediately sent to an emotion analysis engine, which analyzes the user's emotional state. During the analysis process, a generative AI model is used to identify emotions such as joy, sadness, and surprise from the user's smile and voice tone.

[0164] The analyzed emotional data is sent to a server in the cloud. The server uses generative AI technology to generate video based on the user's emotional data and selected theme. Real-time weather and time data are also taken into consideration when generating the video. For example, a user who wants to relax on a rainy day will be provided with a calm indoor scene.

[0165] The generated video data is sent from the server to the terminal. The terminal displays the received video on its screen, providing the user with the most suitable video experience. This allows the user to enjoy videos that match their emotions.

[0166] As a concrete example, consider a scenario where a user is in a cheerful mood and selects the "Caribbean Beach" theme. In this case, the emotion analysis engine recognizes the user's smile, and the server generates an image including bright sunshine, a clear ocean, and upbeat music. This process occurs in real time, allowing the user to enjoy the results immediately.

[0167] An example of a prompt when using a generative AI model is the instruction, "When the user smiles, generate a video of a sunny beach and cheerful music." Based on this, the AI ​​dynamically adjusts each element of the video to provide the optimal experience.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The user operates the device and selects the theme of the video they want to watch. The theme data is obtained as input information. The device records this theme information and prepares to activate the camera and microphone. This prepares the device to capture the user's emotions.

[0171] Step 2:

[0172] The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time. Emotional data is acquired during this process. The collected data is immediately sent to an emotion analysis engine. Here, a generative AI model is used to analyze the emotions from the user's facial expressions and voice based on the emotional data. The output is classification information of the analyzed emotions.

[0173] Step 3:

[0174] The terminal sends analyzed emotion data to the server. This input includes emotion classification information and theme data. The server uses a generative AI to dynamically generate video based on the received emotion data and theme. Specifically, it uses prompts to select the color tone and music for the video, generating optimized video data. The output is the generated customized video data.

[0175] Step 4:

[0176] The server sends the generated video data to the terminal. This input includes the completed video data. The terminal prepares to play the received video on its display and prompts the user to start viewing. The output is a state where the user is viewing a personalized video on the display.

[0177] Step 5:

[0178] Users view videos through a display and enjoy an emotionally responsive video experience. The system continuously optimizes the user experience by collecting new emotional data in real time as needed and updating the video accordingly. As an output, user satisfaction increases.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] In recent years, personalizing content based on user emotions has become increasingly important, but conventional systems have struggled to accurately reflect users' emotional states in video content. Therefore, the challenge lies in appropriately adjusting content based on users' current emotions and providing experiences tailored to their individual needs.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes means for acquiring user emotion data, means for using artificial intelligence to generate video based on the emotion data, and means for acquiring real-time environmental data. This makes it possible to generate personalized video that reflects the user's emotions in real time.

[0184] "User emotional data" refers to data obtained from a user's facial expressions and voice tone, and represents information that indicates the user's current emotional state.

[0185] "Artificial intelligence" is a technology in which computer systems imitate human intellectual activity, and in particular, it has the ability to dynamically adjust content based on emotional data in areas such as video generation and data analysis.

[0186] "Video generation" is the process of creating new video content based on user emotional data and other relevant data.

[0187] "Real-time environmental data" refers to information that includes current weather information and time-related data, and is used to make adjustments to suit the user's situation.

[0188] "Personalized video" refers to video content customized according to the individual preferences and emotional state of a user, providing a visual experience optimized for each user.

[0189] The system for implementing this invention provides a personalized video experience by acquiring user emotion data and generating emotion-based videos in real time. The system mainly consists of three components: a terminal, a server, and the user.

[0190] The device directly interfaces with the user and captures the user's facial expressions and voice tone through its camera and microphone. This data is used to analyze facial expressions using OpenCV and to extract emotional data from the voice using Google AI's Speech-to-Text API.

[0191] The server uses a generative AI model to generate user-friendly videos based on emotional data sent from the terminal. It performs emotional analysis using frameworks such as TensorFlow and PyTorch, and dynamically adjusts the video's color tone, music, and environmental elements based on the results. This process generates personalized videos that match the user's emotions.

[0192] Specifically, when a user is experiencing stress, the system provides videos featuring relaxing natural scenery and soothing music. Conversely, when a user is feeling happy, the system generates videos with bright colors and cheerful music. The generated video data is then transmitted from the server to the user's device and provided to them.

[0193] This system uses prompt statements like the following as input to the generated AI model.

[0194] "The user's current emotional state is 'sadness,' so please provide relaxing visuals."

[0195] "To put the user in a 'joyful' state, please generate a video clip of a city with bright weather accompanied by energetic music."

[0196] The introduction of this system will enable users to enjoy a video experience tailored to their individual needs in real time, leading to a deeper level of satisfaction.

[0197] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0198] Step 1:

[0199] The device uses a camera and microphone to capture the user's facial expressions and voice tone. The input is raw video and audio data, which is captured in real time. The output is sentiment data in an analyzable format.

[0200] Step 2:

[0201] The device uses OpenCV to analyze facial expressions, converts audio data to text using Google AI's Speech-to-Text API, and then estimates the emotion. The input is the video and audio data obtained in step 1, and the output is data representing the user's current emotional state (e.g., joy, sadness, surprise).

[0202] Step 3:

[0203] The device sends analyzed emotion data to the server. The input is data containing the user's emotional state, and the output is a status indicating that the data transmission to the server was successful.

[0204] Step 4:

[0205] The server uses the received emotion data to activate a generative AI model and generate video content tailored to the user. The input is the user's emotion data, which is used to instruct the generative AI model using prompts. The output is dynamically adjusted video content.

[0206] Step 5:

[0207] The server generates video content and sends it to the terminal. The input is video data generated by AI, and the output is the state of transmission to the user's terminal.

[0208] Step 6:

[0209] The device plays back the video data it receives and provides it to the user. The input is the video data received from the server, and the output is the video the user views. During playback, the device optimizes the user experience by setting the optimal display settings based on environmental factors.

[0210] This processing flow allows users to enjoy personalized videos that match their emotions in real time.

[0211] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0212] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0213] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0214] [Second Embodiment]

[0215] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0216] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0217] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0218] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0219] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0221] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0222] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0223] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0224] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0225] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0226] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0227] A specific embodiment for carrying out the present invention is shown below. This system personalizes the video experience based on the user's preferences and dynamically updates the video using real-time weather and time data.

[0228] First, users use their devices to set their preferred themes and locations. These settings include visual themes such as "tropical beach" or "European cityscape." This settings data is formatted by the device and sent to the server.

[0229] The server stores the received user settings information in a database, obtains weather data from external weather information services, and also collects time data. Based on this collected data, AI technology within the server generates video optimized for the user. This AI is driven by a specific program and automatically adjusts the color tone, atmosphere, and dynamic elements of the video to match the user's preferences.

[0230] The generated video data is transmitted to the user's terminal via the network and reflected on the display device. On this display device, the video is updated in real time according to weather and time, providing the user with a realistic and immersive visual experience.

[0231] As a concrete example, consider a scenario where a user selects the themes "Hawaii beach during the day" and "San Francisco night view." In this case, the server generates images of a sunny beach or a city illuminated at night based on the weather in the respective region, providing the optimal display according to the user's time of day. Through these images, the user can experience the feeling of being in those locations, even though they cannot actually visit them.

[0232] Thus, the present invention makes it possible to obtain a mental refreshing effect by providing users with realistic and personalized images.

[0233] The following describes the processing flow.

[0234] Step 1:

[0235] Users interact with applications on their devices to select their preferred themes, locations, times of day, and music genres. This is done through the user interface and the preferences are entered into the device as data.

[0236] Step 2:

[0237] The device organizes the user-specified configuration information and sends it to the server as formatted data. This allows for personalized information based on the user's requests.

[0238] Step 3:

[0239] The server stores user configuration information received from the terminal in a database. This information will be used as a basis for future reference and video generation.

[0240] Step 4:

[0241] The server accesses external weather APIs and time data APIs to obtain the current weather and time for the location selected by the user. This allows for the acquisition of real-time environmental data.

[0242] Step 5:

[0243] The server uses AI to combine user preference information with acquired real-time data to generate optimal video content. The AI ​​generates visual elements, color schemes, and dynamic scenes according to the selected theme.

[0244] Step 6:

[0245] The server sends the generated video data to the terminal. After receiving the data, the terminal prepares to display it on the display device.

[0246] Step 7:

[0247] The terminal outputs the received video data to a display device, providing the user with a visual experience. It also quickly reflects new data when updates are periodically received from the server.

[0248] (Example 1)

[0249] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0250] There is a need for a system that can generate visual data tailored to individual user preferences in real time and dynamically update it in response to environmental changes. Conventional systems have the problem of not being able to effectively combine user settings information with real-time environmental data, which limits the quality of the visual experience.

[0251] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0252] In this invention, the server includes means for generating visual data based on user choices, means for acquiring real-time environmental data and time information from external sources, and means for dynamically updating the visual data based on the environmental data and time information. This makes it possible to provide a visual experience personalized to the user's individual preferences based on real-time environmental changes.

[0253] "User" refers to an individual or group that operates or configures an information processing device.

[0254] "Options" refer to settings that allow users to specify their preferences and conditions when generating visual data.

[0255] "Visual data" refers to video and image data generated to provide users with a visual experience.

[0256] "Environmental data" refers to data such as weather and geographical information obtained from external sources for use in generating and updating visual data.

[0257] "Time information" refers to information about the current time used to influence the generation and updating of visual data.

[0258] "Dynamic updating" refers to the process of adapting the content of visual data based on environmental data and time information that change in real time.

[0259] A "display device" refers to a device such as a screen or projector used to show generated visual data to the user.

[0260] An "information processing device" refers to a computer or server that performs a series of functions, such as generating visual data based on user selection and updating it according to the environment.

[0261] This system aims to provide users with personalized visual experiences, specifically by generating visual data based on themes and locations selected by the user. The server is primarily responsible for generating and dynamically updating visual data using machine learning techniques. Specifically, the server uses software platforms such as TensorFlow and OpenCV to create visual data by utilizing environmental data and time information obtained from external sources.

[0262] Users input their preferred themes through their device. This device can be a smartphone, tablet, or personal computer. Themes selected by users include options such as "tropical beach" or "European cityscape," and this information is sent from the device to the server. Based on this information, the server uses a generative AI model to construct optimal visual data.

[0263] Furthermore, the server acquires real-time environmental data from external weather information services and also obtains current time information from the system. By integrating this information, the server can provide personalized visual data tailored to each user's environment. The visual data generated by the server is transmitted to the terminal via the network and displayed on the terminal's display device.

[0264] As a concrete example, consider a scenario where a user selects "a beach in Hawaii during the daytime" using their device. In this case, the server generates video footage of a sunny beach based on acquired Hawaiian weather data and inserts it at daytime. This visual data provides the user with the illusion of actually being in Hawaii.

[0265] As an example of a prompt, by sending a specific request to the generation AI model, such as "Generate a video themed around a tropical beach, and make it realistic by matching it with the current weather," a video tailored to the user's needs will be generated.

[0266] In this way, the system can provide users with a rich and personalized visual experience, resulting in a better user experience.

[0267] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0268] Step 1:

[0269] Input: Themes and location information selected by the user via their device.

[0270] Specific operation: The user uses a smartphone or PC application to select a theme such as "tropical beach" or "European cityscape."

[0271] Data processing and output: The terminal formats this information in XML or JSON format and sends it to the server. The transmitted data is stored on the server side as user-specific configuration information.

[0272] Step 2:

[0273] Input: User theme information sent to the server.

[0274] Specific operations: Based on the received information, the server calls an external weather information providing service API to obtain real-time weather data. It also obtains the current time information from the internal clock of the system.

[0275] Data processing and output: The weather data and time information are stored in the server as JSON objects respectively.

[0276] Step 3:

[0277] Input: The user's theme information, the weather data and time information obtained from outside.

[0278] Specific operations: Based on this information, the server activates the generated AI model to start the visual data generation process. At this time, "prompt text" is input into the AI model using TensorFlow.

[0279] Data processing and output: The AI model analyzes the input data and generates video data with color tones and effects adjusted based on the user's preferences. The generated video data is stored in memory.

[0280] Step 4:

[0281] Input: The video data generated by the AI model.

[0282] Specific operations: The server transmits the generated video data to the user's terminal via the network.

[0283] Data processing and output: The video data is transmitted using the HTTP protocol, and the terminal renders the received data on the display device. Thus, the user can view the visual data in real time.

[0284] Step 5:

[0285] Input: The displayed video data, and the real-time weather and time change information.

[0286] Specific operation: The user's terminal connects to the server regularly to obtain new weather data and time information.

[0287] Data processing and output: The acquired data is used to update visual data and is dynamically switched to a video that suits the current situation on the terminal. As a result, the visual experience is always updated based on the latest real-time data.

[0288] (Application Example 1)

[0289] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0290] Currently, there are limited means to easily enjoy personalized visual information by providing real-time virtual experiences based on the environment and time that the user wants to experience at that moment. The purpose of the present invention is to provide a new method for generating a virtual experience with a sense of presence that suits the user's preferences by dynamically adapting to the real environment and time.

[0291] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0292] In this invention, the server includes means for generating information based on the user's preferences, means for acquiring real-time environmental data and time data, and means for updating the information based on the environmental data and the time data. As a result, according to the user's selection, the virtual environment adapts to the real conditions in real time, enabling a more personalized experience.

[0293] The "user" refers to a person who uses the system to receive information and enjoys virtual experiences based on individual desires and preferences.

[0294] "Preferences" refer to the specific tendencies and tastes that individual users possess, and are elements that should be considered when selecting and presenting visual information.

[0295] "Information" refers to the data and content that constitute virtual environments and experiences generated based on user preferences.

[0296] "Generative means" refers to the processes and technologies used to construct data according to user preferences and to embody it as visual or experiential information.

[0297] "Environmental data" refers to information about a specific location or situation collected in real time, including weather conditions and surrounding environment.

[0298] "Time data" refers to information about the time of day that is considered when the information is generated, and includes elements based on time of day, such as daytime or nighttime.

[0299] "Means of updating" refers to methods or processes for appropriately modifying or enhancing the content of previously generated information based on acquired real-time data.

[0300] "Display medium" refers to a device or screen that allows users to visually experience a generated virtual environment or information.

[0301] A "virtual experience" refers to a synthetic environment or event that allows users to obtain similar sensations and impressions without directly experiencing them in the real world.

[0302] The following system configuration is necessary to implement the present invention.

[0303] The server runs software to generate information based on user preferences and acquire real-time environmental and time data. Specifically, the server uses machine learning techniques to analyze environmental data and generate visual information tailored to the user's preferences. The hardware used would be a standard network-connected computing device, and the software could utilize Python or data processing libraries (e.g., Pandas, NumPy).

[0304] The terminal is an information display device owned by the user, which displays updated information received from the server. Portable devices such as smartphones and tablets are used for this purpose. The terminal interprets the information provided by the server and runs applications to provide the user with a visual experience. For example, libraries (e.g., PIL, OpenCV) can be used to visually represent real-time environmental changes.

[0305] Users access the system and select their desired theme and time slot. Through the interface, users input their preferences and specify a particular virtual experience. The entered data is sent to the server, and information generated based on this is fed back to the terminal, allowing users to experience environments that are difficult to encounter in reality.

[0306] For example, if a user selects "tropical beach" and requests a sunny midday, the server generates visual information of a sunny afternoon beach in Hawaii and sends it to the device. This allows the user to enjoy the feeling of being there, both visually and psychologically, even while at home.

[0307] A concrete example of a prompt message for a generative AI model would be: "Generate a dynamic image based on the theme the user has chosen, taking into account the local weather and time of day. For example, if the user selects 'tropical beach' and the local weather is sunny, create an image of a white sand beach reflecting sunlight."

[0308] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0309] Step 1:

[0310] The user uses the interface of the terminal to select a preferred theme and desired experience content. The user's preferences and selections are provided as input and output as data transmitted from the terminal to the server.

[0311] Step 2:

[0312] The server receives the selection data transmitted from the user and acquires real-time environmental data and time data from an external environmental data source. In this step, there is the user's preference data and the acquired environmental data as input, and the conditions necessary for information generation are prepared based on these data.

[0313] Step 3:

[0314] The server uses the AI model generated based on the acquired environmental data and time data to generate virtual information that matches the user's preferences. Specifically, the preference data and environmental data as input are input into the AI model by a prompt sentence, and visual virtual information is obtained as output.

[0315] Step 4:

[0316] The generated information is transmitted from the server to the terminal. Here, there is virtual information as input, and the data transfer to the terminal is the output. This process is performed via a network.

[0317] Step 5:

[0318] The terminal displays the virtual information received from the server. The virtual information as input is displayed on a display device, and specific operations are performed to provide the user with the selected experience.

[0319] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0320] A specific embodiment for carrying out the present invention is shown below. This system provides personalized video that responds to the user's emotions by combining an emotion engine that recognizes the user's emotions.

[0321] The user selects their preferred theme through the device and collects emotional data using a camera and microphone for emotion recognition. The device immediately passes this emotional data to an emotion engine, which analyzes the user's current emotional state. The emotion engine classifies emotions such as joy, sadness, and surprise based on the user's voice tone and facial expressions.

[0322] The analyzed emotional data is sent from the device to the server, which uses this data to generate video. AI technology is particularly used to dynamically adjust the video's color tone, music, and environmental elements based on the emotional data and user preferences. For example, when a user wants to relax, the video is generated with calming colors and soothing music.

[0323] Once this generation process is complete, the server sends the generated video data to the terminal. The terminal projects the latest video data onto a display device, providing the user with the most suitable video experience. This system also takes real-time weather and time data into consideration, so for example, if a user is feeling gloomy on a rainy day, it can provide a warm indoor scene.

[0324] As a concrete example, suppose a user is in a cheerful mood and has selected the theme "Caribbean beach." In this case, the emotion engine detects the user's smile, and the server generates a video showing bright sunshine and a clear ocean, mixed with upbeat music. Through this process, the user is provided with a video that matches their emotions, leading to a deeper sense of satisfaction.

[0325] The introduction of this technology will enable users to not only obtain visual information but also enjoy new experiences that are tailored to their emotions.

[0326] The following describes the processing flow.

[0327] Step 1:

[0328] The user launches the application on their device and enables their preferred video theme and the on-device camera and microphone. This prepares the system for inputting the basic data for video generation.

[0329] Step 2:

[0330] The device collects user emotion data by capturing the user's facial expressions with a camera and recording their voice tone with a microphone. This data is then passed to the emotion engine in real time.

[0331] Step 3:

[0332] The emotion engine analyzes the received facial expression and voice data to identify the user's emotional state. For example, it recognizes emotions such as "joy" when there are many smiles and "calmness" when there is prolonged silence.

[0333] Step 4:

[0334] The device sends the analyzed emotional data to the server. Along with this, information about the user's preferred themes and locations is also sent to the server.

[0335] Step 5:

[0336] The server retrieves real-time weather and time data from external APIs. Based on this data, it integrates all the information necessary for generating video.

[0337] Step 6:

[0338] The server's AI technology uses user preferences, emotional states, weather data, and time data to generate optimal video. This process adjusts elements such as the video's color tone, music, and motion.

[0339] Step 7:

[0340] The server sends the generated video data to the terminal. The terminal then displays the video optimized for the monitor based on this data.

[0341] Step 8:

[0342] The device continuously collects emotional data while displaying video to the user, and updates the video by sending new data to the server as needed. This ensures that the video is always appropriate to the user's state.

[0343] (Example 2)

[0344] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0345] Conventional video generation systems generate videos without considering the user's emotional state, making it difficult to provide a video experience tailored to the user's individual emotions. Furthermore, they were insufficient in generating personalized videos that responded to real-time environmental changes. This resulted in decreased user satisfaction.

[0346] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0347] In this invention, the server includes means for analyzing the user's emotions, means for generating video based on the user's preferences, and means for acquiring real-time environmental data. This makes it possible to provide personalized video based on the user's emotions and environmental conditions.

[0348] A "user" is an individual who uses the system to receive personalized, emotion-based videos.

[0349] "Methods for analyzing emotions" refer to a technical process that determines emotions based on a user's facial expressions and voice, and provides this information as data.

[0350] "Means of generating video" refers to technologies used to create unique videos based on the user's emotions and preferences.

[0351] "Real-time environmental data" refers to information that can instantly obtain current weather, time, and other environmental conditions.

[0352] "Updated footage" refers to the latest footage generated to reflect user sentiment data and real-time environmental data.

[0353] A "display device" is a device used to visually present generated video to the user.

[0354] "Generative artificial intelligence" is an AI technology used to dynamically adjust each element of a video based on user input data.

[0355] This invention relates to a system that generates personalized videos based on the user's emotions. The system consists of a terminal, a server, an emotion analysis engine, and a generative AI model.

[0356] First, the user selects the theme of the video they want to watch through their device. The device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice in real time. This emotional data is immediately sent to an emotion analysis engine, which analyzes the user's emotional state. During the analysis process, a generative AI model is used to identify emotions such as joy, sadness, and surprise from the user's smile and voice tone.

[0357] The analyzed emotional data is sent to a server in the cloud. The server uses generative AI technology to generate video based on the user's emotional data and selected theme. Real-time weather and time data are also taken into consideration when generating the video. For example, a user who wants to relax on a rainy day will be provided with a calm indoor scene.

[0358] The generated video data is sent from the server to the terminal. The terminal displays the received video on its screen, providing the user with the most suitable video experience. This allows the user to enjoy videos that match their emotions.

[0359] As a concrete example, consider a scenario where a user is in a cheerful mood and selects the "Caribbean Beach" theme. In this case, the emotion analysis engine recognizes the user's smile, and the server generates an image including bright sunshine, a clear ocean, and upbeat music. This process occurs in real time, allowing the user to enjoy the results immediately.

[0360] An example of a prompt when using a generative AI model is the instruction, "When the user smiles, generate a video of a sunny beach and cheerful music." Based on this, the AI ​​dynamically adjusts each element of the video to provide the optimal experience.

[0361] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0362] Step 1:

[0363] The user operates the device and selects the theme of the video they want to watch. The theme data is obtained as input information. The device records this theme information and prepares to activate the camera and microphone. This prepares the device to capture the user's emotions.

[0364] Step 2:

[0365] The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time. Emotional data is acquired during this process. The collected data is immediately sent to an emotion analysis engine. Here, a generative AI model is used to analyze the emotions from the user's facial expressions and voice based on the emotional data. The output is classification information of the analyzed emotions.

[0366] Step 3:

[0367] The terminal sends analyzed emotion data to the server. This input includes emotion classification information and theme data. The server uses a generative AI to dynamically generate video based on the received emotion data and theme. Specifically, it uses prompts to select the color tone and music for the video, generating optimized video data. The output is the generated customized video data.

[0368] Step 4:

[0369] The server sends the generated video data to the terminal. This input includes the completed video data. The terminal prepares to play the received video on its display and prompts the user to start viewing. The output is a state where the user is viewing a personalized video on the display.

[0370] Step 5:

[0371] Users view videos through a display and enjoy an emotionally responsive video experience. The system continuously optimizes the user experience by collecting new emotional data in real time as needed and updating the video accordingly. As an output, user satisfaction increases.

[0372] (Application Example 2)

[0373] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0374] In recent years, personalizing content based on user emotions has become increasingly important, but conventional systems have struggled to accurately reflect users' emotional states in video content. Therefore, the challenge lies in appropriately adjusting content based on users' current emotions and providing experiences tailored to their individual needs.

[0375] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0376] In this invention, the server includes means for acquiring user emotion data, means for using artificial intelligence to generate video based on the emotion data, and means for acquiring real-time environmental data. This makes it possible to generate personalized video that reflects the user's emotions in real time.

[0377] "User emotional data" refers to data obtained from a user's facial expressions and voice tone, and represents information that indicates the user's current emotional state.

[0378] "Artificial intelligence" is a technology in which computer systems imitate human intellectual activity, and in particular, it has the ability to dynamically adjust content based on emotional data in areas such as video generation and data analysis.

[0379] "Video generation" is the process of creating new video content based on user emotional data and other relevant data.

[0380] "Real-time environmental data" refers to information that includes current weather information and time-related data, and is used to make adjustments to suit the user's situation.

[0381] "Personalized video" refers to video content customized according to the individual preferences and emotional state of a user, providing a visual experience optimized for each user.

[0382] The system for implementing this invention provides a personalized video experience by acquiring user emotion data and generating emotion-based videos in real time. The system mainly consists of three components: a terminal, a server, and the user.

[0383] The device directly interfaces with the user and captures the user's facial expressions and voice tone through its camera and microphone. This data is used to analyze facial expressions using OpenCV and to extract emotional data from the voice using Google AI's Speech-to-Text API.

[0384] The server uses a generative AI model to generate user-friendly videos based on emotional data sent from the terminal. It performs emotional analysis using frameworks such as TensorFlow and PyTorch, and dynamically adjusts the video's color tone, music, and environmental elements based on the results. This process generates personalized videos that match the user's emotions.

[0385] Specifically, when a user is experiencing stress, the system provides videos featuring relaxing natural scenery and soothing music. Conversely, when a user is feeling happy, the system generates videos with bright colors and cheerful music. The generated video data is then transmitted from the server to the user's device and provided to them.

[0386] This system uses prompt statements like the following as input to the generated AI model.

[0387] "The user's current emotional state is 'sadness,' so please provide relaxing visuals."

[0388] "To put the user in a 'joyful' state, please generate a video clip of a city with bright weather accompanied by energetic music."

[0389] The introduction of this system will enable users to enjoy a video experience tailored to their individual needs in real time, leading to a deeper level of satisfaction.

[0390] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0391] Step 1:

[0392] The device uses a camera and microphone to capture the user's facial expressions and voice tone. The input is raw video and audio data, which is captured in real time. The output is sentiment data in an analyzable format.

[0393] Step 2:

[0394] The device uses OpenCV to analyze facial expressions, converts audio data to text using Google AI's Speech-to-Text API, and then estimates the emotion. The input is the video and audio data obtained in step 1, and the output is data representing the user's current emotional state (e.g., joy, sadness, surprise).

[0395] Step 3:

[0396] The device sends analyzed emotion data to the server. The input is data containing the user's emotional state, and the output is a status indicating that the data transmission to the server was successful.

[0397] Step 4:

[0398] The server uses the received emotion data to activate a generative AI model and generate video content tailored to the user. The input is the user's emotion data, which is used to instruct the generative AI model using prompts. The output is dynamically adjusted video content.

[0399] Step 5:

[0400] The server generates video content and sends it to the terminal. The input is video data generated by AI, and the output is the state of transmission to the user's terminal.

[0401] Step 6:

[0402] The device plays back the video data it receives and provides it to the user. The input is the video data received from the server, and the output is the video the user views. During playback, the device optimizes the user experience by setting the optimal display settings based on environmental factors.

[0403] This processing flow allows users to enjoy personalized videos that match their emotions in real time.

[0404] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0405] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0406] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0407] [Third Embodiment]

[0408] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0409] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0410] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0411] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0412] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0413] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0414] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0415] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0416] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0417] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0418] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0419] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0420] A specific embodiment for carrying out the present invention is shown below. This system personalizes the video experience based on the user's preferences and dynamically updates the video using real-time weather and time data.

[0421] First, users use their devices to set their preferred themes and locations. These settings include visual themes such as "tropical beach" or "European cityscape." This settings data is formatted by the device and sent to the server.

[0422] The server stores the received user settings information in a database, obtains weather data from external weather information services, and also collects time data. Based on this collected data, AI technology within the server generates video optimized for the user. This AI is driven by a specific program and automatically adjusts the color tone, atmosphere, and dynamic elements of the video to match the user's preferences.

[0423] The generated video data is transmitted to the user's terminal via the network and reflected on the display device. On this display device, the video is updated in real time according to weather and time, providing the user with a realistic and immersive visual experience.

[0424] As a concrete example, consider a scenario where a user selects the themes "Hawaii beach during the day" and "San Francisco night view." In this case, the server generates images of a sunny beach or a city illuminated at night based on the weather in the respective region, providing the optimal display according to the user's time of day. Through these images, the user can experience the feeling of being in those locations, even though they cannot actually visit them.

[0425] Thus, the present invention makes it possible to obtain a mental refreshing effect by providing users with realistic and personalized images.

[0426] The following describes the processing flow.

[0427] Step 1:

[0428] Users interact with applications on their devices to select their preferred themes, locations, times of day, and music genres. This is done through the user interface and the preferences are entered into the device as data.

[0429] Step 2:

[0430] The device organizes the user-specified configuration information and sends it to the server as formatted data. This allows for personalized information based on the user's requests.

[0431] Step 3:

[0432] The server stores user configuration information received from the terminal in a database. This information will be used as a basis for future reference and video generation.

[0433] Step 4:

[0434] The server accesses external weather APIs and time data APIs to obtain the current weather and time for the location selected by the user. This allows for the acquisition of real-time environmental data.

[0435] Step 5:

[0436] The server uses AI to combine user preference information with acquired real-time data to generate optimal video content. The AI ​​generates visual elements, color schemes, and dynamic scenes according to the selected theme.

[0437] Step 6:

[0438] The server sends the generated video data to the terminal. After receiving the data, the terminal prepares to display it on the display device.

[0439] Step 7:

[0440] The terminal outputs the received video data to a display device, providing the user with a visual experience. It also quickly reflects new data when updates are periodically received from the server.

[0441] (Example 1)

[0442] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0443] There is a need for a system that can generate visual data tailored to individual user preferences in real time and dynamically update it in response to environmental changes. Conventional systems have the problem of not being able to effectively combine user settings information with real-time environmental data, which limits the quality of the visual experience.

[0444] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0445] In this invention, the server includes means for generating visual data based on user choices, means for acquiring real-time environmental data and time information from external sources, and means for dynamically updating the visual data based on the environmental data and time information. This makes it possible to provide a visual experience personalized to the user's individual preferences based on real-time environmental changes.

[0446] "User" refers to an individual or group that operates or configures an information processing device.

[0447] "Options" refer to settings that allow users to specify their preferences and conditions when generating visual data.

[0448] "Visual data" refers to video and image data generated to provide users with a visual experience.

[0449] "Environmental data" refers to data such as weather and geographical information obtained from external sources for use in generating and updating visual data.

[0450] "Time information" refers to information about the current time used to influence the generation and updating of visual data.

[0451] "Dynamic updating" refers to the process of adapting the content of visual data based on environmental data and time information that change in real time.

[0452] A "display device" refers to a device such as a screen or projector used to show generated visual data to the user.

[0453] An "information processing device" refers to a computer or server that performs a series of functions, such as generating visual data based on user selection and updating it according to the environment.

[0454] This system aims to provide users with personalized visual experiences, specifically by generating visual data based on themes and locations selected by the user. The server is primarily responsible for generating and dynamically updating visual data using machine learning techniques. Specifically, the server uses software platforms such as TensorFlow and OpenCV to create visual data by utilizing environmental data and time information obtained from external sources.

[0455] Users input their preferred themes through their device. This device can be a smartphone, tablet, or personal computer. Themes selected by users include options such as "tropical beach" or "European cityscape," and this information is sent from the device to the server. Based on this information, the server uses a generative AI model to construct optimal visual data.

[0456] Furthermore, the server acquires real-time environmental data from external weather information services and also obtains current time information from the system. By integrating this information, the server can provide personalized visual data tailored to each user's environment. The visual data generated by the server is transmitted to the terminal via the network and displayed on the terminal's display device.

[0457] As a concrete example, consider a scenario where a user selects "a beach in Hawaii during the daytime" using their device. In this case, the server generates video footage of a sunny beach based on acquired Hawaiian weather data and inserts it at daytime. This visual data provides the user with the illusion of actually being in Hawaii.

[0458] As an example of a prompt, by sending a specific request to the generation AI model, such as "Generate a video themed around a tropical beach, and make it realistic by matching it with the current weather," a video tailored to the user's needs will be generated.

[0459] In this way, the system can provide users with a rich and personalized visual experience, resulting in a better user experience.

[0460] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0461] Step 1:

[0462] Input: Themes and location information selected by the user via their device.

[0463] Specific operation: The user uses a smartphone or PC application to select a theme such as "tropical beach" or "European cityscape."

[0464] Data processing and output: The terminal formats this information in XML or JSON format and sends it to the server. The transmitted data is stored on the server side as user-specific configuration information.

[0465] Step 2:

[0466] Input: User theme information sent to the server.

[0467] Specific operation: Based on the received information, the server calls an external weather information service API to obtain real-time weather data. It also obtains the current time information from the system's internal clock.

[0468] Data processing and output: Weather data and time information are stored on the server as JSON objects.

[0469] Step 3:

[0470] Input: User theme information and weather and time information obtained from external sources.

[0471] Specific operation: Based on this information, the server activates the generative AI model and starts the process of generating visual data. During this process, TensorFlow is used to input "prompt statements" to the AI ​​model.

[0472] Data Processing and Output: The AI ​​model analyzes the input data and generates video data with adjusted color tones and effects based on the user's preferences. The generated video data is stored in memory.

[0473] Step 4:

[0474] Input: Video data generated by an AI model.

[0475] Specific operation: The server transmits the generated video data to the user's terminal via the network.

[0476] Data Processing and Output: Video data is transmitted using the HTTP protocol, and the terminal renders the received data on the display device. This allows users to view the visual data in real time.

[0477] Step 5:

[0478] Input: Displayed video data, real-time weather, and time-of-day information.

[0479] Specific operation: The user's device periodically connects to the server to obtain new weather data and time information.

[0480] Data Processing and Output: The acquired data is used to update the visual data, dynamically switching to images that match the current situation on the device. This ensures that the visual experience is always updated based on the latest real-time data.

[0481] (Application Example 1)

[0482] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0483] Currently, there are limited means of easily enjoying personalized visual information by providing virtual experiences in real time based on the environment and time that users wish to experience at that moment. The present invention aims to provide a new method for generating immersive virtual experiences that match the user's preferences by dynamically adapting to the real environment and time.

[0484] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0485] In this invention, the server includes means for generating information based on user preferences, means for acquiring real-time environmental and time data, and means for updating the information based on the environmental and time data. This allows the virtual environment to adapt to real-world conditions in real time according to the user's selection, enabling a more personalized experience.

[0486] A "user" refers to a person who uses the system, receives information, and enjoys virtual experiences based on their individual wishes and preferences.

[0487] "Preferences" refer to the specific tendencies and tastes that individual users possess, and are elements that should be considered when selecting and presenting visual information.

[0488] "Information" refers to the data and content that constitute virtual environments and experiences generated based on user preferences.

[0489] "Generative means" refers to the processes and technologies used to construct data according to user preferences and to embody it as visual or experiential information.

[0490] "Environmental data" refers to information about a specific location or situation collected in real time, including weather conditions and surrounding environment.

[0491] "Time data" refers to information about the time of day that is considered when the information is generated, and includes elements based on time of day, such as daytime or nighttime.

[0492] "Means of updating" refers to methods or processes for appropriately modifying or enhancing the content of previously generated information based on acquired real-time data.

[0493] "Display medium" refers to a device or screen that allows users to visually experience a generated virtual environment or information.

[0494] A "virtual experience" refers to a synthetic environment or event that allows users to obtain similar sensations and impressions without directly experiencing them in the real world.

[0495] The following system configuration is necessary to implement the present invention.

[0496] The server runs software to generate information based on user preferences and acquire real-time environmental and time data. Specifically, the server uses machine learning techniques to analyze environmental data and generate visual information tailored to the user's preferences. The hardware used would be a standard network-connected computing device, and the software could utilize Python or data processing libraries (e.g., Pandas, NumPy).

[0497] The terminal is an information display device owned by the user, which displays updated information received from the server. Portable devices such as smartphones and tablets are used for this purpose. The terminal interprets the information provided by the server and runs applications to provide the user with a visual experience. For example, libraries (e.g., PIL, OpenCV) can be used to visually represent real-time environmental changes.

[0498] Users access the system and select their desired theme and time slot. Through the interface, users input their preferences and specify a particular virtual experience. The entered data is sent to the server, and information generated based on this is fed back to the terminal, allowing users to experience environments that are difficult to encounter in reality.

[0499] For example, if a user selects "tropical beach" and requests a sunny midday, the server generates visual information of a sunny afternoon beach in Hawaii and sends it to the device. This allows the user to enjoy the feeling of being there, both visually and psychologically, even while at home.

[0500] A concrete example of a prompt message for a generative AI model would be: "Generate a dynamic image based on the theme the user has chosen, taking into account the local weather and time of day. For example, if the user selects 'tropical beach' and the local weather is sunny, create an image of a white sand beach reflecting sunlight."

[0501] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0502] Step 1:

[0503] The user selects their preferred theme and desired experience using the device's interface. The user's preferences and selections are provided as input, and this is output as data sent from the device to the server.

[0504] Step 2:

[0505] The server receives selection data submitted by the user and retrieves real-time environmental and time data from external environmental data sources. In this step, the input consists of user preference data and retrieved environmental data, and these data are used to create the conditions necessary for information generation.

[0506] Step 3:

[0507] The server uses an AI model based on acquired environmental and time data to generate virtual information tailored to the user's preferences. Specifically, preference data and environmental data are input to the AI ​​model via prompts, and visual virtual information is obtained as output.

[0508] Step 4:

[0509] The generated information is sent from the server to the terminal. Here, virtual information is the input, and the data transfer to the terminal is the output. This process takes place over a network.

[0510] Step 5:

[0511] The terminal displays virtual information received from the server. It displays the virtual information as input on the display device and performs specific actions to provide the user with a preferred experience.

[0512] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0513] A specific embodiment for carrying out the present invention is shown below. This system provides personalized video that responds to the user's emotions by combining an emotion engine that recognizes the user's emotions.

[0514] The user selects their preferred theme through the device and collects emotional data using a camera and microphone for emotion recognition. The device immediately passes this emotional data to an emotion engine, which analyzes the user's current emotional state. The emotion engine classifies emotions such as joy, sadness, and surprise based on the user's voice tone and facial expressions.

[0515] The analyzed emotional data is sent from the device to the server, which uses this data to generate video. AI technology is particularly used to dynamically adjust the video's color tone, music, and environmental elements based on the emotional data and user preferences. For example, when a user wants to relax, the video is generated with calming colors and soothing music.

[0516] Once this generation process is complete, the server sends the generated video data to the terminal. The terminal projects the latest video data onto a display device, providing the user with the most suitable video experience. This system also takes real-time weather and time data into consideration, so for example, if a user is feeling gloomy on a rainy day, it can provide a warm indoor scene.

[0517] As a concrete example, suppose a user is in a cheerful mood and has selected the theme "Caribbean beach." In this case, the emotion engine detects the user's smile, and the server generates a video showing bright sunshine and a clear ocean, mixed with upbeat music. Through this process, the user is provided with a video that matches their emotions, leading to a deeper sense of satisfaction.

[0518] The introduction of this technology will enable users to not only obtain visual information but also enjoy new experiences that are tailored to their emotions.

[0519] The following describes the processing flow.

[0520] Step 1:

[0521] The user launches the application on their device and enables their preferred video theme and the on-device camera and microphone. This prepares the system for inputting the basic data for video generation.

[0522] Step 2:

[0523] The device collects user emotion data by capturing the user's facial expressions with a camera and recording their voice tone with a microphone. This data is then passed to the emotion engine in real time.

[0524] Step 3:

[0525] The emotion engine analyzes the received facial expression and voice data to identify the user's emotional state. For example, it recognizes emotions such as "joy" when there are many smiles and "calmness" when there is prolonged silence.

[0526] Step 4:

[0527] The device sends the analyzed emotional data to the server. Along with this, information about the user's preferred themes and locations is also sent to the server.

[0528] Step 5:

[0529] The server retrieves real-time weather and time data from external APIs. Based on this data, it integrates all the information necessary for generating video.

[0530] Step 6:

[0531] The server's AI technology uses user preferences, emotional states, weather data, and time data to generate optimal video. This process adjusts elements such as the video's color tone, music, and motion.

[0532] Step 7:

[0533] The server sends the generated video data to the terminal. The terminal then displays the video optimized for the monitor based on this data.

[0534] Step 8:

[0535] The device continuously collects emotional data while displaying video to the user, and updates the video by sending new data to the server as needed. This ensures that the video is always appropriate to the user's state.

[0536] (Example 2)

[0537] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0538] Conventional video generation systems generate videos without considering the user's emotional state, making it difficult to provide a video experience tailored to the user's individual emotions. Furthermore, they were insufficient in generating personalized videos that responded to real-time environmental changes. This resulted in decreased user satisfaction.

[0539] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0540] In this invention, the server includes means for analyzing the user's emotions, means for generating video based on the user's preferences, and means for acquiring real-time environmental data. This makes it possible to provide personalized video based on the user's emotions and environmental conditions.

[0541] A "user" is an individual who uses the system to receive personalized, emotion-based videos.

[0542] "Methods for analyzing emotions" refer to a technical process that determines emotions based on a user's facial expressions and voice, and provides this information as data.

[0543] "Means of generating video" refers to technologies used to create unique videos based on the user's emotions and preferences.

[0544] "Real-time environmental data" refers to information that can instantly obtain current weather, time, and other environmental conditions.

[0545] "Updated footage" refers to the latest footage generated to reflect user sentiment data and real-time environmental data.

[0546] A "display device" is a device used to visually present generated video to the user.

[0547] "Generative artificial intelligence" is an AI technology used to dynamically adjust each element of a video based on user input data.

[0548] This invention relates to a system that generates personalized videos based on the user's emotions. The system consists of a terminal, a server, an emotion analysis engine, and a generative AI model.

[0549] First, the user selects the theme of the video they want to watch through their device. The device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice in real time. This emotional data is immediately sent to an emotion analysis engine, which analyzes the user's emotional state. During the analysis process, a generative AI model is used to identify emotions such as joy, sadness, and surprise from the user's smile and voice tone.

[0550] The analyzed emotional data is sent to a server in the cloud. The server uses generative AI technology to generate video based on the user's emotional data and selected theme. Real-time weather and time data are also taken into consideration when generating the video. For example, a user who wants to relax on a rainy day will be provided with a calm indoor scene.

[0551] The generated video data is sent from the server to the terminal. The terminal displays the received video on its screen, providing the user with the most suitable video experience. This allows the user to enjoy videos that match their emotions.

[0552] As a concrete example, consider a scenario where a user is in a cheerful mood and selects the "Caribbean Beach" theme. In this case, the emotion analysis engine recognizes the user's smile, and the server generates an image including bright sunshine, a clear ocean, and upbeat music. This process occurs in real time, allowing the user to enjoy the results immediately.

[0553] An example of a prompt when using a generative AI model is the instruction, "When the user smiles, generate a video of a sunny beach and cheerful music." Based on this, the AI ​​dynamically adjusts each element of the video to provide the optimal experience.

[0554] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0555] Step 1:

[0556] The user operates the device and selects the theme of the video they want to watch. The theme data is obtained as input information. The device records this theme information and prepares to activate the camera and microphone. This prepares the device to capture the user's emotions.

[0557] Step 2:

[0558] The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time. Emotional data is acquired during this process. The collected data is immediately sent to an emotion analysis engine. Here, a generative AI model is used to analyze the emotions from the user's facial expressions and voice based on the emotional data. The output is classification information of the analyzed emotions.

[0559] Step 3:

[0560] The terminal sends analyzed emotion data to the server. This input includes emotion classification information and theme data. The server uses a generative AI to dynamically generate video based on the received emotion data and theme. Specifically, it uses prompts to select the color tone and music for the video, generating optimized video data. The output is the generated customized video data.

[0561] Step 4:

[0562] The server sends the generated video data to the terminal. This input includes the completed video data. The terminal prepares to play the received video on its display and prompts the user to start viewing. The output is a state where the user is viewing a personalized video on the display.

[0563] Step 5:

[0564] Users view videos through a display and enjoy an emotionally responsive video experience. The system continuously optimizes the user experience by collecting new emotional data in real time as needed and updating the video accordingly. As an output, user satisfaction increases.

[0565] (Application Example 2)

[0566] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0567] In recent years, personalizing content based on user emotions has become increasingly important, but conventional systems have struggled to accurately reflect users' emotional states in video content. Therefore, the challenge lies in appropriately adjusting content based on users' current emotions and providing experiences tailored to their individual needs.

[0568] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0569] In this invention, the server includes means for acquiring user emotion data, means for using artificial intelligence to generate video based on the emotion data, and means for acquiring real-time environmental data. This makes it possible to generate personalized video that reflects the user's emotions in real time.

[0570] "User emotional data" refers to data obtained from a user's facial expressions and voice tone, and represents information that indicates the user's current emotional state.

[0571] "Artificial intelligence" is a technology in which computer systems imitate human intellectual activity, and in particular, it has the ability to dynamically adjust content based on emotional data in areas such as video generation and data analysis.

[0572] "Video generation" is the process of creating new video content based on user emotional data and other relevant data.

[0573] "Real-time environmental data" refers to information that includes current weather information and time-related data, and is used to make adjustments to suit the user's situation.

[0574] "Personalized video" refers to video content customized according to the individual preferences and emotional state of a user, providing a visual experience optimized for each user.

[0575] The system for implementing this invention provides a personalized video experience by acquiring user emotion data and generating emotion-based videos in real time. The system mainly consists of three components: a terminal, a server, and the user.

[0576] The device directly interfaces with the user and captures the user's facial expressions and voice tone through its camera and microphone. This data is used to analyze facial expressions using OpenCV and to extract emotional data from the voice using Google AI's Speech-to-Text API.

[0577] The server uses a generative AI model to generate user-friendly videos based on emotional data sent from the terminal. It performs emotional analysis using frameworks such as TensorFlow and PyTorch, and dynamically adjusts the video's color tone, music, and environmental elements based on the results. This process generates personalized videos that match the user's emotions.

[0578] Specifically, when a user is experiencing stress, the system provides videos featuring relaxing natural scenery and soothing music. Conversely, when a user is feeling happy, the system generates videos with bright colors and cheerful music. The generated video data is then transmitted from the server to the user's device and provided to them.

[0579] This system uses prompt statements like the following as input to the generated AI model.

[0580] "The user's current emotional state is 'sadness,' so please provide relaxing visuals."

[0581] "To put the user in a 'joyful' state, please generate a video clip of a city with bright weather accompanied by energetic music."

[0582] The introduction of this system will enable users to enjoy a video experience tailored to their individual needs in real time, leading to a deeper level of satisfaction.

[0583] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0584] Step 1:

[0585] The device uses a camera and microphone to capture the user's facial expressions and voice tone. The input is raw video and audio data, which is captured in real time. The output is sentiment data in an analyzable format.

[0586] Step 2:

[0587] The device uses OpenCV to analyze facial expressions, converts audio data to text using Google AI's Speech-to-Text API, and then estimates the emotion. The input is the video and audio data obtained in step 1, and the output is data representing the user's current emotional state (e.g., joy, sadness, surprise).

[0588] Step 3:

[0589] The device sends analyzed emotion data to the server. The input is data containing the user's emotional state, and the output is a status indicating that the data transmission to the server was successful.

[0590] Step 4:

[0591] The server uses the received emotion data to activate a generative AI model and generate video content tailored to the user. The input is the user's emotion data, which is used to instruct the generative AI model using prompts. The output is dynamically adjusted video content.

[0592] Step 5:

[0593] The server generates video content and sends it to the terminal. The input is video data generated by AI, and the output is the state of transmission to the user's terminal.

[0594] Step 6:

[0595] The device plays back the video data it receives and provides it to the user. The input is the video data received from the server, and the output is the video the user views. During playback, the device optimizes the user experience by setting the optimal display settings based on environmental factors.

[0596] This processing flow allows users to enjoy personalized videos that match their emotions in real time.

[0597] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0598] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0599] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0600] [Fourth Embodiment]

[0601] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0602] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0603] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0604] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0605] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0606] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0607] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0608] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0609] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0610] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0611] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0612] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0613] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0614] A specific embodiment for carrying out the present invention is shown below. This system personalizes the video experience based on the user's preferences and dynamically updates the video using real-time weather and time data.

[0615] First, users use their devices to set their preferred themes and locations. These settings include visual themes such as "tropical beach" or "European cityscape." This settings data is formatted by the device and sent to the server.

[0616] The server stores the received user settings information in a database, obtains weather data from external weather information services, and also collects time data. Based on this collected data, AI technology within the server generates video optimized for the user. This AI is driven by a specific program and automatically adjusts the color tone, atmosphere, and dynamic elements of the video to match the user's preferences.

[0617] The generated video data is transmitted to the user's terminal via the network and reflected on the display device. On this display device, the video is updated in real time according to weather and time, providing the user with a realistic and immersive visual experience.

[0618] As a concrete example, consider a scenario where a user selects the themes "Hawaii beach during the day" and "San Francisco night view." In this case, the server generates images of a sunny beach or a city illuminated at night based on the weather in the respective region, providing the optimal display according to the user's time of day. Through these images, the user can experience the feeling of being in those locations, even though they cannot actually visit them.

[0619] Thus, the present invention makes it possible to obtain a mental refreshing effect by providing users with realistic and personalized images.

[0620] The following describes the processing flow.

[0621] Step 1:

[0622] Users interact with applications on their devices to select their preferred themes, locations, times of day, and music genres. This is done through the user interface and the preferences are entered into the device as data.

[0623] Step 2:

[0624] The device organizes the user-specified configuration information and sends it to the server as formatted data. This allows for personalized information based on the user's requests.

[0625] Step 3:

[0626] The server stores user configuration information received from the terminal in a database. This information will be used as a basis for future reference and video generation.

[0627] Step 4:

[0628] The server accesses external weather APIs and time data APIs to obtain the current weather and time for the location selected by the user. This allows for the acquisition of real-time environmental data.

[0629] Step 5:

[0630] The server uses AI to combine user preference information with acquired real-time data to generate optimal video content. The AI ​​generates visual elements, color schemes, and dynamic scenes according to the selected theme.

[0631] Step 6:

[0632] The server sends the generated video data to the terminal. After receiving the data, the terminal prepares to display it on the display device.

[0633] Step 7:

[0634] The terminal outputs the received video data to a display device, providing the user with a visual experience. It also quickly reflects new data when updates are periodically received from the server.

[0635] (Example 1)

[0636] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0637] There is a need for a system that can generate visual data tailored to individual user preferences in real time and dynamically update it in response to environmental changes. Conventional systems have the problem of not being able to effectively combine user settings information with real-time environmental data, which limits the quality of the visual experience.

[0638] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0639] In this invention, the server includes means for generating visual data based on user choices, means for acquiring real-time environmental data and time information from external sources, and means for dynamically updating the visual data based on the environmental data and time information. This makes it possible to provide a visual experience personalized to the user's individual preferences based on real-time environmental changes.

[0640] "User" refers to an individual or group that operates or configures an information processing device.

[0641] "Options" refer to settings that allow users to specify their preferences and conditions when generating visual data.

[0642] "Visual data" refers to video and image data generated to provide users with a visual experience.

[0643] "Environmental data" refers to data such as weather and geographical information obtained from external sources for use in generating and updating visual data.

[0644] "Time information" refers to information about the current time used to influence the generation and updating of visual data.

[0645] "Dynamic updating" refers to the process of adapting the content of visual data based on environmental data and time information that change in real time.

[0646] A "display device" refers to a device such as a screen or projector used to show generated visual data to the user.

[0647] An "information processing device" refers to a computer or server that performs a series of functions, such as generating visual data based on user selection and updating it according to the environment.

[0648] This system aims to provide users with personalized visual experiences, specifically by generating visual data based on themes and locations selected by the user. The server is primarily responsible for generating and dynamically updating visual data using machine learning techniques. Specifically, the server uses software platforms such as TensorFlow and OpenCV to create visual data by utilizing environmental data and time information obtained from external sources.

[0649] Users input their preferred themes through their device. This device can be a smartphone, tablet, or personal computer. Themes selected by users include options such as "tropical beach" or "European cityscape," and this information is sent from the device to the server. Based on this information, the server uses a generative AI model to construct optimal visual data.

[0650] Furthermore, the server acquires real-time environmental data from external weather information services and also obtains current time information from the system. By integrating this information, the server can provide personalized visual data tailored to each user's environment. The visual data generated by the server is transmitted to the terminal via the network and displayed on the terminal's display device.

[0651] As a concrete example, consider a scenario where a user selects "a beach in Hawaii during the daytime" using their device. In this case, the server generates video footage of a sunny beach based on acquired Hawaiian weather data and inserts it at daytime. This visual data provides the user with the illusion of actually being in Hawaii.

[0652] As an example of a prompt, by sending a specific request to the generation AI model, such as "Generate a video themed around a tropical beach, and make it realistic by matching it with the current weather," a video tailored to the user's needs will be generated.

[0653] In this way, the system can provide users with a rich and personalized visual experience, resulting in a better user experience.

[0654] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0655] Step 1:

[0656] Input: Themes and location information selected by the user via their device.

[0657] Specific operation: The user uses a smartphone or PC application to select a theme such as "tropical beach" or "European cityscape."

[0658] Data processing and output: The terminal formats this information in XML or JSON format and sends it to the server. The transmitted data is stored on the server side as user-specific configuration information.

[0659] Step 2:

[0660] Input: User theme information sent to the server.

[0661] Specific operation: Based on the received information, the server calls an external weather information service API to obtain real-time weather data. It also obtains the current time information from the system's internal clock.

[0662] Data processing and output: Weather data and time information are stored on the server as JSON objects.

[0663] Step 3:

[0664] Input: User theme information and weather and time information obtained from external sources.

[0665] Specific operation: Based on this information, the server activates the generative AI model and starts the process of generating visual data. During this process, TensorFlow is used to input "prompt statements" to the AI ​​model.

[0666] Data Processing and Output: The AI ​​model analyzes the input data and generates video data with adjusted color tones and effects based on the user's preferences. The generated video data is stored in memory.

[0667] Step 4:

[0668] Input: Video data generated by an AI model.

[0669] Specific operation: The server transmits the generated video data to the user's terminal via the network.

[0670] Data Processing and Output: Video data is transmitted using the HTTP protocol, and the terminal renders the received data on the display device. This allows users to view the visual data in real time.

[0671] Step 5:

[0672] Input: Displayed video data, real-time weather, and time-of-day information.

[0673] Specific operation: The user's device periodically connects to the server to obtain new weather data and time information.

[0674] Data Processing and Output: The acquired data is used to update the visual data, dynamically switching to images that match the current situation on the device. This ensures that the visual experience is always updated based on the latest real-time data.

[0675] (Application Example 1)

[0676] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0677] Currently, there are limited means of easily enjoying personalized visual information by providing virtual experiences in real time based on the environment and time that users wish to experience at that moment. The present invention aims to provide a new method for generating immersive virtual experiences that match the user's preferences by dynamically adapting to the real environment and time.

[0678] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0679] In this invention, the server includes means for generating information based on user preferences, means for acquiring real-time environmental and time data, and means for updating the information based on the environmental and time data. This allows the virtual environment to adapt to real-world conditions in real time according to the user's selection, enabling a more personalized experience.

[0680] A "user" refers to a person who uses the system, receives information, and enjoys virtual experiences based on their individual wishes and preferences.

[0681] "Preferences" refer to the specific tendencies and tastes that individual users possess, and are elements that should be considered when selecting and presenting visual information.

[0682] "Information" refers to the data and content that constitute virtual environments and experiences generated based on user preferences.

[0683] "Generative means" refers to the processes and technologies used to construct data according to user preferences and to embody it as visual or experiential information.

[0684] "Environmental data" refers to information about a specific location or situation collected in real time, including weather conditions and surrounding environment.

[0685] "Time data" refers to information about the time of day that is considered when the information is generated, and includes elements based on time of day, such as daytime or nighttime.

[0686] "Means of updating" refers to methods or processes for appropriately modifying or enhancing the content of previously generated information based on acquired real-time data.

[0687] "Display medium" refers to a device or screen that allows users to visually experience a generated virtual environment or information.

[0688] A "virtual experience" refers to a synthetic environment or event that allows users to obtain similar sensations and impressions without directly experiencing them in the real world.

[0689] The following system configuration is necessary to implement the present invention.

[0690] The server runs software to generate information based on user preferences and acquire real-time environmental and time data. Specifically, the server uses machine learning techniques to analyze environmental data and generate visual information tailored to the user's preferences. The hardware used would be a standard network-connected computing device, and the software could utilize Python or data processing libraries (e.g., Pandas, NumPy).

[0691] The terminal is an information display device owned by the user, which displays updated information received from the server. Portable devices such as smartphones and tablets are used for this purpose. The terminal interprets the information provided by the server and runs applications to provide the user with a visual experience. For example, libraries (e.g., PIL, OpenCV) can be used to visually represent real-time environmental changes.

[0692] Users access the system and select their desired theme and time slot. Through the interface, users input their preferences and specify a particular virtual experience. The entered data is sent to the server, and information generated based on this is fed back to the terminal, allowing users to experience environments that are difficult to encounter in reality.

[0693] For example, if a user selects "tropical beach" and requests a sunny midday, the server generates visual information of a sunny afternoon beach in Hawaii and sends it to the device. This allows the user to enjoy the feeling of being there, both visually and psychologically, even while at home.

[0694] A concrete example of a prompt message for a generative AI model would be: "Generate a dynamic image based on the theme the user has chosen, taking into account the local weather and time of day. For example, if the user selects 'tropical beach' and the local weather is sunny, create an image of a white sand beach reflecting sunlight."

[0695] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0696] Step 1:

[0697] The user selects their preferred theme and desired experience using the device's interface. The user's preferences and selections are provided as input, and this is output as data sent from the device to the server.

[0698] Step 2:

[0699] The server receives selection data submitted by the user and retrieves real-time environmental and time data from external environmental data sources. In this step, the input consists of user preference data and retrieved environmental data, and these data are used to create the conditions necessary for information generation.

[0700] Step 3:

[0701] The server uses an AI model based on acquired environmental and time data to generate virtual information tailored to the user's preferences. Specifically, preference data and environmental data are input to the AI ​​model via prompts, and visual virtual information is obtained as output.

[0702] Step 4:

[0703] The generated information is sent from the server to the terminal. Here, virtual information is the input, and the data transfer to the terminal is the output. This process takes place over a network.

[0704] Step 5:

[0705] The terminal displays virtual information received from the server. It displays the virtual information as input on the display device and performs specific actions to provide the user with a preferred experience.

[0706] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0707] A specific embodiment for carrying out the present invention is shown below. This system provides personalized video that responds to the user's emotions by combining an emotion engine that recognizes the user's emotions.

[0708] The user selects their preferred theme through the device and collects emotional data using a camera and microphone for emotion recognition. The device immediately passes this emotional data to an emotion engine, which analyzes the user's current emotional state. The emotion engine classifies emotions such as joy, sadness, and surprise based on the user's voice tone and facial expressions.

[0709] The analyzed emotional data is sent from the device to the server, which uses this data to generate video. AI technology is particularly used to dynamically adjust the video's color tone, music, and environmental elements based on the emotional data and user preferences. For example, when a user wants to relax, the video is generated with calming colors and soothing music.

[0710] Once this generation process is complete, the server sends the generated video data to the terminal. The terminal projects the latest video data onto a display device, providing the user with the most suitable video experience. This system also takes real-time weather and time data into consideration, so for example, if a user is feeling gloomy on a rainy day, it can provide a warm indoor scene.

[0711] As a concrete example, suppose a user is in a cheerful mood and has selected the theme "Caribbean beach." In this case, the emotion engine detects the user's smile, and the server generates a video showing bright sunshine and a clear ocean, mixed with upbeat music. Through this process, the user is provided with a video that matches their emotions, leading to a deeper sense of satisfaction.

[0712] The introduction of this technology will enable users to not only obtain visual information but also enjoy new experiences that are tailored to their emotions.

[0713] The following describes the processing flow.

[0714] Step 1:

[0715] The user launches the application on their device and enables their preferred video theme and the on-device camera and microphone. This prepares the system for inputting the basic data for video generation.

[0716] Step 2:

[0717] The device collects user emotion data by capturing the user's facial expressions with a camera and recording their voice tone with a microphone. This data is then passed to the emotion engine in real time.

[0718] Step 3:

[0719] The emotion engine analyzes the received facial expression and voice data to identify the user's emotional state. For example, it recognizes emotions such as "joy" when there are many smiles and "calmness" when there is prolonged silence.

[0720] Step 4:

[0721] The device sends the analyzed emotional data to the server. Along with this, information about the user's preferred themes and locations is also sent to the server.

[0722] Step 5:

[0723] The server retrieves real-time weather and time data from external APIs. Based on this data, it integrates all the information necessary for generating video.

[0724] Step 6:

[0725] The server's AI technology uses user preferences, emotional states, weather data, and time data to generate optimal video. This process adjusts elements such as the video's color tone, music, and motion.

[0726] Step 7:

[0727] The server sends the generated video data to the terminal. The terminal then displays the video optimized for the monitor based on this data.

[0728] Step 8:

[0729] The device continuously collects emotional data while displaying video to the user, and updates the video by sending new data to the server as needed. This ensures that the video is always appropriate to the user's state.

[0730] (Example 2)

[0731] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0732] Conventional video generation systems generate videos without considering the user's emotional state, making it difficult to provide a video experience tailored to the user's individual emotions. Furthermore, they were insufficient in generating personalized videos that responded to real-time environmental changes. This resulted in decreased user satisfaction.

[0733] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0734] In this invention, the server includes means for analyzing the user's emotions, means for generating video based on the user's preferences, and means for acquiring real-time environmental data. This makes it possible to provide personalized video based on the user's emotions and environmental conditions.

[0735] A "user" is an individual who uses the system to receive personalized, emotion-based videos.

[0736] "Methods for analyzing emotions" refer to a technical process that determines emotions based on a user's facial expressions and voice, and provides this information as data.

[0737] "Means of generating video" refers to technologies used to create unique videos based on the user's emotions and preferences.

[0738] "Real-time environmental data" refers to information that can instantly obtain current weather, time, and other environmental conditions.

[0739] "Updated footage" refers to the latest footage generated to reflect user sentiment data and real-time environmental data.

[0740] A "display device" is a device used to visually present generated video to the user.

[0741] "Generative artificial intelligence" is an AI technology used to dynamically adjust each element of a video based on user input data.

[0742] This invention relates to a system that generates personalized videos based on the user's emotions. The system consists of a terminal, a server, an emotion analysis engine, and a generative AI model.

[0743] First, the user selects the theme of the video they want to watch through their device. The device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice in real time. This emotional data is immediately sent to an emotion analysis engine, which analyzes the user's emotional state. During the analysis process, a generative AI model is used to identify emotions such as joy, sadness, and surprise from the user's smile and voice tone.

[0744] The analyzed emotional data is sent to a server in the cloud. The server uses generative AI technology to generate video based on the user's emotional data and selected theme. Real-time weather and time data are also taken into consideration when generating the video. For example, a user who wants to relax on a rainy day will be provided with a calm indoor scene.

[0745] The generated video data is sent from the server to the terminal. The terminal displays the received video on its screen, providing the user with the most suitable video experience. This allows the user to enjoy videos that match their emotions.

[0746] As a concrete example, consider a scenario where a user is in a cheerful mood and selects the "Caribbean Beach" theme. In this case, the emotion analysis engine recognizes the user's smile, and the server generates an image including bright sunshine, a clear ocean, and upbeat music. This process occurs in real time, allowing the user to enjoy the results immediately.

[0747] An example of a prompt when using a generative AI model is the instruction, "When the user smiles, generate a video of a sunny beach and cheerful music." Based on this, the AI ​​dynamically adjusts each element of the video to provide the optimal experience.

[0748] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0749] Step 1:

[0750] The user operates the device and selects the theme of the video they want to watch. The theme data is obtained as input information. The device records this theme information and prepares to activate the camera and microphone. This prepares the device to capture the user's emotions.

[0751] Step 2:

[0752] The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time. Emotional data is acquired during this process. The collected data is immediately sent to an emotion analysis engine. Here, a generative AI model is used to analyze the emotions from the user's facial expressions and voice based on the emotional data. The output is classification information of the analyzed emotions.

[0753] Step 3:

[0754] The terminal sends analyzed emotion data to the server. This input includes emotion classification information and theme data. The server uses a generative AI to dynamically generate video based on the received emotion data and theme. Specifically, it uses prompts to select the color tone and music for the video, generating optimized video data. The output is the generated customized video data.

[0755] Step 4:

[0756] The server sends the generated video data to the terminal. This input includes the completed video data. The terminal prepares to play the received video on its display and prompts the user to start viewing. The output is a state where the user is viewing a personalized video on the display.

[0757] Step 5:

[0758] Users view videos through a display and enjoy an emotionally responsive video experience. The system continuously optimizes the user experience by collecting new emotional data in real time as needed and updating the video accordingly. As an output, user satisfaction increases.

[0759] (Application Example 2)

[0760] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0761] In recent years, personalizing content based on user emotions has become increasingly important, but conventional systems have struggled to accurately reflect users' emotional states in video content. Therefore, the challenge lies in appropriately adjusting content based on users' current emotions and providing experiences tailored to their individual needs.

[0762] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0763] In this invention, the server includes means for acquiring user emotion data, means for using artificial intelligence to generate video based on the emotion data, and means for acquiring real-time environmental data. This makes it possible to generate personalized video that reflects the user's emotions in real time.

[0764] "User emotional data" refers to data obtained from a user's facial expressions and voice tone, and represents information that indicates the user's current emotional state.

[0765] "Artificial intelligence" is a technology in which computer systems imitate human intellectual activity, and in particular, it has the ability to dynamically adjust content based on emotional data in areas such as video generation and data analysis.

[0766] "Video generation" is the process of creating new video content based on user emotional data and other relevant data.

[0767] "Real-time environmental data" refers to information that includes current weather information and time-related data, and is used to make adjustments to suit the user's situation.

[0768] "Personalized video" refers to video content customized according to the individual preferences and emotional state of a user, providing a visual experience optimized for each user.

[0769] The system for implementing this invention provides a personalized video experience by acquiring user emotion data and generating emotion-based videos in real time. The system mainly consists of three components: a terminal, a server, and the user.

[0770] The device directly interfaces with the user and captures the user's facial expressions and voice tone through its camera and microphone. This data is used to analyze facial expressions using OpenCV and to extract emotional data from the voice using Google AI's Speech-to-Text API.

[0771] The server uses a generative AI model to generate user-friendly videos based on emotional data sent from the terminal. It performs emotional analysis using frameworks such as TensorFlow and PyTorch, and dynamically adjusts the video's color tone, music, and environmental elements based on the results. This process generates personalized videos that match the user's emotions.

[0772] Specifically, when a user is experiencing stress, the system provides videos featuring relaxing natural scenery and soothing music. Conversely, when a user is feeling happy, the system generates videos with bright colors and cheerful music. The generated video data is then transmitted from the server to the user's device and provided to them.

[0773] This system uses prompt statements like the following as input to the generated AI model.

[0774] "The user's current emotional state is 'sadness,' so please provide relaxing visuals."

[0775] "To put the user in a 'joyful' state, please generate a video clip of a city with bright weather accompanied by energetic music."

[0776] The introduction of this system will enable users to enjoy a video experience tailored to their individual needs in real time, leading to a deeper level of satisfaction.

[0777] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0778] Step 1:

[0779] The device uses a camera and microphone to capture the user's facial expressions and voice tone. The input is raw video and audio data, which is captured in real time. The output is sentiment data in an analyzable format.

[0780] Step 2:

[0781] The device uses OpenCV to analyze facial expressions, converts audio data to text using Google AI's Speech-to-Text API, and then estimates the emotion. The input is the video and audio data obtained in step 1, and the output is data representing the user's current emotional state (e.g., joy, sadness, surprise).

[0782] Step 3:

[0783] The device sends analyzed emotion data to the server. The input is data containing the user's emotional state, and the output is a status indicating that the data transmission to the server was successful.

[0784] Step 4:

[0785] The server uses the received emotion data to activate a generative AI model and generate video content tailored to the user. The input is the user's emotion data, which is used to instruct the generative AI model using prompts. The output is dynamically adjusted video content.

[0786] Step 5:

[0787] The server generates video content and sends it to the terminal. The input is video data generated by AI, and the output is the state of transmission to the user's terminal.

[0788] Step 6:

[0789] The device plays back the video data it receives and provides it to the user. The input is the video data received from the server, and the output is the video the user views. During playback, the device optimizes the user experience by setting the optimal display settings based on environmental factors.

[0790] This processing flow allows users to enjoy personalized videos that match their emotions in real time.

[0791] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0792] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0793] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0794] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0795] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0796] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0797] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0798] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0799] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0800] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0801] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0802] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0803] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0804] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0805] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0806] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0807] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0808] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0809] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0810] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0811] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0812] The following is further disclosed regarding the embodiments described above.

[0813] (Claim 1)

[0814] A means of generating videos based on user preferences,

[0815] A means of acquiring real-time weather data and time data,

[0816] means for updating the aforementioned video based on the aforementioned weather data and the aforementioned time data,

[0817] Means for transmitting the updated video to a display device,

[0818] A system that includes this.

[0819] (Claim 2)

[0820] The system according to claim 1, further comprising means for personalizing video generation based on data obtained from the user.

[0821] (Claim 3)

[0822] The system according to claim 1, wherein the video generation means includes means for dynamically adjusting each element of the video using artificial intelligence.

[0823] "Example 1"

[0824] (Claim 1)

[0825] A means of generating visual data based on user choices,

[0826] Means for obtaining real-time environmental data and time information from external sources,

[0827] Means for dynamically updating the visual data based on the environmental data and the time information,

[0828] Means for transmitting the dynamically updated visual data to a display device,

[0829] Information processing device including

[0830] (Claim 2)

[0831] The information processing apparatus according to claim 1, further comprising means for individually tailoring the generation of visual data based on preference information collected from users.

[0832] (Claim 3)

[0833] The information processing apparatus according to claim 1, wherein the visual data generation means includes means for dynamically adjusting each component of the visual data using machine learning technology.

[0834] "Application Example 1"

[0835] (Claim 1)

[0836] A means of generating information based on user preferences,

[0837] A means for acquiring real-time environmental and time data,

[0838] means for updating the aforementioned information based on the environmental data and the time data,

[0839] Means for transmitting the updated information to a display medium,

[0840] A means of providing a virtual experience based on the user's choice,

[0841] A system that includes this.

[0842] (Claim 2)

[0843] The system according to claim 1, further comprising means for adjusting information generation based on data obtained from users.

[0844] (Claim 3)

[0845] The system according to claim 1, wherein the information generation means includes means for automatically adjusting each element of the information using machine learning technology.

[0846] "Example 2 of combining an emotion engine"

[0847] (Claim 1)

[0848] A means of analyzing user emotions,

[0849] A means of generating videos based on user preferences,

[0850] Means for acquiring real-time environmental data,

[0851] A means for updating the aforementioned video based on the aforementioned environmental data and user emotion data,

[0852] Means for transmitting the updated video to a display device,

[0853] A system that includes this.

[0854] (Claim 2)

[0855] The system according to claim 1, further comprising means for personalizing video generation based on emotional data obtained from users.

[0856] (Claim 3)

[0857] The system according to claim 1, wherein the video generation means includes means for dynamically adjusting each element of the video using generating artificial intelligence.

[0858] "Application example 2 when combining with an emotional engine"

[0859] (Claim 1)

[0860] Means of acquiring user sentiment data,

[0861] A means of generating images using artificial intelligence based on the aforementioned emotional data,

[0862] Means for acquiring real-time environmental data,

[0863] Means for transmitting the generated video to a display device,

[0864] A system that includes this.

[0865] (Claim 2)

[0866] The system according to claim 1, further comprising means for personalizing video generation based on emotional data obtained from a user.

[0867] (Claim 3)

[0868] The system according to claim 1, wherein the video generation means includes means for analyzing the user's emotional state and dynamically adjusting the elements of the video. [Explanation of symbols]

[0869] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of generating information based on user preferences, A means for acquiring real-time environmental and time data, means for updating the aforementioned information based on the environmental data and the time data, Means for transmitting the updated information to a display medium, A means of providing a virtual experience based on the user's choice, A system that includes this.

2. The system according to claim 1, further comprising means for adjusting information generation based on data obtained from users.

3. The system according to claim 1, wherein the information generation means includes means for automatically adjusting each element of the information using machine learning technology.

Images