system

Abstract

Provide a system. A means for acquiring user lifestyle data; A means for analyzing the acquired life data using a generative AI model; A means for generating a profile that reproduces the user's thoughts, speech patterns, and habits based on the results of the analysis; means for creating an avatar of the user based on said profile; The system includes means for performing a dialogue with a user using said avatar.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional systems have had difficulty preserving users' daily data and personal characteristics in digital form for future generations. Furthermore, they lack effective means for recreating communication with people with whom users have special relationships, making it particularly difficult to learn conversations with deceased people or the thought processes of successful people. This has led to a demand for systems that utilize individual characteristics and thought patterns to support richer dialogue and decision-making. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for acquiring a user's lifestyle data, a means for analyzing the acquired lifestyle data using a generative AI model, a means for generating a profile that reproduces the user's thoughts, speech patterns, and habits based on the analysis results, a means for creating an avatar of the user based on the generated profile, and a means for conducting a dialogue with the user using the avatar. This system can include a wearable device for acquiring the user's lifestyle data and physical data, and an interface for the user to manually input data. It can also utilize the user's characteristics to record them for future generations and learn the thought processes of specific individuals.

[0006] "User" refers to any individual or entity that uses the system.

[0007] "Life data" refers to information such as behavior, physical condition, and location information related to the user's daily life.

[0008] A "generative AI model" refers to the artificial intelligence algorithms and structures used to analyze data and perform inference.

[0009] A "profile" refers to a digital representation created to replicate a user's thoughts, speech patterns, habits, and other characteristics.

[0010] An "avatar" is a digital character that reproduces the user's thoughts and habits and is capable of interactive dialogue.

[0011] A "wearable device" is an electronic device that can be worn and used, and is a means for obtaining lifestyle data from the user.

[0012] "Interface" refers to a connection means through a screen or operating device for a user to manually input data. [Brief explanation of the drawings]

[0013] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention relates to a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[0035] Data Acquisition Embodiment

[0036] 1. Acquiring wearable device data

[0037] Device:

[0038] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[0039] 2. Acquiring audio data

[0040] Device:

[0041] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[0042] 3. Acquisition of image data

[0043] Device:

[0044] A camera device (for example, a smartphone camera or a dedicated camera) periodically captures the user's face and facial expressions and sends them to a server as image data.

[0045] 4. Data Linkage

[0046] server:

[0047] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[0048] 5. Manual user data registration

[0049] User:

[0050] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[0051] Data analysis and profile generation

[0052] 1. Data integration and analysis

[0053] server:

[0054] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[0055] Avatar Creation

[0056] 1. Avatar model generation

[0057] server:

[0058] Based on the generated user profile, an avatar is created for the user, which is designed to replicate the user's appearance, voice, and behavior.

[0059] 2. Avatar optimization

[0060] server:

[0061] The avatar's movements and facial expressions are adjusted and optimized based on the user's characteristics.

[0062] User interaction embodiment

[0063] 1. Setting up the dialogue system

[0064] Device:

[0065] A dedicated application is provided and configured to allow users to interact with the avatar.

[0066] 2. Running an interactive session

[0067] User:

[0068] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[0069] 3. Dialogue Generation

[0070] server:

[0071] Using generative AI models, the avatar generates appropriate responses based on user input, allowing the avatar to engage in natural dialogue with the user.

[0072] Data storage and sale practices

[0073] 1. Packaging the data

[0074] server:

[0075] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[0076] 2. Sales and Data Management

[0077] server:

[0078] We will carry out procedures to sell the thought process data of a specific person to interested users. Purchasers will be issued an access key, and the data will be managed appropriately.

[0079] Specific examples

[0080] Acquiring wearable device data

[0081] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[0082] Data analysis and profile generation

[0083] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[0084] Avatar Creation

[0085] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[0086] User interaction

[0087] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[0088] Data storage and sale

[0089] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[0090] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[0091] The processing flow will be explained below.

[0092] Step 1: Get the data

[0093] Device:

[0094] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[0095] 2. A voice recording device records the user's everyday conversations.

[0096] 3. The camera device periodically captures the user's face and facial expression.

[0097] User:

[0098] 1. Manually input data such as schedules and notes through a dedicated application.

[0099] server:

[0100] 1. Receive data sent from the terminal and store it in the database.

[0101] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[0102] Step 2: Data synthesis and analysis

[0103] server:

[0104] 1. Centralize all received data and integrate it into a database.

[0105] 2. The integrated data is input into a generative AI model for analysis and learning.

[0106] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a user profile.

[0107] Step 3: Generate an avatar

[0108] server:

[0109] 1. Create a user avatar based on the generated user profile.

[0110] 2. Set the avatar's appearance, voice, and behavior to replicate the user's characteristics.

[0111] 3. Optimize the avatar's movements and facial expressions and adjust them based on user feedback.

[0112] Step 4: Setting up the dialogue system

[0113] Device:

[0114] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[0115] 2. Set up the application so that you can start interacting with the avatar.

[0116] Step 5: Running an interactive session

[0117] User:

[0118] 1. Open the dedicated application and start interacting with the avatar.

[0119] 2. Ask questions or give instructions to the avatar by voice or text.

[0120] server:

[0121] 1. Takes user input and generates an appropriate response using a generative AI model.

[0122] 2. The avatar engages in natural dialogue with the user.

[0123] Step 6: Store and sell your data

[0124] server:

[0125] 1. Package and save the generated user profile and interaction history.

[0126] 2. Proceed with the sale of a specific person's thought process data.

[0127] 3. Issue an access key to the purchaser and manage the data appropriately.

[0128] Through these steps, this system utilizes the user's daily life data to generate an avatar of the user and enable interaction.The system can also store and sell the data, enabling a wide range of uses.

[0129] Example 1

[0130] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0131] Existing interactive avatar systems lack mechanisms for centrally collecting, integrating, and analyzing user data, making it difficult to accurately reproduce the user's behavior and thoughts. Furthermore, there is a need for a system that can effectively collect, integrate, and analyze users' lifestyle data, allowing for natural interactions with individually optimized avatars and enabling commercial use of the collected data.

[0132] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0133] In this invention, the server includes means for acquiring user life data, means for packaging and selling the generated profile and interaction history, and means for integrating data acquired from external data sources. This not only enables effective collection and analysis of user life data and generation of individually optimized avatars that enable natural interactions, but also makes it possible to commercially utilize the collected data.

[0134] "User lifestyle data" refers to information related to a user's daily life, such as wearable device data, audio data, image data, and social media data.

[0135] A "generative AI model" is an artificial intelligence algorithm used to analyze collected data and learn user behavior patterns and thoughts.

[0136] A "profile" is a data structure that reproduces a user's behavior, thoughts, speech patterns, and habits analyzed by a generative AI model.

[0137] An "avatar" is a digital character created to replicate a user's appearance, voice, and behavior.

[0138] "External data sources" are other systems or databases that provide users' life data, such as social media platforms, electronic communication tools, and online payment systems.

[0139] The "dialogue history" is a record of the dialogue that took place between the user and the avatar.

[0140] "Packaging" is the process of organizing and saving the generated profiles and interaction histories as a single data set.

[0141] A "sales vehicle" is a process or system for providing packaged data to users who desire it.

[0142] This invention is a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. Each component of this system will be explained in detail.

[0143] Data Acquisition Embodiment

[0144] Acquiring wearable device data

[0145] Device: A wearable device (such as a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is transferred to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[0146] Acquiring audio data

[0147] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recorded data is converted into text data using voice recognition software (e.g., Google® Speech-to-Text API). The converted text data is sent to a server.

[0148] Acquisition of image data

[0149] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed by a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[0150] Data linkage implementation example

[0151] Server: Collects and integrates user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs), making it possible to create detailed user profiles that cannot be achieved with traditional data alone.

[0152] Embodiment of Manual User Data Registration

[0153] User: Manually inputs schedules, notes, and other data through a dedicated application. The input data is sent to the server for storage and analysis.

[0154] Data Analysis and Profile Generation Embodiments

[0155] Server: Stores all received data in a central database. It then analyzes the data using a generative AI model (e.g., GPT-4®) to learn the user's behavioral patterns, thinking patterns, speech patterns, habits, etc. Based on the results of this analysis, a detailed profile of the user is generated.

[0156] Avatar Model Generation Embodiment

[0157] Server: Based on the analyzed data, a user avatar is generated, which is designed to replicate the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[0158] Avatar Optimization Embodiment

[0159] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (such as deep learning models).

[0160] User interaction embodiment

[0161] Setting up a dialogue system

[0162] Device: A dedicated application is used to set up the avatar so that the user can interact with it. The application configures the avatar's basic settings and establishes a connection with the user.

[0163] Running an interactive session

[0164] User: Opens a dedicated application and starts interacting with the avatar. The user selects voice or text input methods to send questions or instructions to the avatar.

[0165] Dialogue Generation Embodiments

[0166] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (such as GPT-4), which is then returned to the user via the avatar.

[0167] Data storage and sale practices

[0168] Packaging Data

[0169] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[0170] Sales Method

[0171] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[0172] Specific examples

[0173] Acquiring wearable device data

[0174] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[0175] Data analysis and profile generation

[0176] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[0177] Avatar Creation

[0178] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[0179] User interaction

[0180] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[0181] Data storage and sale

[0182] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[0183] Prompt Sentence Examples

[0184] "Please tell us how to learn behavioral patterns based on daily life data collected by users and enable natural interactions with avatars through a dedicated application."

[0185] As described above, the present invention is a system that effectively collects and analyzes user data, and generates and manages avatars that can interact with users. The collected data can also be used commercially.

[0186] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0187] Step 1:

[0188] Acquiring wearable device data

[0189] Device: A wearable device (e.g., a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is then transmitted to a smartphone via Bluetooth or Wi-Fi. The smartphone then uploads this data to a server at regular intervals.

[0190] Input: User's physical data (heart rate, steps, location)

[0191] Output: Real-time body data uploaded to a server

[0192] Step 2:

[0193] Acquiring audio data

[0194] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recordings are stored locally and then converted into text using speech recognition software (e.g., Google Speech-to-Text API). The text is then sent over the internet to a server.

[0195] Input: Recorded audio data

[0196] Output: Text data sent to the server

[0197] Step 3:

[0198] Acquisition of image data

[0199] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed using a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[0200] Input: Captured image data

[0201] Output: Processed image data sent to the server

[0202] Step 4:

[0203] Data Linkage

[0204] Server: Obtains user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs). This data is integrated with existing data and stored in a central database.

[0205] Input: Data retrieved from an external data source

[0206] Output: User life data integrated into a central database

[0207] Step 5:

[0208] Manual user data registration

[0209] User: Manually inputs schedules, memos, and other data using a dedicated application. This data is sent to the server via the application, where it is stored and analyzed.

[0210] Input: Data manually entered by the user (schedule, notes, etc.)

[0211] Output: Manually entered data sent to the server

[0212] Step 6:

[0213] Data integration and analysis

[0214] Server: The server stores all received data in a central database and analyzes it using a generative AI model (e.g., GPT-4), which learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a detailed user profile.

[0215] Input: All data stored in a central database (physical data, audio data, image data, external data, manually entered data)

[0216] Output: A user profile generated based on the analysis.

[0217] Step 7:

[0218] Avatar model generation

[0219] Server: Based on the generated user profile, a user avatar is generated, which reproduces the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[0220] Input: Generated user profile

[0221] Output: The generated avatar

[0222] Step 8:

[0223] Avatar Optimization

[0224] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (e.g., deep learning models).

[0225] Input: Generated avatar, user feature data

[0226] Output: Optimized avatar

[0227] Step 9:

[0228] Setting up a dialogue system

[0229] Device: Allows users to interact with their avatar through a dedicated application, which configures the avatar's basic settings and establishes a connection with the user.

[0230] Input: Avatar, user preference data

[0231] Output: The dialogue system is set up

[0232] Step 10:

[0233] Running an interactive session

[0234] User: Opens a dedicated application and starts interacting with the avatar. The user sends questions or instructions to the avatar via voice or text input.

[0235] Input: User questions or instructions (voice or text)

[0236] Output: Interactive session running

[0237] Step 11:

[0238] Dialogue generation

[0239] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (e.g., GPT-4), which is then returned to the user via the avatar.

[0240] Input: User input data, generative AI model

[0241] Output: The response generated by the avatar

[0242] Step 12:

[0243] Packaging Data

[0244] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[0245] Input: Generated user profile, interaction history

[0246] Output: Packaged dataset

[0247] Step 13:

[0248] Sales and Data Management

[0249] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[0250] Input: Packaged dataset, buyer information

[0251] Output: Issued access key, managed data

[0252] (Application example 1)

[0253] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] Conventional personalized shopping assistant systems did not fully utilize users' lifestyle data, resulting in insufficient suggestions based on the user's preferences and behavioral patterns. Furthermore, the dialogue with the user was often unnatural, resulting in a poor user experience. Furthermore, there was no way to record the user's responses to suggested items and reflect them in the next purchase, which prevented the accuracy of suggestions from improving. This made it difficult to provide a shopping experience tailored to each individual user.

[0255] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0256] In this invention, the server includes means for acquiring user lifestyle data, means for analyzing the acquired lifestyle data using a generative AI model, means for generating a profile based on the analysis that reproduces the user's thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile, means for engaging in dialogue with the user using the avatar, means for suggesting items suitable for the user based on the user's purchasing data and preference data, means for recording the user's responses to the suggested items and reflecting them in future suggestions, means for acquiring the user's physiological data using a wearable device, and means for providing an interface for the user to manually input data. This enables more accurate and personalized shopping suggestions based on the user's lifestyle data and preferences, and a more satisfying shopping experience through natural dialogue with the user.

[0257] "Lifestyle data" refers to various information related to the user's daily life, including physical data from wearable devices, location information, audio data, image data, and activity data on social media.

[0258] A "generative AI model" is an artificial intelligence model that analyzes acquired data and learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[0259] A "profile" is a collection of information that reproduces a user's thoughts, speaking style, habits, behavioral patterns, etc., generated based on the analysis results of a generative AI model.

[0260] An "avatar" is a character that reproduces the user's appearance, voice, actions, facial expressions, etc. based on the user's profile, and is a virtual entity that can interact with the user.

[0261] "Dialogue" refers to communication between a user and an avatar via voice or text.

[0262] "Purchase data" is information about products that a user has purchased in the past, including the purchase date and time, product name, category, price, and the like.

[0263] "Preference data" is information about products and services that a user likes, and is data that indicates the user's preferences based on past behavioral history, purchase history, survey results, etc.

[0264] "User response" refers to the reaction or feedback given by the user regarding the item suggested by the avatar.

[0265] This invention relates to a system that acquires and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. An embodiment of this system will be specifically explained below, focusing on how the server, terminal, and user cooperate to function.

[0266] Data Acquisition Embodiment

[0267] 1. Acquiring wearable device data

[0268] Device: A wearable device (e.g., a smartwatch or fitness tracker) collects a user's physical data (e.g., heart rate, steps, location, etc.) in real time. This data is periodically transmitted to a server via the Internet.

[0269] 2. Acquiring audio data

[0270] Terminal: A voice recording device (e.g., a smartphone or dedicated recording device) is used to record the user's everyday conversation. This voice data is converted into text using a voice recognition system (e.g., Google Speech-to-Text API) and sent to the server.

[0271] 3. Acquisition of image data

[0272] Terminal: A camera device (for example, a smartphone camera or a dedicated camera) is used to periodically capture the user's face and facial expressions, and the captured images are sent to the server as image data.

[0273] 4. Data Linkage

[0274] Server: Acquires data from multiple external data sources (e.g., social media platforms, online payment systems, etc.) and complements and integrates user lifestyle data.

[0275] 5. Manual user data registration

[0276] User: Manually inputs data such as schedules and notes through a dedicated application and sends it to the server.

[0277] Data analysis and profile generation

[0278] 1. Data integration and analysis

[0279] Server: Stores all received data in a database and feeds it into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[0280] Avatar Creation

[0281] 1. Avatar model generation

[0282] Server: Creates a user avatar based on the generated user profile. The avatar is designed to replicate the user's appearance, voice, and behavior.

[0283] 2. Avatar optimization

[0284] Server: Adjusts and optimizes the avatar's movements and facial expressions based on the user's characteristics.

[0285] User interaction embodiment

[0286] 1. Setting up the dialogue system

[0287] Terminal: Provides dedicated applications and is configured to allow users to interact with avatars.

[0288] 2. Running an interactive session

[0289] User: Opens the dedicated application and starts interacting with the avatar. The user inputs questions and instructions to the avatar via voice or text.

[0290] 3. Dialogue Generation

[0291] Server: Using the generative AI model based on the user's input, the avatar generates an appropriate response. The avatar then engages in natural dialogue with the user.

[0292] 4. Recommendations based on purchasing and preference data

[0293] Server: Suggests suitable products and services to users based on their purchase history and preference data.

[0294] 5. Record and reflect responses to proposed items

[0295] Server: Records the user's responses to the suggested items and uses that data to improve the next suggestion.

[0296] Specific examples

[0297] If a user asks the shopping assistant, "What cosmetics should I buy next?", the avatar will suggest, based on analyzed lifestyle data and past purchasing history, "You've been buying a lot of skin care products for sensitive skin lately, so how about this new sensitive skin cream?"

[0298] Example prompts to input to the generative AI model

[0299] "Suggest the next product to buy based on the user's past shopping history and preferences. The user prefers products for sensitive skin.

[0300] Input:

[0301] Past purchases: Skin care, sensitive skin cream, facial cleanser

[0302] User's recent purchase history: Sensitive skin cream, sensitive skin lotion

[0303] User Input:

[0304] What would you recommend for my next shopping trip?”

[0305] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[0306] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0307] Step 1:

[0308] Data collection (device)

[0309] Physical data such as heart rate, number of steps, and location information are collected from the user's wearable device. In addition, a voice recording device records the user's daily conversations, and a camera device collects image data of the user's face and facial expressions. The collected data is periodically sent to a server via the Internet.

[0310] Input: Data from wearable devices, audio recording devices, and camera devices

[0311] Output: User's physical data, voice data, image data sent to the server

[0312] Step 2:

[0313] Data analysis and integration (server)

[0314] All received data is stored in a database and analyzed using a generative AI model (such as GPT-4 or BERT). Voice data is converted into text using a speech recognition system (Google Speech-to-Text API, for example). Through analysis, the system learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[0315] Input: User's physical data, voice data, image data sent to the server

[0316] Output: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[0317] Step 3:

[0318] Profile Generation (Server)

[0319] Based on the analysis results of the generative AI model, a user profile is generated, which recreates and stores the user's thoughts, speech patterns, habits, and behavioral patterns.

[0320] Input: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[0321] Output: User profile

[0322] Step 4:

[0323] Creating an avatar (server)

[0324] Based on the generated profile, an avatar is created for the user. The avatar is designed to replicate the user's appearance, voice, and behavior. The avatar's movements, facial expressions, and other aspects are then optimized based on the user's characteristics.

[0325] Input: User profile

[0326] Output: User avatar

[0327] Step 5:

[0328] Providing interactive applications (terminals)

[0329] It provides an environment where users can interact with avatars through a dedicated application on their device. Users open the application and begin interacting with the avatar. Users' questions and instructions are entered by voice or text.

[0330] Input: User-entered voice or text data

[0331] Output: Display of interactive screen

[0332] Step 6:

[0333] Conversation creation (server)

[0334] Based on user input, a generative AI model is used to generate appropriate responses, which are then served to the user through an avatar.

[0335] Input: User's voice or text input data

[0336] Output: Natural dialogue responses from the avatar

[0337] Step 7:

[0338] Personalized Item Suggestions (Server)

[0339] Based on the user's purchasing data and preference data, the generative AI model proposes suitable products and services to the user. It analyzes past purchase history and preference data to generate the most appropriate proposals.

[0340] Input: Purchasing data, preference data

[0341] Output: Product and service recommendations to the user

[0342] Step 8:

[0343] Reflecting feedback on proposals (server)

[0344] The system records the user's responses to suggested items and updates the database to reflect them in the next suggestion, improving the accuracy of the suggestions.

[0345] Input: User response data

[0346] Output: Updated purchase data, preference data, and recommendation data

[0347] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0348] This invention relates to a system that collects and analyzes a user's lifestyle data and emotional data, generates an avatar of the user based on the data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[0349] Data Acquisition Embodiment

[0350] 1. Acquiring wearable device data

[0351] Device:

[0352] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[0353] 2. Acquiring audio data

[0354] Device:

[0355] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[0356] 3. Acquisition of image data

[0357] Device:

[0358] A camera device (for example, a smartphone camera or a dedicated camera) periodically takes a photo of the user's face and facial expression and sends it to a server as image data.

[0359] 4. Data Linkage

[0360] server:

[0361] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[0362] 5. Manual user data registration

[0363] User:

[0364] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[0365] Acquiring and analyzing emotion data

[0366] 1. Operating the Emotion Engine

[0367] server:

[0368] The emotion engine installed on the server analyzes the user's voice tone, facial expressions, and body language, and through this analysis, recognizes the user's emotional state (joy, anger, sadness, happiness, etc.).

[0369] Device:

[0370] The emotion data acquired by the emotion engine is integrated with lifestyle data.

[0371] Data analysis and profile generation

[0372] 1. Data integration and analysis

[0373] server:

[0374] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[0375] Avatar Creation

[0376] 1. Avatar model generation

[0377] server:

[0378] Based on the generated user profile and emotional data, a user avatar is created, which is designed to replicate the user's appearance, voice, and behavior.

[0379] 2. Avatar optimization

[0380] server:

[0381] Based on emotional data, the avatar's interaction patterns are adjusted and optimized, and improved based on user feedback.

[0382] User interaction embodiment

[0383] 1. Setting up the dialogue system

[0384] Device:

[0385] A dedicated application will be provided to create an environment in which users can interact with avatars.

[0386] 2. Running an interactive session

[0387] User:

[0388] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[0389] 3. Dialogue Generation

[0390] server:

[0391] Based on user input, the generative AI model generates appropriate responses, taking into account emotional data, allowing the avatar to engage in natural conversation with the user.

[0392] Data storage and sale practices

[0393] 1. Packaging the data

[0394] server:

[0395] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[0396] 2. Sales and Data Management

[0397] server:

[0398] The thought process data and emotion data of a specific person will be sold to interested users. An access key will be issued to the purchaser, and the data will be managed appropriately.

[0399] Specific examples

[0400] Acquiring wearable device data

[0401] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[0402] Acquiring emotion data

[0403] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[0404] Data analysis and profile generation

[0405] Server: The server integrates and analyzes the jogging data and emotional data, and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[0406] Avatar Creation

[0407] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[0408] User interaction

[0409] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[0410] Data storage and sale

[0411] Server: Packages the thought processes and emotional data of famous authors and executes the sales process for potential buyers.

[0412] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[0413] The processing flow will be explained below.

[0414] Step 1: Get the data

[0415] Device:

[0416] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[0417] 2. A voice recording device records the user's everyday conversations.

[0418] 3. The camera device periodically captures the user's face and facial expression.

[0419] User:

[0420] 1. Manually input data such as schedules and notes through a dedicated application.

[0421] server:

[0422] 1. Receive data sent from the terminal and store it in the database.

[0423] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[0424] Step 2: Obtaining emotion data

[0425] server:

[0426] 1. Use an emotion engine to analyze the user's tone of voice, facial expressions, and body language.

[0427] 2. Based on the analysis results, the user's emotional data (e.g., joy, anger, sadness, happiness, etc.) is recognized.

[0428] Device:

[0429] 1. Send emotion data to the server.

[0430] Step 3: Data synthesis and analysis

[0431] server:

[0432] 1. Consolidate all received data and store it in a database.

[0433] 2. The integrated data is input into a generative AI model for analysis and learning.

[0434] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, habits, and emotional data to generate a user profile.

[0435] Step 4: Avatar generation and optimization

[0436] server:

[0437] 1. Create a user avatar based on the generated user profile.

[0438] 2. Set the avatar's appearance, voice, and behavior, and prepare dialogue patterns that reflect the user's characteristics and emotional data.

[0439] 3. Adjust the avatar's movements and facial expressions based on emotional data and improve them based on user feedback.

[0440] Step 5: Setting up the dialogue system

[0441] Device:

[0442] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[0443] 2. Set up the application so that you can start interacting with the avatar.

[0444] Step 6: Running an interactive session

[0445] User:

[0446] 1. Open the dedicated application and start interacting with the avatar.

[0447] 2. Ask questions or give instructions to the avatar by voice or text.

[0448] server:

[0449] 1. Takes user input and generates an appropriate response using a generative AI model.

[0450] 2. Emotional data is also taken into account when generating responses, allowing the avatar to engage in natural dialogue with the user.

[0451] Step 7: Store and sell your data

[0452] server:

[0453] 1. Package and save the generated user profile and interaction history.

[0454] 2. Carry out procedures to sell a specific person's thought process data and emotional data.

[0455] 3. Issue an access key to the purchaser and manage the data appropriately.

[0456] Through these steps, a system is constructed that uses the user's lifestyle and emotional data to generate an avatar of the user and enable interaction. This system consistently supports the entire process, from data collection and analysis, to avatar generation, interaction, data storage, and sales.

[0457] Example 2

[0458] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0459] With conventional technology, it was difficult to effectively collect and analyze a user's lifestyle and emotional data and, based on that data, generate an avatar that could converse naturally with the user. It was also difficult to properly store user profiles and conversation histories and package them for commercial use. This resulted in the challenge of being unable to provide services that adequately meet the individual needs of users.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0461] In this invention, the server includes means for acquiring user lifestyle data and emotional data, means for analyzing the data using a generative AI model, means for generating a profile that reproduces the user's behavioral patterns, thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile and emotional data, means for conducting a dialogue with the user using the avatar, and means for packaging and storing the user profile and dialogue history. This enables the generation of an avatar that can conduct natural dialogue according to the individual needs of the user and also facilitates commercial use of the collected data.

[0462] "Life data" refers to information related to the user's daily life, and specifically includes physical data, movement history, activity history, schedule, and the like.

[0463] "Emotional data" refers to information that indicates the user's emotional state, and includes data obtained from vocal tone, facial expressions, body language, and the like.

[0464] A "generative AI model" is an artificial intelligence model that analyzes large amounts of data and learns patterns, and has the ability to generate data generatively.

[0465] A "profile" is a collection of digital data that reproduces a user's behavioral patterns, thoughts, speech patterns, habits, etc.

[0466] An "avatar" is a digital character that mimics a user's appearance, voice, and behavior and is used to interact with the user.

[0467] "Dialogue history" refers to a record of the dialogue between the user and the avatar, and is used for later analysis and improvement.

[0468] "Packaging" refers to organizing data so that it is easy to use, and storing and managing it as a single entity.

[0469] A "server" is a machine that processes and stores data on a computer network and is the core of the system.

[0470] A "terminal" is a device that is directly operated by a user, such as a smartphone or a wearable device.

[0471] An "interface" is a means by which a user inputs data into a system and checks the output results.

[0472] This system collects and analyzes users' lifestyle and emotional data, and generates an avatar that can interact with the user based on that data. This system is primarily composed of three components: a server, a device, and the user.

[0473] Data Acquisition Embodiment

[0474] Acquiring wearable device data

[0475] Device: The user wears a wearable device such as a smartwatch or fitness tracker. This device collects real-time physical data such as heart rate, steps taken, and location. The collected data is sent to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[0476] Acquiring audio data

[0477] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[0478] Acquisition of image data

[0479] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[0480] Data Linkage

[0481] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[0482] Manual user data registration

[0483] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[0484] Acquiring and analyzing emotion data

[0485] Operating the Emotion Engine

[0486] Server: The emotion engine (e.g., Microsoft® Azure® Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to estimate emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis. These data are integrated with life data.

[0487] Data analysis and profile generation

[0488] Data Integration

[0489] Server: All collected data (physical data, audio data, image data, external data, manual data) will be stored in a central database. A database management system (e.g., PostgreSQL) will be used.

[0490] Profile Generation

[0491] Server: A generative AI model (e.g., GPT-3®) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a detailed profile of the user.

[0492] Avatar Creation

[0493] Avatar model generation

[0494] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[0495] Avatar Optimization

[0496] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[0497] User interaction

[0498] Setting up a dialogue system

[0499] Device: A dedicated application is installed on the user's smartphone, providing an interface for interacting with the avatar.

[0500] Running an interactive session

[0501] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[0502] Dialogue generation

[0503] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[0504] Data storage and sale

[0505] Packaging Data

[0506] Server: Packages and stores the generated user profiles and interaction histories. This data is organized for commercial use.

[0507] Sales and Data Management

[0508] Server: The server processes the sale of a specific person's thought process and emotional data to interested users. It issues an access key to the purchaser and ensures proper management and security of the data.

[0509] Specific examples

[0510] Acquiring wearable device data

[0511] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[0512] Acquiring emotion data

[0513] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[0514] Data analysis and profile generation

[0515] Server: Integrates and analyzes jogging data and emotional data to learn a behavioral pattern, such as "the user jogs every morning at 7am."

[0516] Avatar Creation

[0517] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[0518] User interaction

[0519] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, the cities you visited last year were good."

[0520] Data storage and sale

[0521] Server: Packages the thought processes and emotional data of famous people and executes the sales process for potential buyers.

[0522] Prompt Sentence Examples

[0523] Please explain the specific processing steps of a system that collects a user's lifestyle and emotional data, generates an avatar based on that data, and allows the avatar to interact with the user.

[0524] In this way, the present invention provides a system that uses user data to generate and manage avatars that can naturally converse with users, and further utilizes that data commercially.

[0525] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0526] Step 1:

[0527] Data collection

[0528] Device: The user wears a wearable device (e.g., a smartwatch). The device collects real-time physical data such as heart rate, steps, and location information, and transmits it to a smartphone via Bluetooth or Wi-Fi. The smartphone periodically uploads this data to a server via the Internet.

[0529] Input: Sensor data from wearable devices (heart rate, steps, location information)

[0530] Output: Body data sent to the server

[0531] How it works: The smartwatch's sensors detect body data, and the smartphone receives the data and transfers it to a server.

[0532] Step 2:

[0533] Acquiring audio data

[0534] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[0535] Input: Audio data recorded by a smartphone microphone

[0536] Output: The audio data converted to text is sent to the server.

[0537] How it works: Your smartphone records your everyday conversations, and a voice recognition system converts the audio into text and uploads it to a server.

[0538] Step 3:

[0539] Acquisition of image data

[0540] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[0541] Input: Facial image taken with a smartphone camera and facial expression data

[0542] Output: Compressed image data is sent to the server

[0543] How it works: The smartphone camera takes a photo of your face, compresses the image data, and uploads it to a server.

[0544] Step 4:

[0545] Data Linkage

[0546] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[0547] Input: Additional data retrieved from an external data source

[0548] Output: Integrated user life data

[0549] How it works: The server accesses external data sources via APIs to retrieve additional data and integrate it with the user's life data.

[0550] Step 5:

[0551] Manual user data registration

[0552] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[0553] Input: Manual data entered by the user through a dedicated application

[0554] Output: Manual data stored on the server

[0555] How it works: The user enters schedules and notes into a dedicated application, which are then sent to the server in real time.

[0556] Step 6:

[0557] Acquiring and analyzing emotion data

[0558] Server: The emotion engine (e.g., Microsoft Azure Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to infer emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis.

[0559] Input: Voice tone, facial expression data, body language

[0560] Output: Parsed emotion data

[0561] How it works: The server analyzes the audio and image data, and the emotion engine recognizes the user's emotional state.

[0562] Step 7:

[0563] Data integration and analysis

[0564] Server: All collected data (physical data, audio data, image data, external data, manual data) is stored in a central database. A generative AI model (e.g., GPT-3) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits.

[0565] Input: Integrated data stored in a central database

[0566] Output: Parsed user profile

[0567] How it works: A server aggregates various data, and a generative AI model analyzes it to create a profile of the user.

[0568] Step 8:

[0569] Avatar model generation

[0570] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[0571] Input: User profile and emotion data

[0572] Output: Generated 3D avatar

[0573] How it works: 3D model generation software creates an avatar that mimics the user's appearance and behavior, and a voice synthesis system generates the avatar's voice.

[0574] Step 9:

[0575] Avatar Optimization

[0576] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[0577] Input: User feedback and sentiment data

[0578] Output: Optimized avatar

[0579] How it works: The server collects user feedback and uses a generative AI model to optimize the avatar's interaction patterns.

[0580] Step 10:

[0581] Setting up a dialogue system

[0582] Device: A dedicated application is installed on the user's smartphone, which provides an interface for interacting with the avatar.

[0583] Input: Dedicated application

[0584] Output: Avatar interaction interface

[0585] How it works: A dedicated application is installed on a smartphone and provides an interface for interacting with the avatar.

[0586] Step 11:

[0587] Running an interactive session

[0588] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[0589] Input: User questions and instructions

[0590] Output: Avatar response

[0591] How it works: The user opens a dedicated application and interacts with an avatar based on input questions and instructions.

[0592] Step 12:

[0593] Dialogue generation

[0594] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[0595] Input: User input and emotion data

[0596] Output: The generated response

[0597] How it works: The server uses a generative AI model to generate a response, which the avatar relays to the user.

[0598] Step 13:

[0599] Data storage and sale

[0600] Server: Packages and stores the generated user profiles and interaction histories. This data is organized for commercial use.

[0601] Input: User profile and interaction history

[0602] Output: Packaged data

[0603] How it works: The server packages and stores user profiles and interaction histories.

[0604] Sales and Data Management

[0605] Server: The server processes the sale of a specific person's thought process and emotional data to interested users. It issues an access key to the purchaser and ensures proper management and security of the data.

[0606] Input: Sales Request

[0607] Output: Access key

[0608] How it works: The server accepts sales requests and sells and manages data.

[0609] (Application example 2)

[0610] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0611] Conventional avatar generation systems are primarily limited to online use and have not contributed sufficiently to improving customer service in brick-and-mortar stores. In particular, it has been difficult to analyze individual users' lifestyle and emotional data in real time and provide personalized recommendations and support based on that data.

[0612] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0613] In this invention, the server includes means for acquiring user lifestyle data, means for analyzing the data using a generative AI model, means for generating a profile that reproduces the user's thoughts, speech patterns, and habits, means for creating an avatar for the user, means for conducting dialogue with the user, means for collecting the user's shopping history and emotion data, means for recommending optimal products and services in physical stores, and means for providing a dialogue function with an avatar that provides real-time support when the user experiences difficulty in the store. This makes it possible to provide personalized services tailored to the individual needs of users even in physical stores.

[0614] "User's lifestyle data" is information relating to the user's daily life, including, for example, location information, walking data, heart rate, shopping history, and the like.

[0615] A "generative AI model" is an artificial intelligence algorithm used to analyze large amounts of data and generate new information and predictions.

[0616] A "profile" is a collection of information created to reproduce a user's characteristics, such as their way of thinking, speaking, and habits.

[0617] An "avatar" is a virtual persona created to replicate a user's appearance, voice, and behavior.

[0618] "Emotion data" is data that indicates the user's emotional state, and is obtained from voice tone, facial expressions, body movements, and the like.

[0619] "Shopping history" refers to data on products purchased by a user in the past and related information.

[0620] A "physical store" is a sales or commercial facility that a user can physically visit.

[0621] "Recommendation" is the action or function of suggesting optimal products or services based on a user's past data and current needs.

[0622] "Providing support in real time" means providing the information and help a user needs at the exact moment they are using the application.

[0623] The "interactive function" is a function that allows users to have natural conversations with their avatars through voice and text.

[0624] This invention relates to a system that collects and analyzes a user's lifestyle data and emotional data, generates an avatar of the user based on the collected data, and enables interaction with the user. Furthermore, this system can be applied to providing personalized customer service in brick-and-mortar stores.

[0625] 1. Data Acquisition Method

[0626] Acquiring wearable device data

[0627] Device: A wearable device (such as a smartwatch or fitness tracker) collects the user's physical data (heart rate, steps, location, etc.) in real time. This data is periodically sent to a server via the internet.

[0628] Acquiring audio data

[0629] Device: A voice recording device (smartphone or dedicated recording device) records the user's everyday conversation. This voice data is converted into text by a voice recognition system and sent to a server.

[0630] Acquisition of image data

[0631] Device: A camera device (such as a smartphone camera or a dedicated camera) periodically takes a photo of the user's face and facial expression, and sends the image data to the server.

[0632] Data Linkage

[0633] Server: Acquires data from multiple external data sources (social media platforms, electronic communication tools, online payment systems) and complements and integrates user life data.

[0634] Manual user data registration

[0635] User: Through a dedicated application, users can manually input data such as schedules and notes and send it to the server.

[0636] 2. Acquisition and analysis of emotion data

[0637] Operating the Emotion Engine

[0638] Server: The emotion engine installed on the server analyzes the user's tone of voice and facial expressions, and recognizes the user's emotional state through this analysis.

[0639] Terminal: Emotion data acquired by the emotion engine is integrated with lifestyle data.

[0640] 3. Data analysis and profile generation

[0641] Data integration and analysis

[0642] Server: Stores all received data in a database and inputs it into a generative AI model. The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[0643] 4. Create your avatar

[0644] Avatar model generation

[0645] Server: Creates a user avatar based on the generated user profile and emotional data. The avatar is designed to replicate the user's appearance, voice, and behavior.

[0646] 5. User interaction implementation

[0647] Setting up a dialogue system

[0648] Terminal: Provides dedicated applications and creates an environment in which users can interact with avatars.

[0649] Running an interactive session

[0650] User: Opens a dedicated application and starts interacting with the avatar. The user can enter questions or instructions to the avatar via voice or text.

[0651] Dialogue generation

[0652] Server: Based on user input, the generative AI model generates an appropriate response, taking into account emotional data. The avatar then engages in natural dialogue with the user.

[0653] 6. Physical store applications

[0654] Collecting shopping history and sentiment data

[0655] Terminal: Collects user shopping history and emotion data through the application and sends it to the server.

[0656] Recommendation function in physical stores

[0657] Server: Analyzes data using generative AI models to recommend optimal products and services in physical stores.

[0658] Real-time support

[0659] Terminal: Provides interaction with an avatar that provides real-time support to users experiencing difficulties in the store.

[0660] Specific examples

[0661] Acquiring jogging data: When a user goes for a jog every morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[0662] Example of avatar dialogue: At 7am, the avatar asks the user, "Are you going for a jog again today? It looks like a relaxing time."

[0663] Example of interaction with an avatar in a physical store: A user arrives at a physical store and says to the avatar, "I'm looking for a recommended winter coat." The avatar recommends the best coat based on the user's past purchase history and current emotional state.

[0664] Example prompt for a generative AI model:

[0665] "When a user is looking for something in a physical store, create a conversation that recommends the best product based on their past purchase data and current heart rate data."

[0666] This allows us to make the most of user data and create a personalized shopping experience in physical stores.

[0667] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0668] Step 1:

[0669] Data collection

[0670] Device: Collects real-time data such as heart rate, steps, and location information from the user's wearable device (smartwatch or fitness tracker), as well as audio and image data from smartphones and cameras.

[0671] Input: Heart rate, steps, location, audio data, image data.

[0672] Output: The collected life data and emotion data are sent to the server.

[0673] Step 2:

[0674] Data transmission

[0675] Terminal: Sends collected lifestyle and emotional data to a server via the Internet.

[0676] Input: Life and emotion data collected on the device.

[0677] Output: Life data and emotion data sent to the server.

[0678] Step 3:

[0679] Data analysis

[0680] Server: The received lifestyle and emotion data is stored in a cloud-based data analysis system and input into the generative AI model.

[0681] Input: Submitted lifestyle and emotion data.

[0682] Output: A profile of the user's behavioral patterns and emotional state based on the analyzed data.

[0683] Step 4:

[0684] Profile Generation

[0685] Server: The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[0686] Input: Data analysis results.

[0687] Output: A profile reflecting the user's behavioral patterns and emotional state.

[0688] Step 5:

[0689] Avatar Creation

[0690] Server: Creates an avatar for the user based on the generated profile. The avatar is designed to replicate the user's appearance, voice, and behavior.

[0691] Input: The user's profile.

[0692] Output: An avatar that replicates the user's features.

[0693] Step 6:

[0694] Avatar interaction system setup

[0695] Terminal: Creates an environment where users can interact with avatars through a dedicated application.

[0696] Input: The avatar installed in the application.

[0697] Output: An interactive application.

[0698] Step 7:

[0699] Start of user interaction

[0700] User: Opens a dedicated application and starts interacting with the avatar. The user can enter questions or instructions to the avatar via voice or text.

[0701] Input: A user's voice or text question or instruction.

[0702] Output: Input data for the avatar.

[0703] Step 8:

[0704] Response Generation

[0705] Server: Uses a generative AI model based on user input and takes emotional data into account to generate an appropriate response.

[0706] Input: User questions or instructions.

[0707] Output: The response that the avatar provides to the user.

[0708] Step 9:

[0709] In-store recommendations

[0710] Server: Analyzes the user's shopping history and emotional data to recommend optimal products and services in physical stores.

[0711] Input: Shopping history and sentiment data.

[0712] Output: Recommended products or services.

[0713] Step 10:

[0714] Real-time support

[0715] Terminal: Provides an interactive function with an avatar that provides real-time support when users encounter problems in a physical store.

[0716] Input: Real-time user questions and concerns.

[0717] Output: Real-time support and guidance.

[0718] This is the flow of processing for this program. This makes it possible to utilize individual user data and provide personalized services even in physical stores.

[0719] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0720] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0721] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0722] [Second embodiment]

[0723] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0724] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0725] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0726] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0727] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0728] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0729] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0730] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0731] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0732] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0733] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0734] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0735] This invention relates to a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[0736] Data Acquisition Embodiment

[0737] 1. Acquiring wearable device data

[0738] Device:

[0739] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[0740] 2. Acquiring audio data

[0741] Device:

[0742] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[0743] 3. Acquisition of image data

[0744] Device:

[0745] A camera device (for example, a smartphone camera or a dedicated camera) periodically captures the user's face and facial expressions and sends them to a server as image data.

[0746] 4. Data Linkage

[0747] server:

[0748] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[0749] 5. Manual user data registration

[0750] User:

[0751] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[0752] Data analysis and profile generation

[0753] 1. Data integration and analysis

[0754] server:

[0755] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[0756] Avatar Creation

[0757] 1. Avatar model generation

[0758] server:

[0759] Based on the generated user profile, an avatar is created for the user, which is designed to replicate the user's appearance, voice, and behavior.

[0760] 2. Avatar optimization

[0761] server:

[0762] The avatar's movements and facial expressions are adjusted and optimized based on the user's characteristics.

[0763] User interaction embodiment

[0764] 1. Setting up the dialogue system

[0765] Device:

[0766] A dedicated application is provided and configured to allow users to interact with the avatar.

[0767] 2. Running an interactive session

[0768] User:

[0769] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[0770] 3. Dialogue Generation

[0771] server:

[0772] Using generative AI models, the avatar generates appropriate responses based on user input, allowing the avatar to engage in natural dialogue with the user.

[0773] Data storage and sale practices

[0774] 1. Packaging the data

[0775] server:

[0776] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[0777] 2. Sales and Data Management

[0778] server:

[0779] We will carry out procedures to sell the thought process data of a specific person to interested users. Purchasers will be issued an access key, and the data will be managed appropriately.

[0780] Specific examples

[0781] Acquiring wearable device data

[0782] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[0783] Data analysis and profile generation

[0784] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[0785] Avatar Creation

[0786] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[0787] User interaction

[0788] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[0789] Data storage and sale

[0790] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[0791] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[0792] The processing flow will be explained below.

[0793] Step 1: Get the data

[0794] Device:

[0795] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[0796] 2. A voice recording device records the user's everyday conversations.

[0797] 3. The camera device periodically captures the user's face and facial expression.

[0798] User:

[0799] 1. Manually input data such as schedules and notes through a dedicated application.

[0800] server:

[0801] 1. Receive data sent from the terminal and store it in the database.

[0802] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[0803] Step 2: Data synthesis and analysis

[0804] server:

[0805] 1. Centralize all received data and integrate it into a database.

[0806] 2. The integrated data is input into a generative AI model for analysis and learning.

[0807] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a user profile.

[0808] Step 3: Generate an avatar

[0809] server:

[0810] 1. Create a user avatar based on the generated user profile.

[0811] 2. Set the avatar's appearance, voice, and behavior to replicate the user's characteristics.

[0812] 3. Optimize the avatar's movements and facial expressions and adjust them based on user feedback.

[0813] Step 4: Setting up the dialogue system

[0814] Device:

[0815] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[0816] 2. Set up the application so that you can start interacting with the avatar.

[0817] Step 5: Running an interactive session

[0818] User:

[0819] 1. Open the dedicated application and start interacting with the avatar.

[0820] 2. Ask questions or give instructions to the avatar by voice or text.

[0821] server:

[0822] 1. Takes user input and generates an appropriate response using a generative AI model.

[0823] 2. The avatar engages in natural dialogue with the user.

[0824] Step 6: Store and sell your data

[0825] server:

[0826] 1. Package and save the generated user profile and interaction history.

[0827] 2. Proceed with the sale of a specific person's thought process data.

[0828] 3. Issue an access key to the purchaser and manage the data appropriately.

[0829] Through these steps, this system utilizes the user's daily life data to generate an avatar of the user and enable interaction.The system can also store and sell the data, enabling a wide range of uses.

[0830] Example 1

[0831] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0832] Existing interactive avatar systems lack mechanisms for centrally collecting, integrating, and analyzing user data, making it difficult to accurately reproduce the user's behavior and thoughts. Furthermore, there is a need for a system that can effectively collect, integrate, and analyze users' lifestyle data, allowing for natural interactions with individually optimized avatars and enabling commercial use of the collected data.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0834] In this invention, the server includes means for acquiring user life data, means for packaging and selling the generated profile and interaction history, and means for integrating data acquired from external data sources. This not only enables effective collection and analysis of user life data and generation of individually optimized avatars that enable natural interactions, but also makes it possible to commercially utilize the collected data.

[0835] "User lifestyle data" refers to information related to a user's daily life, such as wearable device data, audio data, image data, and social media data.

[0836] A "generative AI model" is an artificial intelligence algorithm used to analyze collected data and learn user behavior patterns and thoughts.

[0837] A "profile" is a data structure that reproduces a user's behavior, thoughts, speech patterns, and habits analyzed by a generative AI model.

[0838] An "avatar" is a digital character created to replicate a user's appearance, voice, and behavior.

[0839] "External data sources" are other systems or databases that provide users' life data, such as social media platforms, electronic communication tools, and online payment systems.

[0840] The "dialogue history" is a record of the dialogue that took place between the user and the avatar.

[0841] "Packaging" is the process of organizing and saving the generated profiles and interaction histories as a single data set.

[0842] A "sales vehicle" is a process or system for providing packaged data to users who desire it.

[0843] This invention is a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. Each component of this system will be explained in detail.

[0844] Data Acquisition Embodiment

[0845] Acquiring wearable device data

[0846] Device: A wearable device (such as a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is transferred to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[0847] Acquiring audio data

[0848] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recorded data is converted into text data using voice recognition software (e.g., Google Speech-to-Text API). The converted text data is sent to a server.

[0849] Acquisition of image data

[0850] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed by a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[0851] Data linkage implementation example

[0852] Server: Collects and integrates user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs), making it possible to create detailed user profiles that cannot be achieved with traditional data alone.

[0853] Embodiment of Manual User Data Registration

[0854] User: Manually inputs schedules, notes, and other data through a dedicated application. The input data is sent to the server for storage and analysis.

[0855] Data Analysis and Profile Generation Embodiments

[0856] Server: Stores all received data in a central database. It then analyzes the data using a generative AI model (e.g., GPT-4) to learn the user's behavioral patterns, thinking patterns, speech patterns, habits, etc. Based on the results of this analysis, a detailed profile of the user is generated.

[0857] Avatar Model Generation Embodiment

[0858] Server: Based on the analyzed data, a user avatar is generated, which is designed to replicate the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[0859] Avatar Optimization Embodiment

[0860] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (such as deep learning models).

[0861] User interaction embodiment

[0862] Setting up a dialogue system

[0863] Device: A dedicated application is used to set up the avatar so that the user can interact with it. The application configures the avatar's basic settings and establishes a connection with the user.

[0864] Running an interactive session

[0865] User: Opens a dedicated application and starts interacting with the avatar. The user selects voice or text input methods to send questions or instructions to the avatar.

[0866] Dialogue Generation Embodiments

[0867] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (such as GPT-4), which is then returned to the user via the avatar.

[0868] Data storage and sale practices

[0869] Packaging Data

[0870] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[0871] Sales Method

[0872] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[0873] Specific examples

[0874] Acquiring wearable device data

[0875] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[0876] Data analysis and profile generation

[0877] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[0878] Avatar Creation

[0879] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[0880] User interaction

[0881] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[0882] Data storage and sale

[0883] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[0884] Prompt Sentence Examples

[0885] "Please tell us how to learn behavioral patterns based on daily life data collected by users and enable natural interactions with avatars through a dedicated application."

[0886] As described above, the present invention is a system that effectively collects and analyzes user data, and generates and manages avatars that can interact with users. The collected data can also be used commercially.

[0887] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0888] Step 1:

[0889] Acquiring wearable device data

[0890] Device: A wearable device (e.g., a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is then transmitted to a smartphone via Bluetooth or Wi-Fi. The smartphone then uploads this data to a server at regular intervals.

[0891] Input: User's physical data (heart rate, steps, location)

[0892] Output: Real-time body data uploaded to a server

[0893] Step 2:

[0894] Acquiring audio data

[0895] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recordings are stored locally and then converted into text using speech recognition software (e.g., Google Speech-to-Text API). The text is then sent over the internet to a server.

[0896] Input: Recorded audio data

[0897] Output: Text data sent to the server

[0898] Step 3:

[0899] Acquisition of image data

[0900] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed using a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[0901] Input: Captured image data

[0902] Output: Processed image data sent to the server

[0903] Step 4:

[0904] Data Linkage

[0905] Server: Obtains user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs). This data is integrated with existing data and stored in a central database.

[0906] Input: Data retrieved from an external data source

[0907] Output: User life data integrated into a central database

[0908] Step 5:

[0909] Manual user data registration

[0910] User: Manually inputs schedules, memos, and other data using a dedicated application. This data is sent to the server via the application, where it is stored and analyzed.

[0911] Input: Data manually entered by the user (schedule, notes, etc.)

[0912] Output: Manually entered data sent to the server

[0913] Step 6:

[0914] Data integration and analysis

[0915] Server: The server stores all received data in a central database and analyzes it using a generative AI model (e.g., GPT-4), which learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a detailed user profile.

[0916] Input: All data stored in a central database (physical data, audio data, image data, external data, manually entered data)

[0917] Output: A user profile generated based on the analysis.

[0918] Step 7:

[0919] Avatar model generation

[0920] Server: Based on the generated user profile, a user avatar is generated, which reproduces the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[0921] Input: Generated user profile

[0922] Output: The generated avatar

[0923] Step 8:

[0924] Avatar Optimization

[0925] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (e.g., deep learning models).

[0926] Input: Generated avatar, user feature data

[0927] Output: Optimized avatar

[0928] Step 9:

[0929] Setting up a dialogue system

[0930] Device: Allows users to interact with their avatar through a dedicated application, which configures the avatar's basic settings and establishes a connection with the user.

[0931] Input: Avatar, user preference data

[0932] Output: The dialogue system is set up

[0933] Step 10:

[0934] Running an interactive session

[0935] User: Opens a dedicated application and starts interacting with the avatar. The user sends questions or instructions to the avatar via voice or text input.

[0936] Input: User questions or instructions (voice or text)

[0937] Output: Interactive session running

[0938] Step 11:

[0939] Dialogue generation

[0940] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (e.g., GPT-4), which is then returned to the user via the avatar.

[0941] Input: User input data, generative AI model

[0942] Output: The response generated by the avatar

[0943] Step 12:

[0944] Packaging Data

[0945] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[0946] Input: Generated user profile, interaction history

[0947] Output: Packaged dataset

[0948] Step 13:

[0949] Sales and Data Management

[0950] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[0951] Input: Packaged dataset, buyer information

[0952] Output: Issued access key, managed data

[0953] (Application example 1)

[0954] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0955] Conventional personalized shopping assistant systems did not fully utilize users' lifestyle data, resulting in insufficient suggestions based on the user's preferences and behavioral patterns. Furthermore, the dialogue with the user was often unnatural, resulting in a poor user experience. Furthermore, there was no way to record the user's responses to suggested items and reflect them in the next purchase, which prevented the accuracy of suggestions from improving. This made it difficult to provide a shopping experience tailored to each individual user.

[0956] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0957] In this invention, the server includes means for acquiring user lifestyle data, means for analyzing the acquired lifestyle data using a generative AI model, means for generating a profile based on the analysis that reproduces the user's thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile, means for engaging in dialogue with the user using the avatar, means for suggesting items suitable for the user based on the user's purchasing data and preference data, means for recording the user's responses to the suggested items and reflecting them in future suggestions, means for acquiring the user's physiological data using a wearable device, and means for providing an interface for the user to manually input data. This enables more accurate and personalized shopping suggestions based on the user's lifestyle data and preferences, and a more satisfying shopping experience through natural dialogue with the user.

[0958] "Lifestyle data" refers to various information related to the user's daily life, including physical data from wearable devices, location information, audio data, image data, and activity data on social media.

[0959] A "generative AI model" is an artificial intelligence model that analyzes acquired data and learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[0960] A "profile" is a collection of information that reproduces a user's thoughts, speaking style, habits, behavioral patterns, etc., generated based on the analysis results of a generative AI model.

[0961] An "avatar" is a character that reproduces the user's appearance, voice, actions, facial expressions, etc. based on the user's profile, and is a virtual entity that can interact with the user.

[0962] "Dialogue" refers to communication between a user and an avatar via voice or text.

[0963] "Purchase data" is information about products that a user has purchased in the past, including the purchase date and time, product name, category, price, and the like.

[0964] "Preference data" is information about products and services that a user likes, and is data that indicates the user's preferences based on past behavioral history, purchase history, survey results, etc.

[0965] "User response" refers to the reaction or feedback given by the user regarding the item suggested by the avatar.

[0966] This invention relates to a system that acquires and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. An embodiment of this system will be specifically explained below, focusing on how the server, terminal, and user cooperate to function.

[0967] Data Acquisition Embodiment

[0968] 1. Acquiring wearable device data

[0969] Device: A wearable device (e.g., a smartwatch or fitness tracker) collects a user's physical data (e.g., heart rate, steps, location, etc.) in real time. This data is periodically transmitted to a server via the Internet.

[0970] 2. Acquiring audio data

[0971] Terminal: A voice recording device (e.g., a smartphone or dedicated recording device) is used to record the user's everyday conversation. This voice data is converted into text using a voice recognition system (e.g., Google Speech-to-Text API) and sent to the server.

[0972] 3. Acquisition of image data

[0973] Terminal: A camera device (for example, a smartphone camera or a dedicated camera) is used to periodically capture the user's face and facial expressions, and the captured images are sent to the server as image data.

[0974] 4. Data Linkage

[0975] Server: Acquires data from multiple external data sources (e.g., social media platforms, online payment systems, etc.) and complements and integrates user lifestyle data.

[0976] 5. Manual user data registration

[0977] User: Manually inputs data such as schedules and notes through a dedicated application and sends it to the server.

[0978] Data analysis and profile generation

[0979] 1. Data integration and analysis

[0980] Server: Stores all received data in a database and feeds it into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[0981] Avatar Creation

[0982] 1. Avatar model generation

[0983] Server: Creates a user avatar based on the generated user profile. The avatar is designed to replicate the user's appearance, voice, and behavior.

[0984] 2. Avatar optimization

[0985] Server: Adjusts and optimizes the avatar's movements and facial expressions based on the user's characteristics.

[0986] User interaction embodiment

[0987] 1. Setting up the dialogue system

[0988] Terminal: Provides dedicated applications and is configured to allow users to interact with avatars.

[0989] 2. Running an interactive session

[0990] User: Opens the dedicated application and starts interacting with the avatar. The user inputs questions and instructions to the avatar via voice or text.

[0991] 3. Dialogue Generation

[0992] Server: Using the generative AI model based on the user's input, the avatar generates an appropriate response. The avatar then engages in natural dialogue with the user.

[0993] 4. Recommendations based on purchasing and preference data

[0994] Server: Suggests suitable products and services to users based on their purchase history and preference data.

[0995] 5. Record and reflect responses to proposed items

[0996] Server: Records the user's responses to the suggested items and uses that data to improve the next suggestion.

[0997] Specific examples

[0998] If a user asks the shopping assistant, "What cosmetics should I buy next?", the avatar will suggest, based on analyzed lifestyle data and past purchasing history, "You've been buying a lot of skin care products for sensitive skin lately, so how about this new sensitive skin cream?"

[0999] Example prompts to input to the generative AI model

[1000] "Suggest the next product to buy based on the user's past shopping history and preferences. The user prefers products for sensitive skin.

[1001] Input:

[1002] Past purchases: Skin care, sensitive skin cream, facial cleanser

[1003] User's recent purchase history: Sensitive skin cream, sensitive skin lotion

[1004] User Input:

[1005] What would you recommend for my next shopping trip?”

[1006] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[1007] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1008] Step 1:

[1009] Data collection (device)

[1010] Physical data such as heart rate, number of steps, and location information are collected from the user's wearable device. In addition, a voice recording device records the user's daily conversations, and a camera device collects image data of the user's face and facial expressions. The collected data is periodically sent to a server via the Internet.

[1011] Input: Data from wearable devices, audio recording devices, and camera devices

[1012] Output: User's physical data, voice data, image data sent to the server

[1013] Step 2:

[1014] Data analysis and integration (server)

[1015] All received data is stored in a database and analyzed using a generative AI model (such as GPT-4 or BERT). Voice data is converted into text using a speech recognition system (Google Speech-to-Text API, for example). Through analysis, the system learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[1016] Input: User's physical data, voice data, image data sent to the server

[1017] Output: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[1018] Step 3:

[1019] Profile Generation (Server)

[1020] Based on the analysis results of the generative AI model, a user profile is generated, which recreates and stores the user's thoughts, speech patterns, habits, and behavioral patterns.

[1021] Input: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[1022] Output: User profile

[1023] Step 4:

[1024] Creating an avatar (server)

[1025] Based on the generated profile, an avatar is created for the user. The avatar is designed to replicate the user's appearance, voice, and behavior. The avatar's movements, facial expressions, and other aspects are then optimized based on the user's characteristics.

[1026] Input: User profile

[1027] Output: User avatar

[1028] Step 5:

[1029] Providing interactive applications (terminals)

[1030] It provides an environment where users can interact with avatars through a dedicated application on their device. Users open the application and begin interacting with the avatar. Users' questions and instructions are entered by voice or text.

[1031] Input: User-entered voice or text data

[1032] Output: Display of interactive screen

[1033] Step 6:

[1034] Conversation creation (server)

[1035] Based on user input, a generative AI model is used to generate appropriate responses, which are then served to the user through an avatar.

[1036] Input: User's voice or text input data

[1037] Output: Natural dialogue responses from the avatar

[1038] Step 7:

[1039] Personalized Item Suggestions (Server)

[1040] Based on the user's purchasing data and preference data, the generative AI model proposes suitable products and services to the user. It analyzes past purchase history and preference data to generate the most appropriate proposals.

[1041] Input: Purchasing data, preference data

[1042] Output: Product and service recommendations to the user

[1043] Step 8:

[1044] Reflecting feedback on proposals (server)

[1045] The system records the user's responses to suggested items and updates the database to reflect them in the next suggestion, improving the accuracy of the suggestions.

[1046] Input: User response data

[1047] Output: Updated purchase data, preference data, and recommendation data

[1048] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1049] This invention relates to a system that collects and analyzes a user's lifestyle data and emotional data, generates an avatar of the user based on the data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[1050] Data Acquisition Embodiment

[1051] 1. Acquiring wearable device data

[1052] Device:

[1053] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[1054] 2. Acquiring audio data

[1055] Device:

[1056] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[1057] 3. Acquisition of image data

[1058] Device:

[1059] A camera device (for example, a smartphone camera or a dedicated camera) periodically takes a photo of the user's face and facial expression and sends it to a server as image data.

[1060] 4. Data Linkage

[1061] server:

[1062] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[1063] 5. Manual user data registration

[1064] User:

[1065] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[1066] Acquiring and analyzing emotion data

[1067] 1. Operating the Emotion Engine

[1068] server:

[1069] The emotion engine installed on the server analyzes the user's voice tone, facial expressions, and body language, and through this analysis, recognizes the user's emotional state (joy, anger, sadness, happiness, etc.).

[1070] Device:

[1071] The emotion data acquired by the emotion engine is integrated with lifestyle data.

[1072] Data analysis and profile generation

[1073] 1. Data integration and analysis

[1074] server:

[1075] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[1076] Avatar Creation

[1077] 1. Avatar model generation

[1078] server:

[1079] Based on the generated user profile and emotional data, a user avatar is created, which is designed to replicate the user's appearance, voice, and behavior.

[1080] 2. Avatar optimization

[1081] server:

[1082] Based on emotional data, the avatar's interaction patterns are adjusted and optimized, and improved based on user feedback.

[1083] User interaction embodiment

[1084] 1. Setting up the dialogue system

[1085] Device:

[1086] A dedicated application will be provided to create an environment in which users can interact with avatars.

[1087] 2. Running an interactive session

[1088] User:

[1089] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[1090] 3. Dialogue Generation

[1091] server:

[1092] Based on user input, the generative AI model generates appropriate responses, taking into account emotional data, allowing the avatar to engage in natural conversation with the user.

[1093] Data storage and sale practices

[1094] 1. Packaging the data

[1095] server:

[1096] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[1097] 2. Sales and Data Management

[1098] server:

[1099] The thought process data and emotion data of a specific person will be sold to interested users. An access key will be issued to the purchaser, and the data will be managed appropriately.

[1100] Specific examples

[1101] Acquiring wearable device data

[1102] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[1103] Acquiring emotion data

[1104] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[1105] Data analysis and profile generation

[1106] Server: The server integrates and analyzes the jogging data and emotional data, and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[1107] Avatar Creation

[1108] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[1109] User interaction

[1110] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[1111] Data storage and sale

[1112] Server: Packages the thought processes and emotional data of famous authors and executes the sales process for potential buyers.

[1113] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[1114] The processing flow will be explained below.

[1115] Step 1: Get the data

[1116] Device:

[1117] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[1118] 2. A voice recording device records the user's everyday conversations.

[1119] 3. The camera device periodically captures the user's face and facial expression.

[1120] User:

[1121] 1. Manually input data such as schedules and notes through a dedicated application.

[1122] server:

[1123] 1. Receive data sent from the terminal and store it in the database.

[1124] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[1125] Step 2: Obtaining emotion data

[1126] server:

[1127] 1. Use an emotion engine to analyze the user's tone of voice, facial expressions, and body language.

[1128] 2. Based on the analysis results, the user's emotional data (e.g., joy, anger, sadness, happiness, etc.) is recognized.

[1129] Device:

[1130] 1. Send emotion data to the server.

[1131] Step 3: Data synthesis and analysis

[1132] server:

[1133] 1. Consolidate all received data and store it in a database.

[1134] 2. The integrated data is input into a generative AI model for analysis and learning.

[1135] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, habits, and emotional data to generate a user profile.

[1136] Step 4: Avatar generation and optimization

[1137] server:

[1138] 1. Create a user avatar based on the generated user profile.

[1139] 2. Set the avatar's appearance, voice, and behavior, and prepare dialogue patterns that reflect the user's characteristics and emotional data.

[1140] 3. Adjust the avatar's movements and facial expressions based on emotional data and improve them based on user feedback.

[1141] Step 5: Setting up the dialogue system

[1142] Device:

[1143] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[1144] 2. Set up the application so that you can start interacting with the avatar.

[1145] Step 6: Running an interactive session

[1146] User:

[1147] 1. Open the dedicated application and start interacting with the avatar.

[1148] 2. Ask questions or give instructions to the avatar by voice or text.

[1149] server:

[1150] 1. Takes user input and generates an appropriate response using a generative AI model.

[1151] 2. Emotional data is also taken into account when generating responses, allowing the avatar to engage in natural dialogue with the user.

[1152] Step 7: Store and sell your data

[1153] server:

[1154] 1. Package and save the generated user profile and interaction history.

[1155] 2. Carry out procedures to sell a specific person's thought process data and emotional data.

[1156] 3. Issue an access key to the purchaser and manage the data appropriately.

[1157] Through these steps, a system is constructed that uses the user's lifestyle and emotional data to generate an avatar of the user and enable interaction. This system consistently supports the entire process, from data collection and analysis, to avatar generation, interaction, data storage, and sales.

[1158] Example 2

[1159] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1160] With conventional technology, it was difficult to effectively collect and analyze a user's lifestyle and emotional data and, based on that data, generate an avatar that could converse naturally with the user. It was also difficult to properly store user profiles and conversation histories and package them for commercial use. This resulted in the challenge of being unable to provide services that adequately meet the individual needs of users.

[1161] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1162] In this invention, the server includes means for acquiring user lifestyle data and emotional data, means for analyzing the data using a generative AI model, means for generating a profile that reproduces the user's behavioral patterns, thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile and emotional data, means for conducting a dialogue with the user using the avatar, and means for packaging and storing the user profile and dialogue history. This enables the generation of an avatar that can conduct natural dialogue according to the individual needs of the user and also facilitates commercial use of the collected data.

[1163] "Life data" refers to information related to the user's daily life, and specifically includes physical data, movement history, activity history, schedule, and the like.

[1164] "Emotional data" refers to information that indicates the user's emotional state, and includes data obtained from vocal tone, facial expressions, body language, and the like.

[1165] A "generative AI model" is an artificial intelligence model that analyzes large amounts of data and learns patterns, and has the ability to generate data generatively.

[1166] A "profile" is a collection of digital data that reproduces a user's behavioral patterns, thoughts, speech patterns, habits, etc.

[1167] An "avatar" is a digital character that mimics a user's appearance, voice, and behavior and is used to interact with the user.

[1168] "Dialogue history" refers to a record of the dialogue between the user and the avatar, and is used for later analysis and improvement.

[1169] "Packaging" refers to organizing data so that it is easy to use, and storing and managing it as a single entity.

[1170] A "server" is a machine that processes and stores data on a computer network and is the core of the system.

[1171] A "terminal" is a device that is directly operated by a user, such as a smartphone or a wearable device.

[1172] An "interface" is a means by which a user inputs data into a system and checks the output results.

[1173] This system collects and analyzes users' lifestyle and emotional data, and generates an avatar that can interact with the user based on that data. This system is primarily composed of three components: a server, a device, and the user.

[1174] Data Acquisition Embodiment

[1175] Acquiring wearable device data

[1176] Device: The user wears a wearable device such as a smartwatch or fitness tracker. This device collects real-time physical data such as heart rate, steps taken, and location. The collected data is sent to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[1177] Acquiring audio data

[1178] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[1179] Acquisition of image data

[1180] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[1181] Data Linkage

[1182] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[1183] Manual user data registration

[1184] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[1185] Acquiring and analyzing emotion data

[1186] Operating the Emotion Engine

[1187] Server: The emotion engine (e.g., Microsoft Azure Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to estimate emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis. These data are integrated with life data.

[1188] Data analysis and profile generation

[1189] Data Integration

[1190] Server: All collected data (physical data, audio data, image data, external data, manual data) will be stored in a central database. A database management system (e.g., PostgreSQL) will be used.

[1191] Profile Generation

[1192] Server: A generative AI model (e.g., GPT-3) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a detailed profile of the user.

[1193] Avatar Creation

[1194] Avatar model generation

[1195] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[1196] Avatar Optimization

[1197] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[1198] User interaction

[1199] Setting up a dialogue system

[1200] Device: A dedicated application is installed on the user's smartphone, providing an interface for interacting with the avatar.

[1201] Running an interactive session

[1202] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[1203] Dialogue generation

[1204] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[1205] Data storage and sale

[1206] Packaging Data

[1207] Server: Packages and stores the generated user profiles and interaction histories. This data is organized for commercial use.

[1208] Sales and Data Management

[1209] Server: The server processes the sale of a specific person's thought process and emotional data to interested users. It issues an access key to the purchaser and ensures proper management and security of the data.

[1210] Specific examples

[1211] Acquiring wearable device data

[1212] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[1213] Acquiring emotion data

[1214] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[1215] Data analysis and profile generation

[1216] Server: Integrates and analyzes jogging data and emotional data to learn a behavioral pattern, such as "the user jogs every morning at 7am."

[1217] Avatar Creation

[1218] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[1219] User interaction

[1220] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, the cities you visited last year were good."

[1221] Data storage and sale

[1222] Server: Packages the thought processes and emotional data of famous people and executes the sales process for potential buyers.

[1223] Prompt Sentence Examples

[1224] Please explain the specific processing steps of a system that collects a user's lifestyle and emotional data, generates an avatar based on that data, and allows the avatar to interact with the user.

[1225] In this way, the present invention provides a system that uses user data to generate and manage avatars that can naturally converse with users, and further utilizes that data commercially.

[1226] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1227] Step 1:

[1228] Data collection

[1229] Device: The user wears a wearable device (e.g., a smartwatch). The device collects real-time physical data such as heart rate, steps, and location information, and transmits it to a smartphone via Bluetooth or Wi-Fi. The smartphone periodically uploads this data to a server via the Internet.

[1230] Input: Sensor data from wearable devices (heart rate, steps, location information)

[1231] Output: Body data sent to the server

[1232] How it works: The smartwatch's sensors detect body data, and the smartphone receives the data and transfers it to a server.

[1233] Step 2:

[1234] Acquiring audio data

[1235] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[1236] Input: Audio data recorded by a smartphone microphone

[1237] Output: The audio data converted to text is sent to the server.

[1238] How it works: Your smartphone records your everyday conversations, and a voice recognition system converts the audio into text and uploads it to a server.

[1239] Step 3:

[1240] Acquisition of image data

[1241] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[1242] Input: Facial image taken with a smartphone camera and facial expression data

[1243] Output: Compressed image data is sent to the server

[1244] How it works: The smartphone camera takes a photo of your face, compresses the image data, and uploads it to a server.

[1245] Step 4:

[1246] Data Linkage

[1247] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[1248] Input: Additional data retrieved from an external data source

[1249] Output: Integrated user life data

[1250] How it works: The server accesses external data sources via APIs to retrieve additional data and integrate it with the user's life data.

[1251] Step 5:

[1252] Manual user data registration

[1253] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[1254] Input: Manual data entered by the user through a dedicated application

[1255] Output: Manual data stored on the server

[1256] How it works: The user enters schedules and notes into a dedicated application, which are then sent to the server in real time.

[1257] Step 6:

[1258] Acquiring and analyzing emotion data

[1259] Server: The emotion engine (e.g., Microsoft Azure Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to infer emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis.

[1260] Input: Voice tone, facial expression data, body language

[1261] Output: Parsed emotion data

[1262] How it works: The server analyzes the audio and image data, and the emotion engine recognizes the user's emotional state.

[1263] Step 7:

[1264] Data integration and analysis

[1265] Server: All collected data (physical data, audio data, image data, external data, manual data) is stored in a central database. A generative AI model (e.g., GPT-3) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits.

[1266] Input: Integrated data stored in a central database

[1267] Output: Parsed user profile

[1268] How it works: A server aggregates various data, and a generative AI model analyzes it to create a profile of the user.

[1269] Step 8:

[1270] Avatar model generation

[1271] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[1272] Input: User profile and emotion data

[1273] Output: Generated 3D avatar

[1274] How it works: 3D model generation software creates an avatar that mimics the user's appearance and behavior, and a voice synthesis system generates the avatar's voice.

[1275] Step 9:

[1276] Avatar Optimization

[1277] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[1278] Input: User feedback and sentiment data

[1279] Output: Optimized avatar

[1280] How it works: The server collects user feedback and uses a generative AI model to optimize the avatar's interaction patterns.

[1281] Step 10:

[1282] Setting up a dialogue system

[1283] Device: A dedicated application is installed on the user's smartphone, which provides an interface for interacting with the avatar.

[1284] Input: Dedicated application

[1285] Output: Avatar interaction interface

[1286] How it works: A dedicated application is installed on a smartphone and provides an interface for interacting with the avatar.

[1287] Step 11:

[1288] Running an interactive session

[1289] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[1290] Input: User questions and instructions

[1291] Output: Avatar response

[1292] How it works: The user opens a dedicated application and interacts with an avatar based on input questions and instructions.

[1293] Step 12:

[1294] Dialogue generation

[1295] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[1296] Input: User input and emotion data

[1297] Output: The generated response

[1298] How it works: The server uses a generative AI model to generate a response, which the avatar relays to the user.

[1299] Step 13:

[1300] Data storage and sale

[1301] Server: Packages and stores the generated user profiles and interaction histories. This data is organized for commercial use.

[1302] Input: User profile and interaction history

[1303] Output: Packaged data

[1304] How it works: The server packages and stores user profiles and interaction histories.

[1305] Sales and Data Management

[1306] Server: The server processes the sale of a specific person's thought process and emotional data to interested users. It issues an access key to the purchaser and ensures proper management and security of the data.

[1307] Input: Sales Request

[1308] Output: Access key

[1309] How it works: The server accepts sales requests and sells and manages data.

[1310] (Application example 2)

[1311] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1312] Conventional avatar generation systems are primarily limited to online use and have not contributed sufficiently to improving customer service in brick-and-mortar stores. In particular, it has been difficult to analyze individual users' lifestyle and emotional data in real time and provide personalized recommendations and support based on that data.

[1313] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1314] In this invention, the server includes means for acquiring user lifestyle data, means for analyzing the data using a generative AI model, means for generating a profile that reproduces the user's thoughts, speech patterns, and habits, means for creating an avatar for the user, means for conducting dialogue with the user, means for collecting the user's shopping history and emotion data, means for recommending optimal products and services in physical stores, and means for providing a dialogue function with an avatar that provides real-time support when the user experiences difficulty in the store. This makes it possible to provide personalized services tailored to the individual needs of users even in physical stores.

[1315] "User's lifestyle data" is information relating to the user's daily life, including, for example, location information, walking data, heart rate, shopping history, and the like.

[1316] A "generative AI model" is an artificial intelligence algorithm used to analyze large amounts of data and generate new information and predictions.

[1317] A "profile" is a collection of information created to reproduce a user's characteristics, such as their way of thinking, speaking, and habits.

[1318] An "avatar" is a virtual persona created to replicate a user's appearance, voice, and behavior.

[1319] "Emotion data" is data that indicates the user's emotional state, and is obtained from voice tone, facial expressions, body movements, and the like.

[1320] "Shopping history" refers to data on products purchased by a user in the past and related information.

[1321] A "physical store" is a sales or commercial facility that a user can physically visit.

[1322] "Recommendation" is the action or function of suggesting optimal products or services based on a user's past data and current needs.

[1323] "Providing support in real time" means providing the information and help a user needs at the exact moment they are using the application.

[1324] The "interactive function" is a function that allows users to have natural conversations with their avatars through voice and text.

[1325] This invention relates to a system that collects and analyzes a user's lifestyle data and emotional data, generates an avatar of the user based on the collected data, and enables interaction with the user. Furthermore, this system can be applied to providing personalized customer service in brick-and-mortar stores.

[1326] 1. Data Acquisition Method

[1327] Acquiring wearable device data

[1328] Device: A wearable device (such as a smartwatch or fitness tracker) collects the user's physical data (heart rate, steps, location, etc.) in real time. This data is periodically sent to a server via the internet.

[1329] Acquiring audio data

[1330] Device: A voice recording device (smartphone or dedicated recording device) records the user's everyday conversation. This voice data is converted into text by a voice recognition system and sent to a server.

[1331] Acquisition of image data

[1332] Device: A camera device (such as a smartphone camera or a dedicated camera) periodically takes a photo of the user's face and facial expression, and sends the image data to the server.

[1333] Data Linkage

[1334] Server: Acquires data from multiple external data sources (social media platforms, electronic communication tools, online payment systems) and complements and integrates user life data.

[1335] Manual user data registration

[1336] User: Through a dedicated application, users can manually input data such as schedules and notes and send it to the server.

[1337] 2. Acquisition and analysis of emotion data

[1338] Operating the Emotion Engine

[1339] Server: The emotion engine installed on the server analyzes the user's tone of voice and facial expressions, and recognizes the user's emotional state through this analysis.

[1340] Terminal: Emotion data acquired by the emotion engine is integrated with lifestyle data.

[1341] 3. Data analysis and profile generation

[1342] Data integration and analysis

[1343] Server: Stores all received data in a database and inputs it into a generative AI model. The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[1344] 4. Create your avatar

[1345] Avatar model generation

[1346] Server: Creates a user avatar based on the generated user profile and emotional data. The avatar is designed to replicate the user's appearance, voice, and behavior.

[1347] 5. User interaction implementation

[1348] Setting up a dialogue system

[1349] Terminal: Provides dedicated applications and creates an environment in which users can interact with avatars.

[1350] Running an interactive session

[1351] User: Opens a dedicated application and starts interacting with the avatar. The user can enter questions or instructions to the avatar via voice or text.

[1352] Dialogue generation

[1353] Server: Based on user input, the generative AI model generates an appropriate response, taking into account emotional data. The avatar then engages in natural dialogue with the user.

[1354] 6. Physical store applications

[1355] Collecting shopping history and sentiment data

[1356] Terminal: Collects user shopping history and emotion data through the application and sends it to the server.

[1357] Recommendation function in physical stores

[1358] Server: Analyzes data using generative AI models to recommend optimal products and services in physical stores.

[1359] Real-time support

[1360] Terminal: Provides interaction with an avatar that provides real-time support to users experiencing difficulties in the store.

[1361] Specific examples

[1362] Acquiring jogging data: When a user goes for a jog every morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[1363] Example of avatar dialogue: At 7am, the avatar asks the user, "Are you going for a jog again today? It looks like a relaxing time."

[1364] Example of interaction with an avatar in a physical store: A user arrives at a physical store and says to the avatar, "I'm looking for a recommended winter coat." The avatar recommends the best coat based on the user's past purchase history and current emotional state.

[1365] Example prompt for a generative AI model:

[1366] "When a user is looking for something in a physical store, create a conversation that recommends the best product based on their past purchase data and current heart rate data."

[1367] This allows us to make the most of user data and create a personalized shopping experience in physical stores.

[1368] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1369] Step 1:

[1370] Data collection

[1371] Device: Collects real-time data such as heart rate, steps, and location information from the user's wearable device (smartwatch or fitness tracker), as well as audio and image data from smartphones and cameras.

[1372] Input: Heart rate, steps, location, audio data, image data.

[1373] Output: The collected life data and emotion data are sent to the server.

[1374] Step 2:

[1375] Data transmission

[1376] Terminal: Sends collected lifestyle and emotional data to a server via the Internet.

[1377] Input: Life and emotion data collected on the device.

[1378] Output: Life data and emotion data sent to the server.

[1379] Step 3:

[1380] Data analysis

[1381] Server: The received lifestyle and emotion data is stored in a cloud-based data analysis system and input into the generative AI model.

[1382] Input: Submitted lifestyle and emotion data.

[1383] Output: A profile of the user's behavioral patterns and emotional state based on the analyzed data.

[1384] Step 4:

[1385] Profile Generation

[1386] Server: The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[1387] Input: Data analysis results.

[1388] Output: A profile reflecting the user's behavioral patterns and emotional state.

[1389] Step 5:

[1390] Avatar Creation

[1391] Server: Creates an avatar for the user based on the generated profile. The avatar is designed to replicate the user's appearance, voice, and behavior.

[1392] Input: The user's profile.

[1393] Output: An avatar that replicates the user's features.

[1394] Step 6:

[1395] Avatar interaction system setup

[1396] Terminal: Creates an environment where users can interact with avatars through a dedicated application.

[1397] Input: The avatar installed in the application.

[1398] Output: An interactive application.

[1399] Step 7:

[1400] Start of user interaction

[1401] User: Opens a dedicated application and starts interacting with the avatar. The user can enter questions or instructions to the avatar via voice or text.

[1402] Input: A user's voice or text question or instruction.

[1403] Output: Input data for the avatar.

[1404] Step 8:

[1405] Response Generation

[1406] Server: Uses a generative AI model based on user input and takes emotional data into account to generate an appropriate response.

[1407] Input: User questions or instructions.

[1408] Output: The response that the avatar provides to the user.

[1409] Step 9:

[1410] In-store recommendations

[1411] Server: Analyzes the user's shopping history and emotional data to recommend optimal products and services in physical stores.

[1412] Input: Shopping history and sentiment data.

[1413] Output: Recommended products or services.

[1414] Step 10:

[1415] Real-time support

[1416] Terminal: Provides an interactive function with an avatar that provides real-time support when users encounter problems in a physical store.

[1417] Input: Real-time user questions and concerns.

[1418] Output: Real-time support and guidance.

[1419] This is the flow of processing for this program. This makes it possible to utilize individual user data and provide personalized services even in physical stores.

[1420] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1421] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1422] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1423] [Third embodiment]

[1424] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1425] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1426] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1427] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1428] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1430] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1431] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1432] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1433] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1434] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1435] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1436] This invention relates to a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[1437] Data Acquisition Embodiment

[1438] 1. Acquiring wearable device data

[1439] Device:

[1440] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[1441] 2. Acquiring audio data

[1442] Device:

[1443] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[1444] 3. Acquisition of image data

[1445] Device:

[1446] A camera device (for example, a smartphone camera or a dedicated camera) periodically captures the user's face and facial expressions and sends them to a server as image data.

[1447] 4. Data Linkage

[1448] server:

[1449] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[1450] 5. Manual user data registration

[1451] User:

[1452] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[1453] Data analysis and profile generation

[1454] 1. Data integration and analysis

[1455] server:

[1456] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[1457] Avatar Creation

[1458] 1. Avatar model generation

[1459] server:

[1460] Based on the generated user profile, an avatar is created for the user, which is designed to replicate the user's appearance, voice, and behavior.

[1461] 2. Avatar optimization

[1462] server:

[1463] The avatar's movements and facial expressions are adjusted and optimized based on the user's characteristics.

[1464] User interaction embodiment

[1465] 1. Setting up the dialogue system

[1466] Device:

[1467] A dedicated application is provided and configured to allow users to interact with the avatar.

[1468] 2. Running an interactive session

[1469] User:

[1470] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[1471] 3. Dialogue Generation

[1472] server:

[1473] Using generative AI models, the avatar generates appropriate responses based on user input, allowing the avatar to engage in natural dialogue with the user.

[1474] Data storage and sale practices

[1475] 1. Packaging the data

[1476] server:

[1477] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[1478] 2. Sales and Data Management

[1479] server:

[1480] We will carry out procedures to sell the thought process data of a specific person to interested users. Purchasers will be issued an access key, and the data will be managed appropriately.

[1481] Specific examples

[1482] Acquiring wearable device data

[1483] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[1484] Data analysis and profile generation

[1485] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[1486] Avatar Creation

[1487] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[1488] User interaction

[1489] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[1490] Data storage and sale

[1491] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[1492] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[1493] The processing flow will be explained below.

[1494] Step 1: Get the data

[1495] Device:

[1496] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[1497] 2. A voice recording device records the user's everyday conversations.

[1498] 3. The camera device periodically captures the user's face and facial expression.

[1499] User:

[1500] 1. Manually input data such as schedules and notes through a dedicated application.

[1501] server:

[1502] 1. Receive data sent from the terminal and store it in the database.

[1503] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[1504] Step 2: Data synthesis and analysis

[1505] server:

[1506] 1. Centralize all received data and integrate it into a database.

[1507] 2. The integrated data is input into a generative AI model for analysis and learning.

[1508] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a user profile.

[1509] Step 3: Generate an avatar

[1510] server:

[1511] 1. Create a user avatar based on the generated user profile.

[1512] 2. Set the avatar's appearance, voice, and behavior to replicate the user's characteristics.

[1513] 3. Optimize the avatar's movements and facial expressions and adjust them based on user feedback.

[1514] Step 4: Setting up the dialogue system

[1515] Device:

[1516] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[1517] 2. Set up the application so that you can start interacting with the avatar.

[1518] Step 5: Running an interactive session

[1519] User:

[1520] 1. Open the dedicated application and start interacting with the avatar.

[1521] 2. Ask questions or give instructions to the avatar by voice or text.

[1522] server:

[1523] 1. Takes user input and generates an appropriate response using a generative AI model.

[1524] 2. The avatar engages in natural dialogue with the user.

[1525] Step 6: Store and sell your data

[1526] server:

[1527] 1. Package and save the generated user profile and interaction history.

[1528] 2. Proceed with the sale of a specific person's thought process data.

[1529] 3. Issue an access key to the purchaser and manage the data appropriately.

[1530] Through these steps, this system utilizes the user's daily life data to generate an avatar of the user and enable interaction.The system can also store and sell the data, enabling a wide range of uses.

[1531] Example 1

[1532] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1533] Existing interactive avatar systems lack mechanisms for centrally collecting, integrating, and analyzing user data, making it difficult to accurately reproduce the user's behavior and thoughts. Furthermore, there is a need for a system that can effectively collect, integrate, and analyze users' lifestyle data, allowing for natural interactions with individually optimized avatars and enabling commercial use of the collected data.

[1534] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1535] In this invention, the server includes means for acquiring user life data, means for packaging and selling the generated profile and interaction history, and means for integrating data acquired from external data sources. This not only enables effective collection and analysis of user life data and generation of individually optimized avatars that enable natural interactions, but also makes it possible to commercially utilize the collected data.

[1536] "User lifestyle data" refers to information related to a user's daily life, such as wearable device data, audio data, image data, and social media data.

[1537] A "generative AI model" is an artificial intelligence algorithm used to analyze collected data and learn user behavior patterns and thoughts.

[1538] A "profile" is a data structure that reproduces a user's behavior, thoughts, speech patterns, and habits analyzed by a generative AI model.

[1539] An "avatar" is a digital character created to replicate a user's appearance, voice, and behavior.

[1540] "External data sources" are other systems or databases that provide users' life data, such as social media platforms, electronic communication tools, and online payment systems.

[1541] The "dialogue history" is a record of the dialogue that took place between the user and the avatar.

[1542] "Packaging" is the process of organizing and saving the generated profiles and interaction histories as a single data set.

[1543] A "sales vehicle" is a process or system for providing packaged data to users who desire it.

[1544] This invention is a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. Each component of this system will be explained in detail.

[1545] Data Acquisition Embodiment

[1546] Acquiring wearable device data

[1547] Device: A wearable device (such as a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is transferred to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[1548] Acquiring audio data

[1549] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recorded data is converted into text data using voice recognition software (e.g., Google Speech-to-Text API). The converted text data is sent to a server.

[1550] Acquisition of image data

[1551] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed by a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[1552] Data linkage implementation example

[1553] Server: Collects and integrates user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs), making it possible to create detailed user profiles that cannot be achieved with traditional data alone.

[1554] Embodiment of Manual User Data Registration

[1555] User: Manually inputs schedules, notes, and other data through a dedicated application. The input data is sent to the server for storage and analysis.

[1556] Data Analysis and Profile Generation Embodiments

[1557] Server: Stores all received data in a central database. It then analyzes the data using a generative AI model (e.g., GPT-4) to learn the user's behavioral patterns, thinking patterns, speech patterns, habits, etc. Based on the results of this analysis, a detailed profile of the user is generated.

[1558] Avatar Model Generation Embodiment

[1559] Server: Based on the analyzed data, a user avatar is generated, which is designed to replicate the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[1560] Avatar Optimization Embodiment

[1561] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (such as deep learning models).

[1562] User interaction embodiment

[1563] Setting up a dialogue system

[1564] Device: A dedicated application is used to set up the avatar so that the user can interact with it. The application configures the avatar's basic settings and establishes a connection with the user.

[1565] Running an interactive session

[1566] User: Opens a dedicated application and starts interacting with the avatar. The user selects voice or text input methods to send questions or instructions to the avatar.

[1567] Dialogue Generation Embodiments

[1568] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (such as GPT-4), which is then returned to the user via the avatar.

[1569] Data storage and sale practices

[1570] Packaging Data

[1571] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[1572] Sales Method

[1573] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[1574] Specific examples

[1575] Acquiring wearable device data

[1576] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[1577] Data analysis and profile generation

[1578] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[1579] Avatar Creation

[1580] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[1581] User interaction

[1582] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[1583] Data storage and sale

[1584] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[1585] Prompt Sentence Examples

[1586] "Please tell us how to learn behavioral patterns based on daily life data collected by users and enable natural interactions with avatars through a dedicated application."

[1587] As described above, the present invention is a system that effectively collects and analyzes user data, and generates and manages avatars that can interact with users. The collected data can also be used commercially.

[1588] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1589] Step 1:

[1590] Acquiring wearable device data

[1591] Device: A wearable device (e.g., a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is then transmitted to a smartphone via Bluetooth or Wi-Fi. The smartphone then uploads this data to a server at regular intervals.

[1592] Input: User's physical data (heart rate, steps, location)

[1593] Output: Real-time body data uploaded to a server

[1594] Step 2:

[1595] Acquiring audio data

[1596] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recordings are stored locally and then converted into text using speech recognition software (e.g., Google Speech-to-Text API). The text is then sent over the internet to a server.

[1597] Input: Recorded audio data

[1598] Output: Text data sent to the server

[1599] Step 3:

[1600] Acquisition of image data

[1601] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed using a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[1602] Input: Captured image data

[1603] Output: Processed image data sent to the server

[1604] Step 4:

[1605] Data Linkage

[1606] Server: Obtains user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs). This data is integrated with existing data and stored in a central database.

[1607] Input: Data retrieved from an external data source

[1608] Output: User life data integrated into a central database

[1609] Step 5:

[1610] Manual user data registration

[1611] User: Manually inputs schedules, memos, and other data using a dedicated application. This data is sent to the server via the application, where it is stored and analyzed.

[1612] Input: Data manually entered by the user (schedule, notes, etc.)

[1613] Output: Manually entered data sent to the server

[1614] Step 6:

[1615] Data integration and analysis

[1616] Server: The server stores all received data in a central database and analyzes it using a generative AI model (e.g., GPT-4), which learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a detailed user profile.

[1617] Input: All data stored in a central database (physical data, audio data, image data, external data, manually entered data)

[1618] Output: A user profile generated based on the analysis.

[1619] Step 7:

[1620] Avatar model generation

[1621] Server: Based on the generated user profile, a user avatar is generated, which reproduces the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[1622] Input: Generated user profile

[1623] Output: The generated avatar

[1624] Step 8:

[1625] Avatar Optimization

[1626] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (e.g., deep learning models).

[1627] Input: Generated avatar, user feature data

[1628] Output: Optimized avatar

[1629] Step 9:

[1630] Setting up a dialogue system

[1631] Device: Allows users to interact with their avatar through a dedicated application, which configures the avatar's basic settings and establishes a connection with the user.

[1632] Input: Avatar, user preference data

[1633] Output: The dialogue system is set up

[1634] Step 10:

[1635] Running an interactive session

[1636] User: Opens a dedicated application and starts interacting with the avatar. The user sends questions or instructions to the avatar via voice or text input.

[1637] Input: User questions or instructions (voice or text)

[1638] Output: Interactive session running

[1639] Step 11:

[1640] Dialogue generation

[1641] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (e.g., GPT-4), which is then returned to the user via the avatar.

[1642] Input: User input data, generative AI model

[1643] Output: The response generated by the avatar

[1644] Step 12:

[1645] Packaging Data

[1646] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[1647] Input: Generated user profile, interaction history

[1648] Output: Packaged dataset

[1649] Step 13:

[1650] Sales and Data Management

[1651] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[1652] Input: Packaged dataset, buyer information

[1653] Output: Issued access key, managed data

[1654] (Application example 1)

[1655] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1656] Conventional personalized shopping assistant systems did not fully utilize users' lifestyle data, resulting in insufficient suggestions based on the user's preferences and behavioral patterns. Furthermore, the dialogue with the user was often unnatural, resulting in a poor user experience. Furthermore, there was no way to record the user's responses to suggested items and reflect them in the next purchase, which prevented the accuracy of suggestions from improving. This made it difficult to provide a shopping experience tailored to each individual user.

[1657] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1658] In this invention, the server includes means for acquiring user lifestyle data, means for analyzing the acquired lifestyle data using a generative AI model, means for generating a profile based on the analysis that reproduces the user's thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile, means for engaging in dialogue with the user using the avatar, means for suggesting items suitable for the user based on the user's purchasing data and preference data, means for recording the user's responses to the suggested items and reflecting them in future suggestions, means for acquiring the user's physiological data using a wearable device, and means for providing an interface for the user to manually input data. This enables more accurate and personalized shopping suggestions based on the user's lifestyle data and preferences, and a more satisfying shopping experience through natural dialogue with the user.

[1659] "Lifestyle data" refers to various information related to the user's daily life, including physical data from wearable devices, location information, audio data, image data, and activity data on social media.

[1660] A "generative AI model" is an artificial intelligence model that analyzes acquired data and learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[1661] A "profile" is a collection of information that reproduces a user's thoughts, speaking style, habits, behavioral patterns, etc., generated based on the analysis results of a generative AI model.

[1662] An "avatar" is a character that reproduces the user's appearance, voice, actions, facial expressions, etc. based on the user's profile, and is a virtual entity that can interact with the user.

[1663] "Dialogue" refers to communication between a user and an avatar via voice or text.

[1664] "Purchase data" is information about products that a user has purchased in the past, including the purchase date and time, product name, category, price, and the like.

[1665] "Preference data" is information about products and services that a user likes, and is data that indicates the user's preferences based on past behavioral history, purchase history, survey results, etc.

[1666] "User response" refers to the reaction or feedback given by the user regarding the item suggested by the avatar.

[1667] This invention relates to a system that acquires and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. An embodiment of this system will be specifically explained below, focusing on how the server, terminal, and user cooperate to function.

[1668] Data Acquisition Embodiment

[1669] 1. Acquiring wearable device data

[1670] Device: A wearable device (e.g., a smartwatch or fitness tracker) collects a user's physical data (e.g., heart rate, steps, location, etc.) in real time. This data is periodically transmitted to a server via the Internet.

[1671] 2. Acquiring audio data

[1672] Terminal: A voice recording device (e.g., a smartphone or dedicated recording device) is used to record the user's everyday conversation. This voice data is converted into text using a voice recognition system (e.g., Google Speech-to-Text API) and sent to the server.

[1673] 3. Acquisition of image data

[1674] Terminal: A camera device (for example, a smartphone camera or a dedicated camera) is used to periodically capture the user's face and facial expressions, and the captured images are sent to the server as image data.

[1675] 4. Data Linkage

[1676] Server: Acquires data from multiple external data sources (e.g., social media platforms, online payment systems, etc.) and complements and integrates user lifestyle data.

[1677] 5. Manual user data registration

[1678] User: Manually inputs data such as schedules and notes through a dedicated application and sends it to the server.

[1679] Data analysis and profile generation

[1680] 1. Data integration and analysis

[1681] Server: Stores all received data in a database and feeds it into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[1682] Avatar Creation

[1683] 1. Avatar model generation

[1684] Server: Creates a user avatar based on the generated user profile. The avatar is designed to replicate the user's appearance, voice, and behavior.

[1685] 2. Avatar optimization

[1686] Server: Adjusts and optimizes the avatar's movements and facial expressions based on the user's characteristics.

[1687] User interaction embodiment

[1688] 1. Setting up the dialogue system

[1689] Terminal: Provides dedicated applications and is configured to allow users to interact with avatars.

[1690] 2. Running an interactive session

[1691] User: Opens the dedicated application and starts interacting with the avatar. The user inputs questions and instructions to the avatar via voice or text.

[1692] 3. Dialogue Generation

[1693] Server: Using the generative AI model based on the user's input, the avatar generates an appropriate response. The avatar then engages in natural dialogue with the user.

[1694] 4. Recommendations based on purchasing and preference data

[1695] Server: Suggests suitable products and services to users based on their purchase history and preference data.

[1696] 5. Record and reflect responses to proposed items

[1697] Server: Records the user's responses to the suggested items and uses that data to improve the next suggestion.

[1698] Specific examples

[1699] If a user asks the shopping assistant, "What cosmetics should I buy next?", the avatar will suggest, based on analyzed lifestyle data and past purchasing history, "You've been buying a lot of skin care products for sensitive skin lately, so how about this new sensitive skin cream?"

[1700] Example prompts to input to the generative AI model

[1701] "Suggest the next product to buy based on the user's past shopping history and preferences. The user prefers products for sensitive skin.

[1702] Input:

[1703] Past purchases: Skin care, sensitive skin cream, facial cleanser

[1704] User's recent purchase history: Sensitive skin cream, sensitive skin lotion

[1705] User Input:

[1706] What would you recommend for my next shopping trip?”

[1707] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[1708] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1709] Step 1:

[1710] Data collection (device)

[1711] Physical data such as heart rate, number of steps, and location information are collected from the user's wearable device. In addition, a voice recording device records the user's daily conversations, and a camera device collects image data of the user's face and facial expressions. The collected data is periodically sent to a server via the Internet.

[1712] Input: Data from wearable devices, audio recording devices, and camera devices

[1713] Output: User's physical data, voice data, image data sent to the server

[1714] Step 2:

[1715] Data analysis and integration (server)

[1716] All received data is stored in a database and analyzed using a generative AI model (such as GPT-4 or BERT). Voice data is converted into text using a speech recognition system (Google Speech-to-Text API, for example). Through analysis, the system learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[1717] Input: User's physical data, voice data, image data sent to the server

[1718] Output: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[1719] Step 3:

[1720] Profile Generation (Server)

[1721] Based on the analysis results of the generative AI model, a user profile is generated, which recreates and stores the user's thoughts, speech patterns, habits, and behavioral patterns.

[1722] Input: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[1723] Output: User profile

[1724] Step 4:

[1725] Creating an avatar (server)

[1726] Based on the generated profile, an avatar is created for the user. The avatar is designed to replicate the user's appearance, voice, and behavior. The avatar's movements, facial expressions, and other aspects are then optimized based on the user's characteristics.

[1727] Input: User profile

[1728] Output: User avatar

[1729] Step 5:

[1730] Providing interactive applications (terminals)

[1731] It provides an environment where users can interact with avatars through a dedicated application on their device. Users open the application and begin interacting with the avatar. Users' questions and instructions are entered by voice or text.

[1732] Input: User-entered voice or text data

[1733] Output: Display of interactive screen

[1734] Step 6:

[1735] Conversation creation (server)

[1736] Based on user input, a generative AI model is used to generate appropriate responses, which are then served to the user through an avatar.

[1737] Input: User's voice or text input data

[1738] Output: Natural dialogue responses from the avatar

[1739] Step 7:

[1740] Personalized Item Suggestions (Server)

[1741] Based on the user's purchasing data and preference data, the generative AI model proposes suitable products and services to the user. It analyzes past purchase history and preference data to generate the most appropriate proposals.

[1742] Input: Purchasing data, preference data

[1743] Output: Product and service recommendations to the user

[1744] Step 8:

[1745] Reflecting feedback on proposals (server)

[1746] The system records the user's responses to suggested items and updates the database to reflect them in the next suggestion, improving the accuracy of the suggestions.

[1747] Input: User response data

[1748] Output: Updated purchase data, preference data, and recommendation data

[1749] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1750] This invention relates to a system that collects and analyzes a user's lifestyle data and emotional data, generates an avatar of the user based on the data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[1751] Data Acquisition Embodiment

[1752] 1. Acquiring wearable device data

[1753] Device:

[1754] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[1755] 2. Acquiring audio data

[1756] Device:

[1757] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[1758] 3. Acquisition of image data

[1759] Device:

[1760] A camera device (for example, a smartphone camera or a dedicated camera) periodically takes a photo of the user's face and facial expression and sends it to a server as image data.

[1761] 4. Data Linkage

[1762] server:

[1763] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[1764] 5. Manual user data registration

[1765] User:

[1766] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[1767] Acquiring and analyzing emotion data

[1768] 1. Operating the Emotion Engine

[1769] server:

[1770] The emotion engine installed on the server analyzes the user's voice tone, facial expressions, and body language, and through this analysis, recognizes the user's emotional state (joy, anger, sadness, happiness, etc.).

[1771] Device:

[1772] The emotion data acquired by the emotion engine is integrated with lifestyle data.

[1773] Data analysis and profile generation

[1774] 1. Data integration and analysis

[1775] server:

[1776] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[1777] Avatar Creation

[1778] 1. Avatar model generation

[1779] server:

[1780] Based on the generated user profile and emotional data, a user avatar is created, which is designed to replicate the user's appearance, voice, and behavior.

[1781] 2. Avatar optimization

[1782] server:

[1783] Based on emotional data, the avatar's interaction patterns are adjusted and optimized, and improved based on user feedback.

[1784] User interaction embodiment

[1785] 1. Setting up the dialogue system

[1786] Device:

[1787] A dedicated application will be provided to create an environment in which users can interact with avatars.

[1788] 2. Running an interactive session

[1789] User:

[1790] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[1791] 3. Dialogue Generation

[1792] server:

[1793] Based on user input, the generative AI model generates appropriate responses, taking into account emotional data, allowing the avatar to engage in natural conversation with the user.

[1794] Data storage and sale practices

[1795] 1. Packaging the data

[1796] server:

[1797] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[1798] 2. Sales and Data Management

[1799] server:

[1800] The thought process data and emotion data of a specific person will be sold to interested users. An access key will be issued to the purchaser, and the data will be managed appropriately.

[1801] Specific examples

[1802] Acquiring wearable device data

[1803] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[1804] Acquiring emotion data

[1805] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[1806] Data analysis and profile generation

[1807] Server: The server integrates and analyzes the jogging data and emotional data, and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[1808] Avatar Creation

[1809] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[1810] User interaction

[1811] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[1812] Data storage and sale

[1813] Server: Packages the thought processes and emotional data of famous authors and executes the sales process for potential buyers.

[1814] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[1815] The processing flow will be explained below.

[1816] Step 1: Get the data

[1817] Device:

[1818] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[1819] 2. A voice recording device records the user's everyday conversations.

[1820] 3. The camera device periodically captures the user's face and facial expression.

[1821] User:

[1822] 1. Manually input data such as schedules and notes through a dedicated application.

[1823] server:

[1824] 1. Receive data sent from the terminal and store it in the database.

[1825] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[1826] Step 2: Obtaining emotion data

[1827] server:

[1828] 1. Use an emotion engine to analyze the user's tone of voice, facial expressions, and body language.

[1829] 2. Based on the analysis results, the user's emotional data (e.g., joy, anger, sadness, happiness, etc.) is recognized.

[1830] Device:

[1831] 1. Send emotion data to the server.

[1832] Step 3: Data synthesis and analysis

[1833] server:

[1834] 1. Consolidate all received data and store it in a database.

[1835] 2. The integrated data is input into a generative AI model for analysis and learning.

[1836] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, habits, and emotional data to generate a user profile.

[1837] Step 4: Avatar generation and optimization

[1838] server:

[1839] 1. Create a user avatar based on the generated user profile.

[1840] 2. Set the avatar's appearance, voice, and behavior, and prepare dialogue patterns that reflect the user's characteristics and emotional data.

[1841] 3. Adjust the avatar's movements and facial expressions based on emotional data and improve them based on user feedback.

[1842] Step 5: Setting up the dialogue system

[1843] Device:

[1844] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[1845] 2. Set up the application so that you can start interacting with the avatar.

[1846] Step 6: Running an interactive session

[1847] User:

[1848] 1. Open the dedicated application and start interacting with the avatar.

[1849] 2. Ask questions or give instructions to the avatar by voice or text.

[1850] server:

[1851] 1. Takes user input and generates an appropriate response using a generative AI model.

[1852] 2. Emotional data is also taken into account when generating responses, allowing the avatar to engage in natural dialogue with the user.

[1853] Step 7: Store and sell your data

[1854] server:

[1855] 1. Package and save the generated user profile and interaction history.

[1856] 2. Carry out procedures to sell a specific person's thought process data and emotional data.

[1857] 3. Issue an access key to the purchaser and manage the data appropriately.

[1858] Through these steps, a system is constructed that uses the user's lifestyle and emotional data to generate an avatar of the user and enable interaction. This system consistently supports the entire process, from data collection and analysis, to avatar generation, interaction, data storage, and sales.

[1859] Example 2

[1860] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1861] With conventional technology, it was difficult to effectively collect and analyze a user's lifestyle and emotional data and, based on that data, generate an avatar that could converse naturally with the user. It was also difficult to properly store user profiles and conversation histories and package them for commercial use. This resulted in the challenge of being unable to provide services that adequately meet the individual needs of users.

[1862] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1863] In this invention, the server includes means for acquiring user lifestyle data and emotional data, means for analyzing the data using a generative AI model, means for generating a profile that reproduces the user's behavioral patterns, thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile and emotional data, means for conducting a dialogue with the user using the avatar, and means for packaging and storing the user profile and dialogue history. This enables the generation of an avatar that can conduct natural dialogue according to the individual needs of the user and also facilitates commercial use of the collected data.

[1864] "Life data" refers to information related to the user's daily life, and specifically includes physical data, movement history, activity history, schedule, and the like.

[1865] "Emotional data" refers to information that indicates the user's emotional state, and includes data obtained from vocal tone, facial expressions, body language, and the like.

[1866] A "generative AI model" is an artificial intelligence model that analyzes large amounts of data and learns patterns, and has the ability to generate data generatively.

[1867] A "profile" is a collection of digital data that reproduces a user's behavioral patterns, thoughts, speech patterns, habits, etc.

[1868] An "avatar" is a digital character that mimics a user's appearance, voice, and behavior and is used to interact with the user.

[1869] "Dialogue history" refers to a record of the dialogue between the user and the avatar, and is used for later analysis and improvement.

[1870] "Packaging" refers to organizing data so that it is easy to use, and storing and managing it as a single entity.

[1871] A "server" is a machine that processes and stores data on a computer network and is the core of the system.

[1872] A "terminal" is a device that is directly operated by a user, such as a smartphone or a wearable device.

[1873] An "interface" is a means by which a user inputs data into a system and checks the output results.

[1874] This system collects and analyzes users' lifestyle and emotional data, and generates an avatar that can interact with the user based on that data. This system is primarily composed of three components: a server, a device, and the user.

[1875] Data Acquisition Embodiment

[1876] Acquiring wearable device data

[1877] Device: The user wears a wearable device such as a smartwatch or fitness tracker. This device collects real-time physical data such as heart rate, steps taken, and location. The collected data is sent to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[1878] Acquiring audio data

[1879] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[1880] Acquisition of image data

[1881] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[1882] Data Linkage

[1883] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[1884] Manual user data registration

[1885] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[1886] Acquiring and analyzing emotion data

[1887] Operating the Emotion Engine

[1888] Server: The emotion engine (e.g., Microsoft Azure Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to estimate emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis. These data are integrated with life data.

[1889] Data analysis and profile generation

[1890] Data Integration

[1891] Server: All collected data (physical data, audio data, image data, external data, manual data) will be stored in a central database. A database management system (e.g., PostgreSQL) will be used.

[1892] Profile Generation

[1893] Server: A generative AI model (e.g., GPT-3) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a detailed profile of the user.

[1894] Avatar Creation

[1895] Avatar model generation

[1896] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[1897] Avatar Optimization

[1898] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[1899] User interaction

[1900] Setting up a dialogue system

[1901] Device: A dedicated application is installed on the user's smartphone, providing an interface for interacting with the avatar.

[1902] Running an interactive session

[1903] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[1904] Dialogue generation

[1905] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[1906] Data storage and sale

[1907] Packaging Data

[1908] Server: Packages and stores the generated user profiles and interaction histories. This data is organized for commercial use.

[1909] Sales and Data Management

[1910] Server: The server processes the sale of a specific person's thought process and emotional data to interested users. It issues an access key to the purchaser and ensures proper management and security of the data.

[1911] Specific examples

[1912] Acquiring wearable device data

[1913] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[1914] Acquiring emotion data

[1915] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[1916] Data analysis and profile generation

[1917] Server: Integrates and analyzes jogging data and emotional data to learn a behavioral pattern, such as "the user jogs every morning at 7am."

[1918] Avatar Creation

[1919] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[1920] User interaction

[1921] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, the cities you visited last year were good."

[1922] Data storage and sale

[1923] Server: Packages the thought processes and emotional data of famous people and executes the sales process for potential buyers.

[1924] Prompt Sentence Examples

[1925] Please explain the specific processing steps of a system that collects a user's lifestyle and emotional data, generates an avatar based on that data, and allows the avatar to interact with the user.

[1926] In this way, the present invention provides a system that uses user data to generate and manage avatars that can naturally converse with users, and further utilizes that data commercially.

[1927] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1928] Step 1:

[1929] Data collection

[1930] Device: The user wears a wearable device (e.g., a smartwatch). The device collects real-time physical data such as heart rate, steps, and location information, and transmits it to a smartphone via Bluetooth or Wi-Fi. The smartphone periodically uploads this data to a server via the Internet.

[1931] Input: Sensor data from wearable devices (heart rate, steps, location information)

[1932] Output: Body data sent to the server

[1933] How it works: The smartwatch's sensors detect body data, and the smartphone receives the data and transfers it to a server.

[1934] Step 2:

[1935] Acquiring audio data

[1936] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[1937] Input: Audio data recorded by a smartphone microphone

[1938] Output: The audio data converted to text is sent to the server.

[1939] How it works: Your smartphone records your everyday conversations, and a voice recognition system converts the audio into text and uploads it to a server.

[1940] Step 3:

[1941] Acquisition of image data

[1942] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[1943] Input: Facial image taken with a smartphone camera and facial expression data

[1944] Output: Compressed image data is sent to the server

[1945] How it works: The smartphone camera takes a photo of your face, compresses the image data, and uploads it to a server.

[1946] Step 4:

[1947] Data Linkage

[1948] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[1949] Input: Additional data retrieved from an external data source

[1950] Output: Integrated user life data

[1951] How it works: The server accesses external data sources via APIs to retrieve additional data and integrate it with the user's life data.

[1952] Step 5:

[1953] Manual user data registration

[1954] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[1955] Input: Manual data entered by the user through a dedicated application

[1956] Output: Manual data stored on the server

[1957] How it works: The user enters schedules and notes into a dedicated application, which are then sent to the server in real time.

[1958] Step 6:

[1959] Acquiring and analyzing emotion data

[1960] Server: The emotion engine (e.g., Microsoft Azure Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to infer emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis.

[1961] Input: Voice tone, facial expression data, body language

[1962] Output: Parsed emotion data

[1963] How it works: The server analyzes the audio and image data, and the emotion engine recognizes the user's emotional state.

[1964] Step 7:

[1965] Data integration and analysis

[1966] Server: All collected data (physical data, audio data, image data, external data, manual data) is stored in a central database. A generative AI model (e.g., GPT-3) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits.

[1967] Input: Integrated data stored in a central database

[1968] Output: Parsed user profile

[1969] How it works: A server aggregates various data, and a generative AI model analyzes it to create a profile of the user.

[1970] Step 8:

[1971] Avatar model generation

[1972] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[1973] Input: User profile and emotion data

[1974] Output: Generated 3D avatar

[1975] How it works: 3D model generation software creates an avatar that mimics the user's appearance and behavior, and a voice synthesis system generates the avatar's voice.

[1976] Step 9:

[1977] Avatar Optimization

[1978] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[1979] Input: User feedback and sentiment data

[1980] Output: Optimized avatar

[1981] How it works: The server collects user feedback and uses a generative AI model to optimize the avatar's interaction patterns.

[1982] Step 10:

[1983] Setting up a dialogue system

[1984] Device: A dedicated application is installed on the user's smartphone, which provides an interface for interacting with the avatar.

[1985] Input: Dedicated application

[1986] Output: Avatar interaction interface

[1987] How it works: A dedicated application is installed on a smartphone and provides an interface for interacting with the avatar.

[1988] Step 11:

[1989] Running an interactive session

[1990] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[1991] Input: User questions and instructions

[1992] Output: Avatar response

[1993] How it works: The user opens a dedicated application and interacts with an avatar based on input questions and instructions.

[1994] Step 12:

[1995] Dialogue generation

[1996] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[1997] Input: User input and emotion data

[1998] Output: The generated response

[1999] How it works: The server uses a generative AI model to generate a response, which the avatar relays to the user.

[2000] Step 13:

[2001] Data storage and sale

[2002] Server: Packages and stores the generated user profiles and interaction histories. This data is organized for commercial use.

[2003] Input: User profile and interaction history

[2004] Output: Packaged data

[2005] How it works: The server packages and stores user profiles and interaction histories.

[2006] Sales and Data Management

[2007] Server: The server processes the sale of a specific person's thought process and emotional data to interested users. It issues an access key to the purchaser and ensures proper management and security of the data.

[2008] Input: Sales Request

[2009] Output: Access key

[2010] How it works: The server accepts sales requests and sells and manages data.

[2011] (Application example 2)

[2012] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2013] Conventional avatar generation systems are primarily limited to online use and have not contributed sufficiently to improving customer service in brick-and-mortar stores. In particular, it has been difficult to analyze individual users' lifestyle and emotional data in real time and provide personalized recommendations and support based on that data.

[2014] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2015] In this invention, the server includes means for acquiring user lifestyle data, means for analyzing the data using a generative AI model, means for generating a profile that reproduces the user's thoughts, speech patterns, and habits, means for creating an avatar for the user, means for conducting dialogue with the user, means for collecting the user's shopping history and emotion data, means for recommending optimal products and services in physical stores, and means for providing a dialogue function with an avatar that provides real-time support when the user experiences difficulty in the store. This makes it possible to provide personalized services tailored to the individual needs of users even in physical stores.

[2016] "User's lifestyle data" is information relating to the user's daily life, including, for example, location information, walking data, heart rate, shopping history, and the like.

[2017] A "generative AI model" is an artificial intelligence algorithm used to analyze large amounts of data and generate new information and predictions.

[2018] A "profile" is a collection of information created to reproduce a user's characteristics, such as their way of thinking, speaking, and habits.

[2019] An "avatar" is a virtual persona created to replicate a user's appearance, voice, and behavior.

[2020] "Emotion data" is data that indicates the user's emotional state, and is obtained from voice tone, facial expressions, body movements, and the like.

[2021] "Shopping history" refers to data on products purchased by a user in the past and related information.

[2022] A "physical store" is a sales or commercial facility that a user can physically visit.

[2023] "Recommendation" is the action or function of suggesting optimal products or services based on a user's past data and current needs.

[2024] "Providing support in real time" means providing the information and help a user needs at the exact moment they are using the application.

[2025] The "interactive function" is a function that allows users to have natural conversations with their avatars through voice and text.

[2026] This invention relates to a system that collects and analyzes a user's lifestyle data and emotional data, generates an avatar of the user based on the collected data, and enables interaction with the user. Furthermore, this system can be applied to providing personalized customer service in brick-and-mortar stores.

[2027] 1. Data Acquisition Method

[2028] Acquiring wearable device data

[2029] Device: A wearable device (such as a smartwatch or fitness tracker) collects the user's physical data (heart rate, steps, location, etc.) in real time. This data is periodically sent to a server via the internet.

[2030] Acquiring audio data

[2031] Device: A voice recording device (smartphone or dedicated recording device) records the user's everyday conversation. This voice data is converted into text by a voice recognition system and sent to a server.

[2032] Acquisition of image data

[2033] Device: A camera device (such as a smartphone camera or a dedicated camera) periodically takes a photo of the user's face and facial expression, and sends the image data to the server.

[2034] Data Linkage

[2035] Server: Acquires data from multiple external data sources (social media platforms, electronic communication tools, online payment systems) and complements and integrates user life data.

[2036] Manual user data registration

[2037] User: Through a dedicated application, users can manually input data such as schedules and notes and send it to the server.

[2038] 2. Acquisition and analysis of emotion data

[2039] Operating the Emotion Engine

[2040] Server: The emotion engine installed on the server analyzes the user's tone of voice and facial expressions, and recognizes the user's emotional state through this analysis.

[2041] Terminal: Emotion data acquired by the emotion engine is integrated with lifestyle data.

[2042] 3. Data analysis and profile generation

[2043] Data integration and analysis

[2044] Server: Stores all received data in a database and inputs it into a generative AI model. The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[2045] 4. Create your avatar

[2046] Avatar model generation

[2047] Server: Creates a user avatar based on the generated user profile and emotional data. The avatar is designed to replicate the user's appearance, voice, and behavior.

[2048] 5. User interaction implementation

[2049] Setting up a dialogue system

[2050] Terminal: Provides dedicated applications and creates an environment in which users can interact with avatars.

[2051] Running an interactive session

[2052] User: Opens a dedicated application and starts interacting with the avatar. The user can enter questions or instructions to the avatar via voice or text.

[2053] Dialogue generation

[2054] Server: Based on user input, the generative AI model generates an appropriate response, taking into account emotional data. The avatar then engages in natural dialogue with the user.

[2055] 6. Physical store applications

[2056] Collecting shopping history and sentiment data

[2057] Terminal: Collects user shopping history and emotion data through the application and sends it to the server.

[2058] Recommendation function in physical stores

[2059] Server: Analyzes data using generative AI models to recommend optimal products and services in physical stores.

[2060] Real-time support

[2061] Terminal: Provides interaction with an avatar that provides real-time support to users experiencing difficulties in the store.

[2062] Specific examples

[2063] Acquiring jogging data: When a user goes for a jog every morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[2064] Example of avatar dialogue: At 7am, the avatar asks the user, "Are you going for a jog again today? It looks like a relaxing time."

[2065] Example of interaction with an avatar in a physical store: A user arrives at a physical store and says to the avatar, "I'm looking for a recommended winter coat." The avatar recommends the best coat based on the user's past purchase history and current emotional state.

[2066] Example prompt for a generative AI model:

[2067] "When a user is looking for something in a physical store, create a conversation that recommends the best product based on their past purchase data and current heart rate data."

[2068] This allows us to make the most of user data and create a personalized shopping experience in physical stores.

[2069] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2070] Step 1:

[2071] Data collection

[2072] Device: Collects real-time data such as heart rate, steps, and location information from the user's wearable device (smartwatch or fitness tracker), as well as audio and image data from smartphones and cameras.

[2073] Input: Heart rate, steps, location, audio data, image data.

[2074] Output: The collected life data and emotion data are sent to the server.

[2075] Step 2:

[2076] Data transmission

[2077] Terminal: Sends collected lifestyle and emotional data to a server via the Internet.

[2078] Input: Life and emotion data collected on the device.

[2079] Output: Life data and emotion data sent to the server.

[2080] Step 3:

[2081] Data analysis

[2082] Server: The received lifestyle and emotion data is stored in a cloud-based data analysis system and input into the generative AI model.

[2083] Input: Submitted lifestyle and emotion data.

[2084] Output: A profile of the user's behavioral patterns and emotional state based on the analyzed data.

[2085] Step 4:

[2086] Profile Generation

[2087] Server: The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[2088] Input: Data analysis results.

[2089] Output: A profile reflecting the user's behavioral patterns and emotional state.

[2090] Step 5:

[2091] Avatar Creation

[2092] Server: Creates an avatar for the user based on the generated profile. The avatar is designed to replicate the user's appearance, voice, and behavior.

[2093] Input: The user's profile.

[2094] Output: An avatar that replicates the user's features.

[2095] Step 6:

[2096] Avatar interaction system setup

[2097] Terminal: Creates an environment where users can interact with avatars through a dedicated application.

[2098] Input: The avatar installed in the application.

[2099] Output: An interactive application.

[2100] Step 7:

[2101] Start of user interaction

[2102] User: Opens a dedicated application and starts interacting with the avatar. The user can enter questions or instructions to the avatar via voice or text.

[2103] Input: A user's voice or text question or instruction.

[2104] Output: Input data for the avatar.

[2105] Step 8:

[2106] Response Generation

[2107] Server: Uses a generative AI model based on user input and takes emotional data into account to generate an appropriate response.

[2108] Input: User questions or instructions.

[2109] Output: The response that the avatar provides to the user.

[2110] Step 9:

[2111] In-store recommendations

[2112] Server: Analyzes the user's shopping history and emotional data to recommend optimal products and services in physical stores.

[2113] Input: Shopping history and sentiment data.

[2114] Output: Recommended products or services.

[2115] Step 10:

[2116] Real-time support

[2117] Terminal: Provides an interactive function with an avatar that provides real-time support when users encounter problems in a physical store.

[2118] Input: Real-time user questions and concerns.

[2119] Output: Real-time support and guidance.

[2120] This is the flow of processing for this program. This makes it possible to utilize individual user data and provide personalized services even in physical stores.

[2121] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2122] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2123] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2124] [Fourth embodiment]

[2125] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2126] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2127] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2128] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2129] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2130] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2131] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2132] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2133] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2134] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2135] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2136] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2137] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2138] This invention relates to a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[2139] Data Acquisition Embodiment

[2140] 1. Acquiring wearable device data

[2141] Device:

[2142] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[2143] 2. Acquiring audio data

[2144] Device:

[2145] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[2146] 3. Acquisition of image data

[2147] Device:

[2148] A camera device (for example, a smartphone camera or a dedicated camera) periodically captures the user's face and facial expressions and sends them to a server as image data.

[2149] 4. Data Linkage

[2150] server:

[2151] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[2152] 5. Manual user data registration

[2153] User:

[2154] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[2155] Data analysis and profile generation

[2156] 1. Data integration and analysis

[2157] server:

[2158] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[2159] Avatar Creation

[2160] 1. Avatar model generation

[2161] server:

[2162] Based on the generated user profile, an avatar is created for the user, which is designed to replicate the user's appearance, voice, and behavior.

[2163] 2. Avatar optimization

[2164] server:

[2165] The avatar's movements and facial expressions are adjusted and optimized based on the user's characteristics.

[2166] User interaction embodiment

[2167] 1. Setting up the dialogue system

[2168] Device:

[2169] A dedicated application is provided and configured to allow users to interact with the avatar.

[2170] 2. Running an interactive session

[2171] User:

[2172] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[2173] 3. Dialogue Generation

[2174] server:

[2175] Using generative AI models, the avatar generates appropriate responses based on user input, allowing the avatar to engage in natural dialogue with the user.

[2176] Data storage and sale practices

[2177] 1. Packaging the data

[2178] server:

[2179] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[2180] 2. Sales and Data Management

[2181] server:

[2182] We will carry out procedures to sell the thought process data of a specific person to interested users. Purchasers will be issued an access key, and the data will be managed appropriately.

[2183] Specific examples

[2184] Acquiring wearable device data

[2185] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[2186] Data analysis and profile generation

[2187] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[2188] Avatar Creation

[2189] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[2190] User interaction

[2191] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[2192] Data storage and sale

[2193] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[2194] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[2195] The processing flow will be explained below.

[2196] Step 1: Get the data

[2197] Device:

[2198] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[2199] 2. A voice recording device records the user's everyday conversations.

[2200] 3. The camera device periodically captures the user's face and facial expression.

[2201] User:

[2202] 1. Manually input data such as schedules and notes through a dedicated application.

[2203] server:

[2204] 1. Receive data sent from the terminal and store it in the database.

[2205] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[2206] Step 2: Data synthesis and analysis

[2207] server:

[2208] 1. Centralize all received data and integrate it into a database.

[2209] 2. The integrated data is input into a generative AI model for analysis and learning.

[2210] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a user profile.

[2211] Step 3: Generate an avatar

[2212] server:

[2213] 1. Create a user avatar based on the generated user profile.

[2214] 2. Set the avatar's appearance, voice, and behavior to replicate the user's characteristics.

[2215] 3. Optimize the avatar's movements and facial expressions and adjust them based on user feedback.

[2216] Step 4: Setting up the dialogue system

[2217] Device:

[2218] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[2219] 2. Set up the application so that you can start interacting with the avatar.

[2220] Step 5: Running an interactive session

[2221] User:

[2222] 1. Open the dedicated application and start interacting with the avatar.

[2223] 2. Ask questions or give instructions to the avatar by voice or text.

[2224] server:

[2225] 1. Takes user input and generates an appropriate response using a generative AI model.

[2226] 2. The avatar engages in natural dialogue with the user.

[2227] Step 6: Store and sell your data

[2228] server:

[2229] 1. Package and save the generated user profile and interaction history.

[2230] 2. Proceed with the sale of a specific person's thought process data.

[2231] 3. Issue an access key to the purchaser and manage the data appropriately.

[2232] Through these steps, this system utilizes the user's daily life data to generate an avatar of the user and enable interaction.The system can also store and sell the data, enabling a wide range of uses.

[2233] Example 1

[2234] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2235] Existing interactive avatar systems lack mechanisms for centrally collecting, integrating, and analyzing user data, making it difficult to accurately reproduce the user's behavior and thoughts. Furthermore, there is a need for a system that can effectively collect, integrate, and analyze users' lifestyle data, allowing for natural interactions with individually optimized avatars and enabling commercial use of the collected data.

[2236] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2237] In this invention, the server includes means for acquiring user life data, means for packaging and selling the generated profile and interaction history, and means for integrating data acquired from external data sources. This not only enables effective collection and analysis of user life data and generation of individually optimized avatars that enable natural interactions, but also makes it possible to commercially utilize the collected data.

[2238] "User lifestyle data" refers to information related to a user's daily life, such as wearable device data, audio data, image data, and social media data.

[2239] A "generative AI model" is an artificial intelligence algorithm used to analyze collected data and learn user behavior patterns and thoughts.

[2240] A "profile" is a data structure that reproduces a user's behavior, thoughts, speech patterns, and habits analyzed by a generative AI model.

[2241] An "avatar" is a digital character created to replicate a user's appearance, voice, and behavior.

[2242] "External data sources" are other systems or databases that provide users' life data, such as social media platforms, electronic communication tools, and online payment systems.

[2243] The "dialogue history" is a record of the dialogue that took place between the user and the avatar.

[2244] "Packaging" is the process of organizing and saving the generated profiles and interaction histories as a single data set.

[2245] A "sales vehicle" is a process or system for providing packaged data to users who desire it.

[2246] This invention is a system that collects and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. Each component of this system will be explained in detail.

[2247] Data Acquisition Embodiment

[2248] Acquiring wearable device data

[2249] Device: A wearable device (such as a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is transferred to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[2250] Acquiring audio data

[2251] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recorded data is converted into text data using voice recognition software (e.g., Google Speech-to-Text API). The converted text data is sent to a server.

[2252] Acquisition of image data

[2253] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed by a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[2254] Data linkage implementation example

[2255] Server: Collects and integrates user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs), making it possible to create detailed user profiles that cannot be achieved with traditional data alone.

[2256] Embodiment of Manual User Data Registration

[2257] User: Manually inputs schedules, notes, and other data through a dedicated application. The input data is sent to the server for storage and analysis.

[2258] Data Analysis and Profile Generation Embodiments

[2259] Server: Stores all received data in a central database. It then analyzes the data using a generative AI model (e.g., GPT-4) to learn the user's behavioral patterns, thinking patterns, speech patterns, habits, etc. Based on the results of this analysis, a detailed profile of the user is generated.

[2260] Avatar Model Generation Embodiment

[2261] Server: Based on the analyzed data, a user avatar is generated, which is designed to replicate the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[2262] Avatar Optimization Embodiment

[2263] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (such as deep learning models).

[2264] User interaction embodiment

[2265] Setting up a dialogue system

[2266] Device: A dedicated application is used to set up the avatar so that the user can interact with it. The application configures the avatar's basic settings and establishes a connection with the user.

[2267] Running an interactive session

[2268] User: Opens a dedicated application and starts interacting with the avatar. The user selects voice or text input methods to send questions or instructions to the avatar.

[2269] Dialogue Generation Embodiments

[2270] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (such as GPT-4), which is then returned to the user via the avatar.

[2271] Data storage and sale practices

[2272] Packaging Data

[2273] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[2274] Sales Method

[2275] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[2276] Specific examples

[2277] Acquiring wearable device data

[2278] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[2279] Data analysis and profile generation

[2280] Server: The server analyzes this jogging data and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[2281] Avatar Creation

[2282] Server: Based on the learned behavioral patterns, the avatar is configured to ask the user at 7am, "Are you going jogging again today?"

[2283] User interaction

[2284] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[2285] Data storage and sale

[2286] Server: Packages data on the thought processes of famous authors and executes the sales process for potential buyers.

[2287] Prompt Sentence Examples

[2288] "Please tell us how to learn behavioral patterns based on daily life data collected by users and enable natural interactions with avatars through a dedicated application."

[2289] As described above, the present invention is a system that effectively collects and analyzes user data, and generates and manages avatars that can interact with users. The collected data can also be used commercially.

[2290] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2291] Step 1:

[2292] Acquiring wearable device data

[2293] Device: A wearable device (e.g., a smartwatch or fitness tracker) records the user's heart rate, steps, and location in real time. This data is then transmitted to a smartphone via Bluetooth or Wi-Fi. The smartphone then uploads this data to a server at regular intervals.

[2294] Input: User's physical data (heart rate, steps, location)

[2295] Output: Real-time body data uploaded to a server

[2296] Step 2:

[2297] Acquiring audio data

[2298] Device: A smartphone or dedicated recording device records the user's everyday conversations. The recordings are stored locally and then converted into text using speech recognition software (e.g., Google Speech-to-Text API). The text is then sent over the internet to a server.

[2299] Input: Recorded audio data

[2300] Output: Text data sent to the server

[2301] Step 3:

[2302] Acquisition of image data

[2303] Device: A smartphone camera or a dedicated camera device periodically captures the user's face and facial expressions. The captured image data is processed using a facial recognition algorithm (e.g., OpenCV) and then sent to a server.

[2304] Input: Captured image data

[2305] Output: Processed image data sent to the server

[2306] Step 4:

[2307] Data Linkage

[2308] Server: Obtains user life data from multiple external data sources (e.g., social media platform APIs, electronic communication tool APIs, online payment system APIs). This data is integrated with existing data and stored in a central database.

[2309] Input: Data retrieved from an external data source

[2310] Output: User life data integrated into a central database

[2311] Step 5:

[2312] Manual user data registration

[2313] User: Manually inputs schedules, memos, and other data using a dedicated application. This data is sent to the server via the application, where it is stored and analyzed.

[2314] Input: Data manually entered by the user (schedule, notes, etc.)

[2315] Output: Manually entered data sent to the server

[2316] Step 6:

[2317] Data integration and analysis

[2318] Server: The server stores all received data in a central database and analyzes it using a generative AI model (e.g., GPT-4), which learns the user's behavioral patterns, thoughts, speech patterns, and habits to generate a detailed user profile.

[2319] Input: All data stored in a central database (physical data, audio data, image data, external data, manually entered data)

[2320] Output: A user profile generated based on the analysis.

[2321] Step 7:

[2322] Avatar model generation

[2323] Server: Based on the generated user profile, a user avatar is generated, which reproduces the user's appearance (generated from image data), voice (generated from audio data), and behavior (generated from wearable device data and other data).

[2324] Input: Generated user profile

[2325] Output: The generated avatar

[2326] Step 8:

[2327] Avatar Optimization

[2328] Server: Adjusts and optimizes the movements and facial expressions of the created avatar based on the user's characteristics, using facial expression recognition technology and natural language processing technology (e.g., deep learning models).

[2329] Input: Generated avatar, user feature data

[2330] Output: Optimized avatar

[2331] Step 9:

[2332] Setting up a dialogue system

[2333] Device: Allows users to interact with their avatar through a dedicated application, which configures the avatar's basic settings and establishes a connection with the user.

[2334] Input: Avatar, user preference data

[2335] Output: The dialogue system is set up

[2336] Step 10:

[2337] Running an interactive session

[2338] User: Opens a dedicated application and starts interacting with the avatar. The user sends questions or instructions to the avatar via voice or text input.

[2339] Input: User questions or instructions (voice or text)

[2340] Output: Interactive session running

[2341] Step 11:

[2342] Dialogue generation

[2343] Server: Based on the user's input, the avatar generates an appropriate response using a generative AI model (e.g., GPT-4), which is then returned to the user via the avatar.

[2344] Input: User input data, generative AI model

[2345] Output: The response generated by the avatar

[2346] Step 12:

[2347] Packaging Data

[2348] Server: Packages and stores the generated user profiles and interaction histories. This data is then organized for later commercial use.

[2349] Input: Generated user profile, interaction history

[2350] Output: Packaged dataset

[2351] Step 13:

[2352] Sales and Data Management

[2353] Server: The server processes the sale of a specific person's thought process data to interested users. Purchasers are issued an access key to access the data, and the data is managed appropriately.

[2354] Input: Packaged dataset, buyer information

[2355] Output: Issued access key, managed data

[2356] (Application example 1)

[2357] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2358] Conventional personalized shopping assistant systems did not fully utilize users' lifestyle data, resulting in insufficient suggestions based on the user's preferences and behavioral patterns. Furthermore, the dialogue with the user was often unnatural, resulting in a poor user experience. Furthermore, there was no way to record the user's responses to suggested items and reflect them in the next purchase, which prevented the accuracy of suggestions from improving. This made it difficult to provide a shopping experience tailored to each individual user.

[2359] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2360] In this invention, the server includes means for acquiring user lifestyle data, means for analyzing the acquired lifestyle data using a generative AI model, means for generating a profile based on the analysis that reproduces the user's thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile, means for engaging in dialogue with the user using the avatar, means for suggesting items suitable for the user based on the user's purchasing data and preference data, means for recording the user's responses to the suggested items and reflecting them in future suggestions, means for acquiring the user's physiological data using a wearable device, and means for providing an interface for the user to manually input data. This enables more accurate and personalized shopping suggestions based on the user's lifestyle data and preferences, and a more satisfying shopping experience through natural dialogue with the user.

[2361] "Lifestyle data" refers to various information related to the user's daily life, including physical data from wearable devices, location information, audio data, image data, and activity data on social media.

[2362] A "generative AI model" is an artificial intelligence model that analyzes acquired data and learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[2363] A "profile" is a collection of information that reproduces a user's thoughts, speaking style, habits, behavioral patterns, etc., generated based on the analysis results of a generative AI model.

[2364] An "avatar" is a character that reproduces the user's appearance, voice, actions, facial expressions, etc. based on the user's profile, and is a virtual entity that can interact with the user.

[2365] "Dialogue" refers to communication between a user and an avatar via voice or text.

[2366] "Purchase data" is information about products that a user has purchased in the past, including the purchase date and time, product name, category, price, and the like.

[2367] "Preference data" is information about products and services that a user likes, and is data that indicates the user's preferences based on past behavioral history, purchase history, survey results, etc.

[2368] "User response" refers to the reaction or feedback given by the user regarding the item suggested by the avatar.

[2369] This invention relates to a system that acquires and analyzes a user's daily life data, generates an avatar of the user based on that data, and realizes a dialogue with the user. An embodiment of this system will be specifically explained below, focusing on how the server, terminal, and user cooperate to function.

[2370] Data Acquisition Embodiment

[2371] 1. Acquiring wearable device data

[2372] Device: A wearable device (e.g., a smartwatch or fitness tracker) collects a user's physical data (e.g., heart rate, steps, location, etc.) in real time. This data is periodically transmitted to a server via the Internet.

[2373] 2. Acquiring audio data

[2374] Terminal: A voice recording device (e.g., a smartphone or dedicated recording device) is used to record the user's everyday conversation. This voice data is converted into text using a voice recognition system (e.g., Google Speech-to-Text API) and sent to the server.

[2375] 3. Acquisition of image data

[2376] Terminal: A camera device (for example, a smartphone camera or a dedicated camera) is used to periodically capture the user's face and facial expressions, and the captured images are sent to the server as image data.

[2377] 4. Data Linkage

[2378] Server: Acquires data from multiple external data sources (e.g., social media platforms, online payment systems, etc.) and complements and integrates user lifestyle data.

[2379] 5. Manual user data registration

[2380] User: Manually inputs data such as schedules and notes through a dedicated application and sends it to the server.

[2381] Data analysis and profile generation

[2382] 1. Data integration and analysis

[2383] Server: Stores all received data in a database and feeds it into a generative AI model (e.g., GPT-4 or BERT). The generative AI model analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a user profile.

[2384] Avatar Creation

[2385] 1. Avatar model generation

[2386] Server: Creates a user avatar based on the generated user profile. The avatar is designed to replicate the user's appearance, voice, and behavior.

[2387] 2. Avatar optimization

[2388] Server: Adjusts and optimizes the avatar's movements and facial expressions based on the user's characteristics.

[2389] User interaction embodiment

[2390] 1. Setting up the dialogue system

[2391] Terminal: Provides dedicated applications and is configured to allow users to interact with avatars.

[2392] 2. Running an interactive session

[2393] User: Opens the dedicated application and starts interacting with the avatar. The user inputs questions and instructions to the avatar via voice or text.

[2394] 3. Dialogue Generation

[2395] Server: Using the generative AI model based on the user's input, the avatar generates an appropriate response. The avatar then engages in natural dialogue with the user.

[2396] 4. Recommendations based on purchasing and preference data

[2397] Server: Suggests suitable products and services to users based on their purchase history and preference data.

[2398] 5. Record and reflect responses to proposed items

[2399] Server: Records the user's responses to the suggested items and uses that data to improve the next suggestion.

[2400] Specific examples

[2401] If a user asks the shopping assistant, "What cosmetics should I buy next?", the avatar will suggest, based on analyzed lifestyle data and past purchasing history, "You've been buying a lot of skin care products for sensitive skin lately, so how about this new sensitive skin cream?"

[2402] Example prompts to input to the generative AI model

[2403] "Suggest the next product to buy based on the user's past shopping history and preferences. The user prefers products for sensitive skin.

[2404] Input:

[2405] Past purchases: Skin care, sensitive skin cream, facial cleanser

[2406] User's recent purchase history: Sensitive skin cream, sensitive skin lotion

[2407] User Input:

[2408] What would you recommend for my next shopping trip?”

[2409] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[2410] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2411] Step 1:

[2412] Data collection (device)

[2413] Physical data such as heart rate, number of steps, and location information are collected from the user's wearable device. In addition, a voice recording device records the user's daily conversations, and a camera device collects image data of the user's face and facial expressions. The collected data is periodically sent to a server via the Internet.

[2414] Input: Data from wearable devices, audio recording devices, and camera devices

[2415] Output: User's physical data, voice data, image data sent to the server

[2416] Step 2:

[2417] Data analysis and integration (server)

[2418] All received data is stored in a database and analyzed using a generative AI model (such as GPT-4 or BERT). Voice data is converted into text using a speech recognition system (Google Speech-to-Text API, for example). Through analysis, the system learns the user's behavioral patterns, thoughts, speaking style, habits, etc.

[2419] Input: User's physical data, voice data, image data sent to the server

[2420] Output: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[2421] Step 3:

[2422] Profile Generation (Server)

[2423] Based on the analysis results of the generative AI model, a user profile is generated, which recreates and stores the user's thoughts, speech patterns, habits, and behavioral patterns.

[2424] Input: Analyzed data on user behavior patterns, thoughts, speech patterns, and habits

[2425] Output: User profile

[2426] Step 4:

[2427] Creating an avatar (server)

[2428] Based on the generated profile, an avatar is created for the user. The avatar is designed to replicate the user's appearance, voice, and behavior. The avatar's movements, facial expressions, and other aspects are then optimized based on the user's characteristics.

[2429] Input: User profile

[2430] Output: User avatar

[2431] Step 5:

[2432] Providing interactive applications (terminals)

[2433] It provides an environment where users can interact with avatars through a dedicated application on their device. Users open the application and begin interacting with the avatar. Users' questions and instructions are entered by voice or text.

[2434] Input: User-entered voice or text data

[2435] Output: Display of interactive screen

[2436] Step 6:

[2437] Conversation creation (server)

[2438] Based on user input, a generative AI model is used to generate appropriate responses, which are then served to the user through an avatar.

[2439] Input: User's voice or text input data

[2440] Output: Natural dialogue responses from the avatar

[2441] Step 7:

[2442] Personalized Item Suggestions (Server)

[2443] Based on the user's purchasing data and preference data, the generative AI model proposes suitable products and services to the user. It analyzes past purchase history and preference data to generate the most appropriate proposals.

[2444] Input: Purchasing data, preference data

[2445] Output: Product and service recommendations to the user

[2446] Step 8:

[2447] Reflecting feedback on proposals (server)

[2448] The system records the user's responses to suggested items and updates the database to reflect them in the next suggestion, improving the accuracy of the suggestions.

[2449] Input: User response data

[2450] Output: Updated purchase data, preference data, and recommendation data

[2451] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2452] This invention relates to a system that collects and analyzes a user's lifestyle data and emotional data, generates an avatar of the user based on the data, and enables a dialogue with the user. This section explains in detail how the server, terminal, and user cooperate to function in an embodiment of this system.

[2453] Data Acquisition Embodiment

[2454] 1. Acquiring wearable device data

[2455] Device:

[2456] Wearable devices (e.g., smartwatches and fitness trackers) collect users' physical data (e.g., heart rate, steps, location, etc.) in real time, and periodically transmit this data to a server via the Internet.

[2457] 2. Acquiring audio data

[2458] Device:

[2459] A voice recording device (e.g., a smartphone or dedicated recording device) records the user's everyday conversations. This voice data is converted into text by a voice recognition system and sent to a server.

[2460] 3. Acquisition of image data

[2461] Device:

[2462] A camera device (for example, a smartphone camera or a dedicated camera) periodically takes a photo of the user's face and facial expression and sends it to a server as image data.

[2463] 4. Data Linkage

[2464] server:

[2465] It acquires data from multiple external data sources (e.g., social media platforms, electronic communication tools, online payment systems) to complement and integrate users' lifestyle data.

[2466] 5. Manual user data registration

[2467] User:

[2468] Using a dedicated application, data such as schedules and notes can be manually entered and sent to the server.

[2469] Acquiring and analyzing emotion data

[2470] 1. Operating the Emotion Engine

[2471] server:

[2472] The emotion engine installed on the server analyzes the user's voice tone, facial expressions, and body language, and through this analysis, recognizes the user's emotional state (joy, anger, sadness, happiness, etc.).

[2473] Device:

[2474] The emotion data acquired by the emotion engine is integrated with lifestyle data.

[2475] Data analysis and profile generation

[2476] 1. Data integration and analysis

[2477] server:

[2478] All received data is stored in a database and fed into a generative AI model, which analyzes the data and learns the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, a user profile is generated.

[2479] Avatar Creation

[2480] 1. Avatar model generation

[2481] server:

[2482] Based on the generated user profile and emotional data, a user avatar is created, which is designed to replicate the user's appearance, voice, and behavior.

[2483] 2. Avatar optimization

[2484] server:

[2485] Based on emotional data, the avatar's interaction patterns are adjusted and optimized, and improved based on user feedback.

[2486] User interaction embodiment

[2487] 1. Setting up the dialogue system

[2488] Device:

[2489] A dedicated application will be provided to create an environment in which users can interact with avatars.

[2490] 2. Running an interactive session

[2491] User:

[2492] The user opens a dedicated application and begins interacting with the avatar, asking questions or giving instructions to the avatar via voice or text.

[2493] 3. Dialogue Generation

[2494] server:

[2495] Based on user input, the generative AI model generates appropriate responses, taking into account emotional data, allowing the avatar to engage in natural conversation with the user.

[2496] Data storage and sale practices

[2497] 1. Packaging the data

[2498] server:

[2499] The resulting user profiles and interaction histories are packaged and stored, and this data is organized for sale.

[2500] 2. Sales and Data Management

[2501] server:

[2502] The thought process data and emotion data of a specific person will be sold to interested users. An access key will be issued to the purchaser, and the data will be managed appropriately.

[2503] Specific examples

[2504] Acquiring wearable device data

[2505] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[2506] Acquiring emotion data

[2507] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[2508] Data analysis and profile generation

[2509] Server: The server integrates and analyzes the jogging data and emotional data, and learns the behavioral pattern that "the user jogs every morning at 7 o'clock."

[2510] Avatar Creation

[2511] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[2512] User interaction

[2513] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, Kyoto, which you visited last year, would be a good choice."

[2514] Data storage and sale

[2515] Server: Packages the thought processes and emotional data of famous authors and executes the sales process for potential buyers.

[2516] In this way, the present invention provides a system that uses user data to generate and manage avatars that can interact with the users themselves, and further utilizes that data commercially.

[2517] The processing flow will be explained below.

[2518] Step 1: Get the data

[2519] Device:

[2520] 1. Wearable devices collect the user's physical data (heart rate, steps, location, etc.) in real time.

[2521] 2. A voice recording device records the user's everyday conversations.

[2522] 3. The camera device periodically captures the user's face and facial expression.

[2523] User:

[2524] 1. Manually input data such as schedules and notes through a dedicated application.

[2525] server:

[2526] 1. Receive data sent from the terminal and store it in the database.

[2527] 2. Obtain relevant data from external data sources (social media, communication tools, etc.).

[2528] Step 2: Obtaining emotion data

[2529] server:

[2530] 1. Use an emotion engine to analyze the user's tone of voice, facial expressions, and body language.

[2531] 2. Based on the analysis results, the user's emotional data (e.g., joy, anger, sadness, happiness, etc.) is recognized.

[2532] Device:

[2533] 1. Send emotion data to the server.

[2534] Step 3: Data synthesis and analysis

[2535] server:

[2536] 1. Consolidate all received data and store it in a database.

[2537] 2. The integrated data is input into a generative AI model for analysis and learning.

[2538] 3. The generative AI model learns the user's behavioral patterns, thoughts, speech patterns, habits, and emotional data to generate a user profile.

[2539] Step 4: Avatar generation and optimization

[2540] server:

[2541] 1. Create a user avatar based on the generated user profile.

[2542] 2. Set the avatar's appearance, voice, and behavior, and prepare dialogue patterns that reflect the user's characteristics and emotional data.

[2543] 3. Adjust the avatar's movements and facial expressions based on emotional data and improve them based on user feedback.

[2544] Step 5: Setting up the dialogue system

[2545] Device:

[2546] 1. Provide a dedicated application and create an environment where users can interact with avatars.

[2547] 2. Set up the application so that you can start interacting with the avatar.

[2548] Step 6: Running an interactive session

[2549] User:

[2550] 1. Open the dedicated application and start interacting with the avatar.

[2551] 2. Ask questions or give instructions to the avatar by voice or text.

[2552] server:

[2553] 1. Takes user input and generates an appropriate response using a generative AI model.

[2554] 2. Emotional data is also taken into account when generating responses, allowing the avatar to engage in natural dialogue with the user.

[2555] Step 7: Store and sell your data

[2556] server:

[2557] 1. Package and save the generated user profile and interaction history.

[2558] 2. Carry out procedures to sell a specific person's thought process data and emotional data.

[2559] 3. Issue an access key to the purchaser and manage the data appropriately.

[2560] Through these steps, a system is constructed that uses the user's lifestyle and emotional data to generate an avatar of the user and enable interaction. This system consistently supports the entire process, from data collection and analysis, to avatar generation, interaction, data storage, and sales.

[2561] Example 2

[2562] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2563] With conventional technology, it was difficult to effectively collect and analyze a user's lifestyle and emotional data and, based on that data, generate an avatar that could converse naturally with the user. It was also difficult to properly store user profiles and conversation histories and package them for commercial use. This resulted in the challenge of being unable to provide services that adequately meet the individual needs of users.

[2564] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2565] In this invention, the server includes means for acquiring user lifestyle data and emotional data, means for analyzing the data using a generative AI model, means for generating a profile that reproduces the user's behavioral patterns, thoughts, speech patterns, and habits, means for creating an avatar for the user based on the profile and emotional data, means for conducting a dialogue with the user using the avatar, and means for packaging and storing the user profile and dialogue history. This enables the generation of an avatar that can conduct natural dialogue according to the individual needs of the user and also facilitates commercial use of the collected data.

[2566] "Life data" refers to information related to the user's daily life, and specifically includes physical data, movement history, activity history, schedule, and the like.

[2567] "Emotional data" refers to information that indicates the user's emotional state, and includes data obtained from vocal tone, facial expressions, body language, and the like.

[2568] A "generative AI model" is an artificial intelligence model that analyzes large amounts of data and learns patterns, and has the ability to generate data generatively.

[2569] A "profile" is a collection of digital data that reproduces a user's behavioral patterns, thoughts, speech patterns, habits, etc.

[2570] An "avatar" is a digital character that mimics a user's appearance, voice, and behavior and is used to interact with the user.

[2571] "Dialogue history" refers to a record of the dialogue between the user and the avatar, and is used for later analysis and improvement.

[2572] "Packaging" refers to organizing data so that it is easy to use, and storing and managing it as a single entity.

[2573] A "server" is a machine that processes and stores data on a computer network and is the core of the system.

[2574] A "terminal" is a device that is directly operated by a user, such as a smartphone or a wearable device.

[2575] An "interface" is a means by which a user inputs data into a system and checks the output results.

[2576] This system collects and analyzes users' lifestyle and emotional data, and generates an avatar that can interact with the user based on that data. This system is primarily composed of three components: a server, a device, and the user.

[2577] Data Acquisition Embodiment

[2578] Acquiring wearable device data

[2579] Device: The user wears a wearable device such as a smartwatch or fitness tracker. This device collects real-time physical data such as heart rate, steps taken, and location. The collected data is sent to a smartphone via Bluetooth or Wi-Fi and then uploaded to a server via the internet.

[2580] Acquiring audio data

[2581] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[2582] Acquisition of image data

[2583] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[2584] Data Linkage

[2585] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[2586] Manual user data registration

[2587] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[2588] Acquiring and analyzing emotion data

[2589] Operating the Emotion Engine

[2590] Server: The emotion engine (e.g., Microsoft Azure Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to estimate emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis. These data are integrated with life data.

[2591] Data analysis and profile generation

[2592] Data Integration

[2593] Server: All collected data (physical data, audio data, image data, external data, manual data) will be stored in a central database. A database management system (e.g., PostgreSQL) will be used.

[2594] Profile Generation

[2595] Server: A generative AI model (e.g., GPT-3) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits. Based on this information, it generates a detailed profile of the user.

[2596] Avatar Creation

[2597] Avatar model generation

[2598] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[2599] Avatar Optimization

[2600] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[2601] User interaction

[2602] Setting up a dialogue system

[2603] Device: A dedicated application is installed on the user's smartphone, providing an interface for interacting with the avatar.

[2604] Running an interactive session

[2605] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[2606] Dialogue generation

[2607] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[2608] Data storage and sale

[2609] Packaging Data

[2610] Server: Packages and stores the generated user profiles and interaction histories. This data is organized for commercial use.

[2611] Sales and Data Management

[2612] Server: The server processes the sale of a specific person's thought process and emotional data to interested users. It issues an access key to the purchaser and ensures proper management and security of the data.

[2613] Specific examples

[2614] Acquiring wearable device data

[2615] Device: When a user goes for a jog each morning, the wearable device records the distance, time, and heart rate and sends this data to a server.

[2616] Acquiring emotion data

[2617] Server: The server's emotion engine analyzes the user's voice tone and facial expressions while jogging and determines their emotional state as "the user is relaxed while jogging."

[2618] Data analysis and profile generation

[2619] Server: Integrates and analyzes jogging data and emotional data to learn a behavioral pattern, such as "the user jogs every morning at 7am."

[2620] Avatar Creation

[2621] Server: Based on the learned data, the avatar is set to speak to the user at 7am, asking, "Are you going jogging again today? That looks like a relaxing time."

[2622] User interaction

[2623] User: Interacts with an avatar in a dedicated application and asks the avatar, "Where should we go on our next vacation?" The avatar then suggests, "Based on your past travel data, the cities you visited last year were good."

[2624] Data storage and sale

[2625] Server: Packages the thought processes and emotional data of famous people and executes the sales process for potential buyers.

[2626] Prompt Sentence Examples

[2627] Please explain the specific processing steps of a system that collects a user's lifestyle and emotional data, generates an avatar based on that data, and allows the avatar to interact with the user.

[2628] In this way, the present invention provides a system that uses user data to generate and manage avatars that can naturally converse with users, and further utilizes that data commercially.

[2629] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2630] Step 1:

[2631] Data collection

[2632] Device: The user wears a wearable device (e.g., a smartwatch). The device collects real-time physical data such as heart rate, steps, and location information, and transmits it to a smartphone via Bluetooth or Wi-Fi. The smartphone periodically uploads this data to a server via the Internet.

[2633] Input: Sensor data from wearable devices (heart rate, steps, location information)

[2634] Output: Body data sent to the server

[2635] How it works: The smartwatch's sensors detect body data, and the smartphone receives the data and transfers it to a server.

[2636] Step 2:

[2637] Acquiring audio data

[2638] Device: The smartphone microphone records the user's everyday conversations. The recorded data is converted into text using a speech recognition system (e.g., Google Speech-to-Text API) and then sent to the server.

[2639] Input: Audio data recorded by a smartphone microphone

[2640] Output: The audio data converted to text is sent to the server.

[2641] How it works: Your smartphone records your everyday conversations, and a voice recognition system converts the audio into text and uploads it to a server.

[2642] Step 3:

[2643] Acquisition of image data

[2644] Device: The smartphone camera takes photos of the user's face and facial expressions at regular intervals. The captured images are compressed and uploaded to the server.

[2645] Input: Facial image taken with a smartphone camera and facial expression data

[2646] Output: Compressed image data is sent to the server

[2647] How it works: The smartphone camera takes a photo of your face, compresses the image data, and uploads it to a server.

[2648] Step 4:

[2649] Data Linkage

[2650] Server: The server obtains additional data from external data sources (e.g., social media, electronic communication tools, online payment systems), which complements and integrates the user's life data.

[2651] Input: Additional data retrieved from an external data source

[2652] Output: Integrated user life data

[2653] How it works: The server accesses external data sources via APIs to retrieve additional data and integrate it with the user's life data.

[2654] Step 5:

[2655] Manual user data registration

[2656] User: Manually inputs schedules, notes, and other information using a dedicated application. The input data is sent to the server in real time.

[2657] Input: Manual data entered by the user through a dedicated application

[2658] Output: Manual data stored on the server

[2659] How it works: The user enters schedules and notes into a dedicated application, which are then sent to the server in real time.

[2660] Step 6:

[2661] Acquiring and analyzing emotion data

[2662] Server: The emotion engine (e.g., Microsoft Azure Emotion API) installed on the server analyzes voice tone, facial expressions, and body language. Voice tone is used to infer emotions from voice signals, and facial expression data is used to recognize emotional states through image analysis.

[2663] Input: Voice tone, facial expression data, body language

[2664] Output: Parsed emotion data

[2665] How it works: The server analyzes the audio and image data, and the emotion engine recognizes the user's emotional state.

[2666] Step 7:

[2667] Data integration and analysis

[2668] Server: All collected data (physical data, audio data, image data, external data, manual data) is stored in a central database. A generative AI model (e.g., GPT-3) takes the integrated data as input and analyzes the user's behavioral patterns, thoughts, speech patterns, and habits.

[2669] Input: Integrated data stored in a central database

[2670] Output: Parsed user profile

[2671] How it works: A server aggregates various data, and a generative AI model analyzes it to create a profile of the user.

[2672] Step 8:

[2673] Avatar model generation

[2674] Server: Based on the generated user profile and emotional data, a 3D avatar is generated that mimics the user's appearance, voice, and behavior. 3D model generation software (e.g., Blender) and a voice synthesis system (e.g., a text-to-speech engine) are used.

[2675] Input: User profile and emotion data

[2676] Output: Generated 3D avatar

[2677] How it works: 3D model generation software creates an avatar that mimics the user's appearance and behavior, and a voice synthesis system generates the avatar's voice.

[2678] Step 9:

[2679] Avatar Optimization

[2680] Server: Adjusts the avatar's interaction patterns based on emotional data. Collects user feedback and uses generative AI models to optimize the avatar's behavior and dialogue.

[2681] Input: User feedback and sentiment data

[2682] Output: Optimized avatar

[2683] How it works: The server collects user feedback and uses a generative AI model to optimize the avatar's interaction patterns.

[2684] Step 10:

[2685] Setting up a dialogue system

[2686] Device: A dedicated application is installed on the user's smartphone, which provides an interface for interacting with the avatar.

[2687] Input: Dedicated application

[2688] Output: Avatar interaction interface

[2689] How it works: A dedicated application is installed on a smartphone and provides an interface for interacting with the avatar.

[2690] Step 11:

[2691] Running an interactive session

[2692] User: Opens a dedicated application and begins interacting with the avatar. The user can enter questions or instructions by voice or text.

[2693] Input: User questions and instructions

[2694] Output: Avatar response

[2695] How it works: The user opens a dedicated application and interacts with an avatar based on input questions and instructions.

[2696] Step 12:

[2697] Dialogue generation

[2698] Server: Passes user input to the generative AI model, which takes emotional data into account to generate an appropriate response. The generated response is then conveyed to the user via an avatar.

[2699] Input: User input and emotion data

[2700] Output: The generated response

[2701] How it works: The server uses a generative AI model to generate a response, which the ...

Claims

1. A means for acquiring user lifestyle data; A means for analyzing the acquired life data using a generative AI model; A means for generating a profile that reproduces the user's thoughts, speech patterns, and habits based on the results of the analysis; means for creating an avatar of the user based on said profile; The system includes means for performing a dialogue with a user using said avatar.

2. The system of claim 1 , further comprising means for acquiring physiological data of the user using a wearable device.

3. 10. The system of claim 1, further comprising means for providing an interface for a user to manually enter data.

Images