System

Abstract

A system is provided.SOLUTION: A system for using generative AI to make predictions about a person's future, including means for collecting data from the person's devices and online accounts, means for storing the collected data in a large-scale cloud storage along a time series, means for analyzing the stored data using natural language processing and image recognition techniques, means for providing the person's future predictions and recommendations based on the analyzed data using generative AI, and interface means for managing interactions with the user and providing answers from the generative AI to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Until now, there has been no means to comprehensively manage and analyze information about an individual's past events and experiences, current situation, and future predictions. There has been a need for a system that centrally collects and analyzes information scattered across individual devices and online accounts and provides useful guidance for the future. Furthermore, there has been a lack of technology to efficiently search and display personal information, making it difficult for users to access important past information. Furthermore, future predictions based on this information have been limited, making it impossible to support individual decision-making. [Means for solving the problem]

[0005] The present invention provides a system that uses generative AI to generate information about an individual's past, present, and future, and to predict their future. In one embodiment of the present invention, the system includes: a means for collecting data from an individual's devices and online accounts; a means for storing the collected data in chronological order in large-scale cloud storage; a means for analyzing the stored data and classifying it using natural language processing and image recognition technology; a means for using generative AI to provide the individual with future predictions and advice based on the analyzed data; and an interface means for managing interactions with the user and providing the user with answers from the generative AI. This allows individuals to efficiently utilize past data and gain useful insights into the future. Furthermore, by including means for automatic data collection and search / display, user convenience can be enhanced.

[0006] "Generative AI" is a type of artificial intelligence that learns large amounts of data in advance and generates optimal answers and predictions in response to questions or instructions.

[0007] "Personal devices" refers to electronic devices owned by a user, such as smartphones, tablets, laptops, and digital cameras.

[0008] "Online account" refers to the accounts of various web services and cloud services that users use on the Internet.

[0009] "Data collection methods" refers to the programs and methods used to obtain the required information from a user's device or online account.

[0010] "Large-scale cloud storage" refers to a part of cloud services that store and manage large amounts of data via the Internet.

[0011] "Natural language processing" is an artificial intelligence technology that analyzes and understands the meaning of text data, and here refers to the technology used to analyze user input and collected text data.

[0012] "Image recognition technology" is an artificial intelligence technology that analyzes image data and understands its content, and here refers to the technology used to analyze the content of photos and videos.

[0013] "Interface means" refers to a user interface for exchanging information between a user and a system, as well as the programs and methods that operate behind the user interface.

[0014] "Means for storing data in chronological order" refers to a program or method for collecting data and storing it in a manner that maintains its order.

[0015] "Means for providing future predictions and advice" refers to programs or methods that use generative AI to predict future events or actions based on past and current data, and provide specific suggestions or advice to users.

[0016] "Scheduled Management Method" refers to a program or method for automatically collecting data from devices or online accounts on a regular basis.

[0017] "Means for search and display" refers to a program or method that allows a user to search for data based on a specific period or event and display the results. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] Overall system overview

[0040] The present invention is a system that provides past information, current information, and future predictions for an individual. This system utilizes generative AI and a large-scale cloud infrastructure to collect data from an individual's devices and online accounts, analyze the collected data, and use generative AI to provide future predictions and advice. The system includes the following main means:

[0041] Data collection methods

[0042] The terminal collects data from the user's devices and online accounts. This data includes photos, videos, text documents, audio files, and interaction history with the generating AI. The server uses the user's authentication information to access the online accounts and periodically retrieve the data. This collection process is automated and designed to minimize user interaction.

[0043] Large-scale cloud storage and time series databases

[0044] The server stores the collected data in large-scale cloud storage. The data is organized chronologically and constructed as a time-series database. This data is a detailed record of the user's past actions and thoughts and is updated regularly. Data is saved and updated automatically at regular intervals.

[0045] Data Analysis Methods

[0046] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. This analysis allows the data to be managed efficiently and made easier to search.

[0047] Generative AI for future prediction and advice

[0048] The server provides the user with future predictions and advice based on the data analyzed using the generative AI. For example, if a user requests advice on their future career path, the generative AI will take into account their past experiences and current situation to generate optimal advice. By learning from the user's past data, the generative AI can make individually customized suggestions.

[0049] Interface Means

[0050] The server provides an interface for users to interact with the generative AI. This interface is implemented as a web or mobile app, through which users can input questions or inquiries and receive answers from the generative AI. The interface is designed to allow users to visualize data and easily check past events and future predictions.

[0051] Specific examples

[0052] User questions and future predictions

[0053] For example, a case will be described where a user asks, "I want to see photos from a trip I took in the summer five years ago."

[0054] 1. Submit a question

[0055] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[0056] 2. Database Queries

[0057] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[0058] 3. Analysis of Generative AI

[0059] The server uses generative AI to analyze the captured photo data and tag it with appropriate metadata (e.g., travel destination, event name).

[0060] 4. Results display

[0061] The server returns the analysis results to the interface, where the user can view the photos.

[0062] Advice on future career paths

[0063] Next, a case where a user "seeks advice on future career paths" will be described.

[0064] 1. Submit a question

[0065] Through the interface, users type in, "I'm looking for advice on my future career path."

[0066] 2. Database Queries

[0067] The server retrieves past career-related data from a time-series database.

[0068] 3. Analysis and advice generation by generative AI

[0069] Based on the acquired data, the generative AI generates optimal career path advice, taking into account the user's skills, experience, and interests.

[0070] 4. Results display

[0071] The server returns the generated advice to the interface, where the user can view the advice.

[0072] The above is an embodiment of the present invention. This system allows users to efficiently utilize past data and gain useful insights into the future. Furthermore, by including means for automatic data collection and search / display, user convenience can be improved.

[0073] The processing flow will be explained below.

[0074] Processing steps from data collection to future prediction

[0075] Data collection

[0076] Step 1:

[0077] A user logs in through the interface and enters authentication information (e.g., username and password).

[0078] Step 2:

[0079] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[0080] Step 3:

[0081] Users register information about the devices they use and their online accounts through the interface.

[0082] Step 4:

[0083] The server receives the registration information and requests access permissions for each device and account.

[0084] Step 5:

[0085] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[0086] Step 6:

[0087] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[0088] Data storage

[0089] Step 7:

[0090] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[0091] Step 8:

[0092] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[0093] Data analysis

[0094] Step 9:

[0095] The server downloads the data stored in the cloud storage and begins analysis.

[0096] Step 10:

[0097] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[0098] Generative AI for future predictions and advice

[0099] Step 11:

[0100] Users input their questions and inquiries through the interface.

[0101] Step 12:

[0102] The server receives the user's question and passes it to the generation AI for analysis.

[0103] Step 13:

[0104] The generative AI references a time-series database and generates answers to questions based on the data obtained.

[0105] Step 14:

[0106] The server sends the generated AI's answer back to the interface.

[0107] Providing an interface

[0108] Step 15:

[0109] Through the interface, users can view past data and advice provided by the generated AI.

[0110] Step 16:

[0111] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[0112] Example: The process of searching for travel photos

[0113] Step 1:

[0114] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[0115] Step 2:

[0116] The server receives the user's query and searches the time-series database for relevant photo data.

[0117] Step 3:

[0118] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[0119] Step 4:

[0120] The server sends the analysis results back to the interface, where the user can view the photos.

[0121] Example 1

[0122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] In modern society, digital data about individuals' lives and activities is scattered across a wide range of devices and online accounts, creating a need for effective collection, storage, and analysis of this data to provide future predictions and advice tailored to individual needs.However, current systems often collect data manually, and data analysis and future predictions are not standardized, making it difficult to provide users with information that is sufficiently useful.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0125] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in a distributed storage in chronological order, means for analyzing the stored data and classifying and tagging it using natural language processing and image recognition technology, means for using artificial intelligence to provide personal future predictions and advice based on the analyzed data, and display means for managing dialogue with the user and providing answers from the artificial intelligence to the user. This integrates a series of processes from automatic data collection to analysis, future predictions, and the provision of advice, allowing users to effectively utilize their own data and obtain more useful information.

[0126] "Personal devices" refers to various types of electronic devices owned by users, including smartphones, tablets, and PCs.

[0127] "Online account" refers to various service accounts that users use on the Internet, including accounts for email, cloud storage, social media, photo sharing services, etc.

[0128] "Data" refers to information collected from personal devices and online accounts, including photos, videos, text documents, audio files, and interaction history.

[0129] "Collection Methods" refers to features or systems that automatically collect data from personal devices or online accounts. These collection methods use API calls and authentication information to obtain data.

[0130] "Means for storing data in distributed storage in chronological order" refers to a method or system for organizing collected data in chronological order and storing it in a distributed data storage system such as cloud storage.

[0131] "Natural language processing" refers to the technology of performing semantic analysis on text data to extract themes, emotions, keywords, etc. It is used to structure unstructured text data.

[0132] "Image recognition technology" refers to the technology of analyzing visual data such as photographs and videos to detect and classify specific people, objects, scenes, etc.

[0133] "Artificial intelligence" refers to machine learning algorithms and models for data analysis and future prediction, including generative AI models, which make predictions and recommendations based on analyzed data.

[0134] "Display means" refers to the interface or application that allows users to view the answers and analysis results from the generated AI. Specifically, this applies to web apps and mobile apps.

[0135] "Schedule management function" refers to a system or function that manages the time schedule for automatic and regular data collection, so that data collection can be carried out continuously at the appropriate time.

[0136] A "specific period" refers to a specific time range specified by the user, including, for example, a specific past day, week, month, or year.

[0137] "Visualization" refers to the process of displaying data in a form that is easy for a user to understand, and includes the use of charts, graphs, photographs, etc.

[0138] The present invention is a system that provides past information, current information, and future predictions for an individual. It utilizes generative AI and a distributed cloud platform to collect and analyze data from an individual's devices and online accounts, and provides future predictions and advice based on the collected data. An embodiment of this system is described in detail below.

[0139] Data collection

[0140] The device collects data from the user's various electronic devices and online accounts. This data collection is automated, for example, by periodically obtaining data using an API. A specific example is the process of downloading photos and videos from the user's cloud storage (online account). This requires OAuth authentication and obtaining an API key. The server uses these credentials to access the user's devices and online services and collect data.

[0141] Data storage

[0142] The server stores the collected data in cloud storage. Specifically, it uses a distributed storage system (e.g., Amazon S3 or Google Cloud Storage) to organize and store the data in chronological order. This allows the data to be managed as a time-series database, efficiently recording the user's past actions and events.

[0143] Data analysis

[0144] The server uses natural language processing (NLP) and image recognition technologies to analyze the stored data. Specific technologies include Google Cloud Natural Language and Amazon Comprehend for semantic analysis of text data, and AWS Rekognition and Google Cloud Vision for content analysis of images and videos. This analysis process classifies the data and assigns appropriate tags, making it easier to search and reference.

[0145] Providing future predictions and advice

[0146] The server uses a generative AI model to provide future predictions and advice to users based on the analyzed data. The system uses models such as OpenAI's GPT series and Google's BERT to learn user behavior and patterns and generate optimal future predictions and advice. For example, if a user enters a question such as "I'm looking for advice on my future career path" into the interface, the server will suggest the optimal career path taking into account past experience and current situation. An example of a prompt sentence in this case would be "I'm looking for advice on my future career path."

[0147] Results display

[0148] The server implements an interface to provide users with future predictions and advice generated by the generative AI. This interface is designed as a web or mobile app and can be built using, for example, React or Flutter. Users can enter questions through this interface and view answers from the generative AI. They can also visualize past data and check future predictions through this interface.

[0149] Specific examples

[0150] Specific examples of the present invention are shown below.

[0151] Travel photo search

[0152] For example, consider the case where a user asks, "I'd like to see photos from a trip I took in the summer five years ago."

[0153] 1. Submit a question

[0154] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[0155] 2. Database Queries

[0156] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[0157] 3. Analysis of Generative AI

[0158] The server analyzes the captured photo data and tags it with appropriate metadata (e.g., travel destination, event name).

[0159] 4. Results display

[0160] The server returns the analysis results to the interface, where the user can view the photos.

[0161] Career path advice

[0162] Let us also consider the case where a user asks, "I'm looking for advice on my future career path."

[0163] 1. Submit a question

[0164] Through the interface, users type in, "I'm looking for advice on my future career path."

[0165] 2. Database Queries

[0166] The server retrieves past career-related data from a time-series database.

[0167] 3. Analysis and advice generation by generative AI

[0168] The generative AI model uses the acquired data to generate optimal career path advice, taking into account the user's skills, experience, and interests.

[0169] 4. Results display

[0170] The server returns the generated advice to the interface, where the user can view the advice.

[0171] In this way, the present invention can effectively collect, store, and analyze user data to provide useful predictions and advice for the future.

[0172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0173] Step 1: Data collection

[0174] The server uses the user's authentication information (input) to automatically collect data from the user's device and online account. This process involves obtaining data such as photos, videos, text documents, and audio files (output) via APIs. For example, the server periodically obtains the user's photos and videos using the Google Photos API or Dropbox API. The server downloads this data and temporarily stores it in local storage.

[0175] Step 2: Save data

[0176] The server transfers the collected data to distributed cloud storage. In this process, the data is uploaded to cloud storage services such as Amazon S3 and Google Cloud Storage (input). The uploaded data (output) is organized in chronological order and recorded in a user database. This allows the collected data to be managed efficiently, making it easier to search and analyze later.

[0177] Step 3: Data analysis

[0178] The server analyzes the stored data. The input here is data read from cloud storage. The server analyzes the text data using Google Cloud Natural Language and AWS Comprehend to extract key topics and sentiment (output). It also analyzes image and video data using AWS Rekognition and Google Cloud Vision to tag specific objects and scenes (input to output). This categorizes the content of the photos and videos and stores them in a database.

[0179] Step 4: Generate future predictions and advice

[0180] The user submits a question or request for advice through the interface (input). For example, they may enter a prompt such as "I'm looking for advice on my future career path." The server retrieves the user's past data from a time-series database (input) and inputs this data into a generative AI model (e.g., GPT-4). The generative AI model generates future predictions and advice from the retrieved data (output). The output results may include, for example, specific career path suggestions or recommended skill sets.

[0181] Step 5: View the results

[0182] The server returns the generated future predictions and advice to the user's interface. The input here is the output result from the generative AI model. The user can view the displayed results through a web app or mobile app. For example, the results are displayed in an interface built using React or Flutter. This allows the user to easily visualize past data and check future predictions.

[0183] Through the above processing steps, the system of the present invention can effectively collect, store, and analyze user data, and provide future predictions and advice.

[0184] (Application example 1)

[0185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0186] In modern online shopping, it is important for users to receive product recommendations based on their past purchase and browsing history. However, current systems do not adequately provide accurate predictions or personalized product recommendations based on individual users' behavioral patterns. Therefore, there is a need for more accurate recommendation systems that can help users find products of interest and improve their satisfaction.

[0187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0188] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, interface means for managing dialogue with the user and providing the user with answers from the generation AI, means for analyzing past purchase history and browsing history to predict future purchasing trends and generate personalized product recommendations, and means for visually presenting the generated product recommendations to the user. This enables highly accurate product recommendations based on the user's individual behavioral patterns.

[0189] "Generative AI" is an artificial intelligence system that learns patterns from large datasets and makes predictions and generation.

[0190] "Personal device" refers to an information processing device owned and used by an individual, such as a smartphone, tablet, or PC.

[0191] "Online Account" refers to a collection of personal authentication information and related data for accessing services on the Internet.

[0192] "Data collection methods" refers to the mechanisms and methods used to obtain the required information from an individual's device or online account.

[0193] A "time series database" refers to a database system for organizing and storing data in chronological order.

[0194] "Large-scale cloud storage" refers to a remote server storage system for storing and managing large amounts of data via the Internet.

[0195] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[0196] "Image recognition technology" refers to the technology that allows a computer to extract, analyze, and classify information from image data.

[0197] "Interface means" refers to the means or method for exchanging information between a user and a computer system.

[0198] "Purchase history" refers to a record of products and services purchased by a user in the past.

[0199] "Browsing history" refers to a record of pages and content that a user has previously accessed on the Internet.

[0200] "Personalized product recommendations" refers to suggesting products that have specific benefits or interests to a user based on their individual purchasing history and behavioral patterns.

[0201] "Visual presentation means" refers to a method or device for visually displaying information to a user.

[0202] MODE FOR CARRYING OUT THE INVENTION

[0203] The present invention is a personalized product recommendation system using generative AI to improve user experience in online shopping. The system includes the following main means:

[0204] Data collection methods

[0205] Personal devices collect purchase and browsing history from users' devices and online accounts. This data is collected automatically and periodically and stored in cloud storage. Data collection is done using communication protocols such as external APIs.

[0206] Large-scale cloud storage and time series databases

[0207] The server stores the collected data in a time-series database in cloud storage. This organizes the data in chronological order, making it easy to refer to the user's behavioral history. Cloud storage services capable of managing large amounts of data (such as Amazon S3 or Google Cloud Storage) are used.

[0208] Data Analysis Methods

[0209] The server uses natural language processing (NLP) and image recognition technology to analyze the stored data. Purchase history and browsing history data are classified and tagged using NLP technology, which allows the interests and concerns of individual users to be clarified.

[0210] Generative AI for future prediction and advice

[0211] The server uses generative AI to predict the user's future purchasing trends based on the analyzed data. The generative AI model used is a deep learning model (e.g., Transformer or GPT model). This model learns from past data and generates individually customized product recommendations.

[0212] Interface Means

[0213] Users can interact with the generative AI through a smartphone app. This interface allows users to visually browse personalized product recommendations. The interface is user-friendly and easy to use.

[0214] Specific use cases

[0215] For example, consider a case where a user wants to predict the next product they are likely to purchase based on their past purchase history of books and electronic devices. The system operates as follows.

[0216] 1. Data collection step: The user's past purchase history and browsing history are collected from the device to cloud storage.

[0217] 2. Data storage step: The collected data is stored in cloud storage as a time series database.

[0218] 3. Data analysis step: The stored data is analyzed using NLP and image recognition technologies to clarify interests and concerns.

[0219] 4. Generative AI step: Based on the analyzed data, the generative AI model predicts the user's future purchasing trends and generates product recommendations.

[0220] 5. Interface step: The user visually browses the generated product recommendations through the smartphone app and makes a purchase decision.

[0221] Examples of prompt statements

[0222] User ID: user1234

[0223] Purchase History:

[0224] 1. Product name: Book A, Purchase date: 2021-10-01, Price: 1,500 yen

[0225] 2. Product name: Home appliance B, Purchase date: 2021-11-15, Price: 30,000 yen

[0226] 3. Product Name: Cosme C, Purchase Date: 2022-02-10, Price: 2,500 yen

[0227] question:

[0228] Predict what this user is likely to purchase this month."

[0229] The above is an embodiment of the present invention, and the system allows users to receive personalized and highly accurate product recommendations, improving their shopping experience.

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1: Data collection

[0232] The device collects the user's purchase history and browsing history. Specifically, it accesses the user's online account via API and obtains data on previously purchased products and viewed pages. The input is the user ID and authentication information, and the output is a set of purchase history and browsing history data.

[0233] Step 2: Save data

[0234] The server organizes the collected data in chronological order and stores it in large-scale cloud storage. The input is the collected purchase history and browsing history data, and the output is the data stored in cloud storage. Amazon S3 and Google Cloud Storage are used as cloud storage.

[0235] Step 3: Data analysis

[0236] The server analyzes the stored data using NLP and image recognition technology. The input is purchase history and browsing history data retrieved from cloud storage, and the output is the analyzed data (e.g., tagged product information). NLP models such as BERT are used.

[0237] Step 4: Generate future predictions

[0238] The server uses a generative AI model to predict the user's future purchasing trends from the analyzed data. The input is the analyzed data, and the output is personalized product recommendations. Transformer and GPT are used as generative AI models.

[0239] Step 5: Providing recommendations

[0240] The server sends the generated product recommendations to a smartphone app and provides them visually to the user. The input is the product recommendations generated by the generative AI model, and the output is the recommendation results displayed on the user's smartphone app. The user interface is designed so that users can easily check the product recommendations.

[0241] Specific examples

[0242] For example, consider a case where a product that a user is likely to purchase next is predicted based on the purchase history of books and electronic devices that the user has purchased in the past.

[0243] 1. Step 1:

[0244] Input: User ID and authentication information. Purchase history and browsing history are also obtained through API.

[0245] Output: A set of purchase and browsing history data.

[0246] Specific behavior: Sends an API request and receives data in JSON format.

[0247] 2. Step 2:

[0248] Input: Collected purchasing and browsing history data.

[0249] Output: Data stored in cloud storage.

[0250] Specific operation: Organize the data in chronological order and upload it to Amazon S3.

[0251] 3. Step 3:

[0252] Input: Purchase and browsing history data retrieved from cloud storage.

[0253] Output: Parsed data (e.g. tagged product information).

[0254] Specific operation: Extracts and tags product names and categories using NLP technology.

[0255] 4. Step 4:

[0256] Input: Parsed data.

[0257] Output: Personalized product recommendations.

[0258] What it does: Uses generative AI models to predict future purchasing trends and generate product lists.

[0259] 5. Step 5:

[0260] Input: Product recommendations generated by a generative AI model.

[0261] Output: Recommendation results displayed on the user's smartphone app.

[0262] What it does: Sends recommendations to the app and displays them for the user to review.

[0263] These are the specific processing steps of the system. Each step works closely together to improve the user experience, achieving personalized and highly accurate product recommendations.

[0264] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0265] Overall system overview

[0266] This invention is a system that combines a generative AI and an emotion engine to provide a person's past information, current information, and future predictions. The system collects data from a person's devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. The system also uses the emotion engine to recognize the user's emotions and adjusts the advice provided by the generative AI based on that information.

[0267] Data collection methods

[0268] The terminal collects data from the user's devices and online accounts, including photos, videos, text documents, audio files, and interaction history with the generative AI. The server accesses the online accounts based on the user's authentication information and periodically retrieves the data. The collected data is then stored in large-scale cloud storage.

[0269] Large-scale cloud storage and time series databases

[0270] The server stores the collected data in chronological order in large-scale cloud storage. The data is organized in chronological order and constructed as a time-series database. This data records the user's past actions and thoughts in detail and is updated regularly.

[0271] Data Analysis Methods

[0272] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. The results of this analysis are then fed into generative AI and emotion engines.

[0273] Emotion Engine Means

[0274] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes text and voice data to identify the user's emotional state. Changes in emotions are tracked based on the user's interaction history, and the AI ​​uses this data to improve the quality of advice.

[0275] Generative AI for future prediction and advice

[0276] The server provides future predictions and advice based on the data analyzed using the generative AI and the emotional data obtained from the emotion engine. For example, if a user requests advice on their future career path, the generative AI will consider their past experience, current situation, and emotional data to generate optimal advice.

[0277] Interface Means

[0278] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. The interface is also designed to visualize past data and future prediction results so that users can easily check them.

[0279] Specific examples

[0280] User questions and future predictions

[0281] 1. Submit a question

[0282] The user types into the interface, "I'm looking for advice on my future career path."

[0283] 2. Database Queries

[0284] The server retrieves past career-related data and emotion data from a time-series database.

[0285] 3. Analysis and advice generation by generative AI

[0286] Based on the acquired data and data from the emotion engine, the generative AI generates optimal career path advice, taking into account the user's skills, experience, interests, and emotional state.

[0287] 4. Results display

[0288] The server returns the generated advice to the interface, where the user can view the advice.

[0289] The process of searching for travel photos

[0290] 1. Submit a question

[0291] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[0292] 2. Database Queries

[0293] The server searches the time-series database and retrieves photo data that falls within the specified period.

[0294] 3. Analysis of Generative AI

[0295] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[0296] 4. Results display

[0297] The server returns the analysis results to the interface, where the user can view the photos.

[0298] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[0299] The processing flow will be explained below.

[0300] Processing steps from data collection to future prediction and emotion engine

[0301] Data collection

[0302] Step 1:

[0303] A user logs in through the interface and enters authentication information (e.g., username and password).

[0304] Step 2:

[0305] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[0306] Step 3:

[0307] Users register information about the devices they use and their online accounts through the interface.

[0308] Step 4:

[0309] The server receives the registration information and requests access permissions for each device and account.

[0310] Step 5:

[0311] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[0312] Step 6:

[0313] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[0314] Data storage

[0315] Step 7:

[0316] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[0317] Step 8:

[0318] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[0319] Data analysis

[0320] Step 9:

[0321] The server downloads the data stored in the cloud storage and begins analysis.

[0322] Step 10:

[0323] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[0324] Emotion recognition by emotion engine

[0325] Step 11:

[0326] The server acquires the user's text data and voice data and analyzes it using an emotion engine.

[0327] Step 12:

[0328] The emotion engine identifies the user's emotional state based on the captured text and voice data.

[0329] Step 13:

[0330] The emotion engine tracks changes in emotions based on the user's interaction history and provides that data to the generative AI.

[0331] Generative AI for future predictions and advice

[0332] Step 14:

[0333] Users input their questions and inquiries through the interface.

[0334] Step 15:

[0335] The server receives the user's question and passes it to the generation AI for analysis.

[0336] Step 16:

[0337] The generative AI references data provided by the time series database and emotion engine to generate answers to questions.

[0338] Step 17:

[0339] The server sends the generated AI's answer back to the interface.

[0340] Providing an interface

[0341] Step 18:

[0342] Through the interface, users can view past data and advice provided by the generated AI.

[0343] Step 19:

[0344] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[0345] Example: The process of searching for travel photos

[0346] Step 1:

[0347] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[0348] Step 2:

[0349] The server receives the user's query and searches the time-series database for relevant photo data.

[0350] Step 3:

[0351] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[0352] Step 4:

[0353] The server sends the analysis results back to the interface, where the user can view the photos.

[0354] Example: Advice on future career paths

[0355] Step 1:

[0356] The user types into the interface, "I'm looking for advice on my future career path."

[0357] Step 2:

[0358] The server retrieves past career-related data and emotion data from a time-series database.

[0359] Step 3:

[0360] The generative AI uses the acquired data and data from the emotion engine to generate optimal career path advice taking into account the user's skills, experience, interests, and emotional state.

[0361] Step 4:

[0362] The server returns the generated advice to the interface, where the user can view the advice.

[0363] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[0364] Example 2

[0365] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0366] In modern society, there is a wide range of data on individuals' lives and activities, and it is a major challenge to effectively collect and analyze this data, and to provide future predictions and appropriate advice. Furthermore, there is a demand for providing appropriate advice that takes into account the user's emotional state, but a system to achieve this has not yet been fully established.

[0367] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0368] In this invention, the server includes means for collecting data from personal communication devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for inputting the analysis results into an emotion engine and identifying the individual's emotional state, means for using a generative AI to provide future predictions and advice for the individual based on the analyzed data and emotional data, and interface means for managing dialogue with the user and providing the user with answers from the generative AI. This makes it possible to comprehensively collect and analyze a variety of user data and provide highly accurate future predictions and advice that also take the user's emotional state into consideration.

[0369] "Personal communication terminal" is a general term for devices owned by users, such as mobile phones, smartphones, tablets, and personal computers.

[0370] "Online account" is a general term for accounts for various services (social media, cloud storage, email, etc.) that users use on the Internet.

[0371] "Data collection methods" refers to technologies and processes used to automatically obtain data from personal communication devices and online accounts.

[0372] "Means for storing data in large-scale cloud storage in chronological order" refers to a method for organizing and storing collected data in a cloud storage system in chronological order.

[0373] "Large-scale cloud storage" refers to a cloud-based storage system that allows for efficient storage, management, and access of large amounts of data.

[0374] "Means of analyzing data" refers to the techniques and processes used to properly classify, tag, and index collected data.

[0375] "Natural language processing" refers to the technology of using computers to process the natural language that humans use on a daily basis.

[0376] "Image recognition technology" refers to technology that uses computer vision technology to recognize objects, patterns, characters, etc. from images and videos.

[0377] An "emotion engine" refers to technology that analyzes text and voice data to identify a user's emotional state.

[0378] "Generative AI" refers to artificial intelligence that uses machine learning and artificial intelligence techniques to generate new information and advice from data.

[0379] "Means for providing future predictions and advice" refers to technologies and processes that present future scenarios and specific advice to users based on analyzed data and emotional data.

[0380] "Interface means" refers to a user interface through which a user interacts with a system and inputs and outputs information.

[0381] "Schedule management means" refers to the techniques and processes for managing the schedule for regular data collection.

[0382] "Means for searching and displaying events and image data" refers to techniques and processes for searching for events and image data related to a specific period and visually presenting them to a user.

[0383] Overall system overview

[0384] This invention is a system that combines a generative AI and an emotion engine to provide an individual's past information, current information, and future predictions. This system collects data from an individual's communication devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. It also uses the emotion engine to recognize the user's emotions and adjusts the content of the advice provided by the generative AI based on that information.

[0385] Hardware and software used

[0386] Devices and Hardware

[0387] Personal communication devices (smartphones, tablets, computers)

[0388] Cloud servers (Amazon Web Services, Google Cloud Platform, etc.)

[0389] Software and Libraries

[0390] Data collection: API (Google Photos API, Dropbox API, etc.)

[0391] Cloud storage: Amazon S3, Google Cloud Storage

[0392] Time series databases: InfluxDB, TimescaleDB

[0393] Natural Language Processing (NLP): spaCy, NLTK

[0394] Image Recognition: TensorFlow, PyTorch

[0395] Sentiment analysis library: Affectiva, IBM Watson API

[0396] Generative AI model: GPT-3, Transformer model

[0397] Front-end frameworks: React, Vue.js

[0398] Data collection

[0399] The server collects data from users' communication devices and online accounts. For example, data collected from smartphones and PCs includes photos, videos, text documents, audio files, and conversation history with the AI. Data collected using APIs is stored in cloud storage.

[0400] Data storage and time series database construction

[0401] The collected data is stored in chronological order in cloud storage by the server, using Amazon S3 or Google Cloud Storage. The stored data is then constructed into a time-series database using InfluxDB or TimescaleDB. The data is indexed in chronological order and used for subsequent analysis and search.

[0402] Data analysis

[0403] The server analyzes the stored data using natural language processing and image recognition techniques. Text data is tokenized, tagged with parts of speech, and analyzed semantically using NLP libraries (e.g., spaCy and NLTK). Photos and videos are subjected to object and face recognition using TensorFlow and PyTorch. The results of these analyses are stored in a database and fed to the generative AI and emotion engine.

[0404] Emotion Engine

[0405] The server uses text and voice data to identify the user's emotional state. Using an emotion analysis library (such as Affectiva or IBM Watson API), it classifies the user's emotions into categories such as "happiness," "sadness," and "surprise." This emotion data is then used by the generative AI to provide advice.

[0406] Generative AI for future predictions and advice

[0407] The server uses generative AI to provide future predictions and advice based on the analyzed data and emotional data. It uses GPT-3 and Transformer models as generative AI models to generate future scenarios taking into account the user's past experiences and current situation. For example, if a user requests advice on their career path, it will suggest "what type of job would be suitable as their next career step."

[0408] Interface

[0409] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app and is built using front-end frameworks such as React or Vue.js. Users can enter questions or inquiries through the interface and receive answers from the generative AI. The interface is also designed to visualize past data and future predictions so that users can easily check them.

[0410] Specific examples

[0411] 1. Career Advice Prompts

[0412] The user types into the interface, "I'm looking for advice on my next career path."

[0413] 2. Photo search prompt

[0414] The user types into the interface, "I want to see photos from my summer trip five years ago."

[0415] Through the overall configuration and specific processing flow of this system, users can comprehensively collect and analyze a variety of data, and receive future predictions and advice that take into account their emotional state.

[0416] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0417] Step 1: Data collection

[0418] Specific operations: The device collects data from the user's communication devices and online accounts. For example, it automatically retrieves photos, videos, text documents, audio files, and interaction history with the generating AI from smartphones and PCs.

[0419] Input: User authentication and online account information

[0420] Output: Various collected data (photos, videos, text documents, audio files, conversation history)

[0421] Step 2: Save data

[0422] How it works: The server stores the collected data in chronological order in large-scale cloud storage, using Amazon S3 or Google Cloud Storage.

[0423] Input: Various collected data

[0424] Output: Data organized in chronological order is saved on cloud storage.

[0425] Step 3: Building a time series database

[0426] What it does: The server builds a time-series database from the stored data, indexing the data in chronological order using a database like InfluxDB or TimescaleDB.

[0427] Input: Time-series data stored on cloud storage

[0428] Output: Data indexed into a time series database

[0429] Step 4: Data analysis (natural language processing)

[0430] What it does: The server analyzes the stored text data using natural language processing techniques (e.g., spaCy or NLTK), including tokenization, part-of-speech tagging, and semantic analysis.

[0431] Input: Text data in a time series database

[0432] Output: Analysis results (tokenization, part-of-speech tags, semantic analysis)

[0433] Step 5: Data analysis (image recognition)

[0434] Specific operation: The server analyzes the stored photos and videos using image recognition technology (TensorFlow or PyTorch), performs object recognition and facial recognition, and saves the results as metadata.

[0435] Input: Photo and video data in a time series database

[0436] Output: Image recognition results (object recognition, face recognition, etc.)

[0437] Step 6: Sentiment Analysis

[0438] Specific operation: The server analyzes text and audio data using an emotion engine to identify the user's emotional state. It uses emotion analysis libraries (Affectiva and IBM Watson API).

[0439] Input: Natural Language Processing and Audio Data

[0440] Output: Emotion data (labels such as "happy", "sad", "surprise" etc.)

[0441] Step 7: Future prediction and advice generation

[0442] Specific operation: The server uses a generative AI model (such as GPT-3 or a Transformer model) to provide future predictions and advice based on the analyzed data and emotional data.

[0443] Input: Parsed data and sentiment data

[0444] Output: Future prediction and advice (scenario based on prompt)

[0445] Step 8: User Interface

[0446] Specific operation: The server provides an interface for users to interact with the generative AI and emotion engine. Web and mobile apps are built using front-end frameworks such as React and Vue.js. Based on user input, past data is visualized and future prediction results are displayed.

[0447] Input: Questions and inquiries from users

[0448] Output: Answers from the generative AI, visualized historical data, and future prediction results

[0449] (Application example 2)

[0450] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0451] While conventional generative AI systems often predict the future based on an individual's past and current information, they have not yet been able to provide advice that takes into account the user's emotional state or purchasing history, making it difficult to provide specific, individually customized suggestions.In addition, there has been a lack of systems that automatically make optimal suggestions based on the user's emotions and consumption trends in electronic payments and promotions.

[0452] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in large-scale cloud storage in chronological order, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, emotion analysis means for recognizing the user's emotions, analyzing the data, and reflecting the data in the advice provided by the generation AI, means for suggesting optimal payment methods and promotions based on the purchase history and emotion data collected from the personal devices and online accounts, and means for predicting future consumption trends based on the personal purchasing patterns and emotion data. This enables users to receive optimal advice that takes into account their emotional state and purchasing activities.

[0453] "Generative AI" is a technology that uses artificial intelligence to learn from data and generate new information and predictions.

[0454] "Emotion analysis" is a method of recognizing a user's emotional state from data such as text and voice, and quantitatively evaluating that emotion.

[0455] A "time series database" is a database for organizing and storing collected data in chronological order.

[0456] "Natural language processing" is a technology that allows computers to process and understand human language.

[0457] "Image recognition" is a technology for analyzing image data and recognizing its content and characteristics.

[0458] An "interface" is the means by which a user interacts with or obtains information from a system.

[0459] "Cloud storage" is a storage service for storing and managing large amounts of data via the Internet.

[0460] "Purchase history" is a record of purchases made by a user in the past.

[0461] "Promotion" is a marketing activity to promote the sale of a product or service.

[0462] "Consumption trends" refers to patterns and tendencies of users' purchasing behavior.

[0463] "Schedule management" is a management method for collecting and updating data on a regular basis.

[0464] Embodiments of the present invention will be described in detail below.

[0465] System Overview

[0466] This system combines generative AI and a sentiment analysis engine to provide information on a user's past, present, and future, and to suggest optimal payment methods and promotions for individual electronic payment services.

[0467] Data collection

[0468] The server collects data from users' devices and online accounts, including their past purchase history, spending patterns, current financial situation, and emotional state. The data is collected automatically and periodically, and stored in chronological order in large-scale cloud storage. Specific hardware used includes smartphones and cloud storage servers (e.g., AWS, Google Cloud).

[0469] Data analysis

[0470] The stored data is analyzed by the server using natural language processing (NLP) and image recognition techniques to categorize, tag, and index the data. TensorFlow is used for image recognition, and the results are fed into a generative AI and sentiment analysis engine.

[0471] Emotion analysis

[0472] The server analyzes the user's text data and dialogue history and uses an emotion analysis engine to identify their emotional state. Specifically, it uses the NLTK library to track changes in emotion, and the generated AI uses that data.

[0473] Future predictions and advice

[0474] The server provides future predictions and advice based on the data analyzed using generative AI. For example, it suggests optimal payment methods and promotions based on the user's purchasing history and emotional data. It uses a generative AI model (e.g., GPT-3) to generate advice customized for each user.

[0475] Interface

[0476] The interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. It is also designed to visualize past data and future predictions so that users can easily check them. For example, if a user inputs, "I'm worried about my recent spending. Please tell me which credit card I should use," the server will display advice generated by the generative AI based on the appropriate data.

[0477] Examples and prompts

[0478] For example, if a user types, "I'm worried about my recent spending. Can you tell me which credit card I should use?", the following prompt is sent to the generating AI:

[0479] "I'm worried about my recent spending. Which credit card should I use?"

[0480] Purchase History: All purchase data from last year

[0481] Sentiment data: Anxiety score for recent spending

[0482] Based on this, the AI ​​generates advice such as, "Since your expenses have increased recently, we recommend that you use a low-interest credit card. Also, there are currently promotions available that offer cashback and points." and provides this to the user.

[0483] The above is an embodiment of the present invention, which allows a user to receive optimal advice that takes into account their emotional state and purchasing behavior.

[0484] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0485] Step 1:

[0486] Users access the system via a smartphone or web interface and input their questions or concerns. An example of input could be text data such as, "I'm worried about my recent spending. Please tell me which credit card I should use."

[0487] Input: User question text

[0488] Output: The user's question is sent to the server

[0489] Step 2:

[0490] The server automatically collects data from users' devices and online accounts, including their purchasing history, spending patterns, and financial status, and stores the collected data in a time-series database.

[0491] Input: User credentials and online account data

[0492] Output: Purchase history and spending patterns recorded in a time series database

[0493] Step 3:

[0494] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technologies. Text data is semantically analyzed using NLP technology, and image data is analyzed using image recognition.

[0495] Input: Purchase history and spending pattern data obtained from a time series database

[0496] Output: A classified and tagged dataset

[0497] Step 4:

[0498] The server analyzes the acquired text data and dialogue history using an emotion analysis engine to identify the user's emotional state, and calculates an emotion score using the NLTK library.

[0499] Input: User text data and interaction history

[0500] Output: Sentiment score

[0501] Step 5:

[0502] The server uses a generative AI model (such as GPT-3) to generate future predictions and optimal advice based on the emotion data obtained from the emotion analysis engine and the stored purchase history data, and sends specific prompts to the generative AI.

[0503] Input: Emotion data and purchase history data

[0504] Output: Future prediction results and advice from generative AI

[0505] Step 6:

[0506] The server returns the generated advice to the user and visualizes it through an interface, allowing the user to easily browse the advice.

[0507] Input: Generated advice data

[0508] Output: Advice displayed on the interface

[0509] The above are the specific processing steps of the system that realizes the application example.

[0510] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0511] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0512] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0513] [Second embodiment]

[0514] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0515] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0516] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0517] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0518] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0519] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0520] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0521] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0522] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0523] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0524] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0525] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0526] Overall system overview

[0527] The present invention is a system that provides past information, current information, and future predictions for an individual. This system utilizes generative AI and a large-scale cloud infrastructure to collect data from an individual's devices and online accounts, analyze the collected data, and use generative AI to provide future predictions and advice. The system includes the following main means:

[0528] Data collection methods

[0529] The terminal collects data from the user's devices and online accounts. This data includes photos, videos, text documents, audio files, and interaction history with the generating AI. The server uses the user's authentication information to access the online accounts and periodically retrieve the data. This collection process is automated and designed to minimize user interaction.

[0530] Large-scale cloud storage and time series databases

[0531] The server stores the collected data in large-scale cloud storage. The data is organized chronologically and constructed as a time-series database. This data is a detailed record of the user's past actions and thoughts and is updated regularly. Data is saved and updated automatically at regular intervals.

[0532] Data Analysis Methods

[0533] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. This analysis allows the data to be managed efficiently and made easier to search.

[0534] Generative AI for future prediction and advice

[0535] The server provides the user with future predictions and advice based on the data analyzed using the generative AI. For example, if a user requests advice on their future career path, the generative AI will take into account their past experiences and current situation to generate optimal advice. By learning from the user's past data, the generative AI can make individually customized suggestions.

[0536] Interface Means

[0537] The server provides an interface for users to interact with the generative AI. This interface is implemented as a web or mobile app, through which users can input questions or inquiries and receive answers from the generative AI. The interface is designed to allow users to visualize data and easily check past events and future predictions.

[0538] Specific examples

[0539] User questions and future predictions

[0540] For example, a case will be described where a user asks, "I want to see photos from a trip I took in the summer five years ago."

[0541] 1. Submit a question

[0542] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[0543] 2. Database Queries

[0544] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[0545] 3. Analysis of Generative AI

[0546] The server uses generative AI to analyze the captured photo data and tag it with appropriate metadata (e.g., travel destination, event name).

[0547] 4. Results display

[0548] The server returns the analysis results to the interface, where the user can view the photos.

[0549] Advice on future career paths

[0550] Next, a case where a user "seeks advice on future career paths" will be described.

[0551] 1. Submit a question

[0552] Through the interface, users type in, "I'm looking for advice on my future career path."

[0553] 2. Database Queries

[0554] The server retrieves past career-related data from a time-series database.

[0555] 3. Analysis and advice generation by generative AI

[0556] Based on the acquired data, the generative AI generates optimal career path advice, taking into account the user's skills, experience, and interests.

[0557] 4. Results display

[0558] The server returns the generated advice to the interface, where the user can view the advice.

[0559] The above is an embodiment of the present invention. This system allows users to efficiently utilize past data and gain useful insights into the future. Furthermore, by including means for automatic data collection and search / display, user convenience can be improved.

[0560] The processing flow will be explained below.

[0561] Processing steps from data collection to future prediction

[0562] Data collection

[0563] Step 1:

[0564] A user logs in through the interface and enters authentication information (e.g., username and password).

[0565] Step 2:

[0566] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[0567] Step 3:

[0568] Users register information about the devices they use and their online accounts through the interface.

[0569] Step 4:

[0570] The server receives the registration information and requests access permissions for each device and account.

[0571] Step 5:

[0572] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[0573] Step 6:

[0574] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[0575] Data storage

[0576] Step 7:

[0577] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[0578] Step 8:

[0579] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[0580] Data analysis

[0581] Step 9:

[0582] The server downloads the data stored in the cloud storage and begins analysis.

[0583] Step 10:

[0584] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[0585] Generative AI for future predictions and advice

[0586] Step 11:

[0587] Users input their questions and inquiries through the interface.

[0588] Step 12:

[0589] The server receives the user's question and passes it to the generation AI for analysis.

[0590] Step 13:

[0591] The generative AI references a time-series database and generates answers to questions based on the data obtained.

[0592] Step 14:

[0593] The server sends the generated AI's answer back to the interface.

[0594] Providing an interface

[0595] Step 15:

[0596] Through the interface, users can view past data and advice provided by the generated AI.

[0597] Step 16:

[0598] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[0599] Example: The process of searching for travel photos

[0600] Step 1:

[0601] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[0602] Step 2:

[0603] The server receives the user's query and searches the time-series database for relevant photo data.

[0604] Step 3:

[0605] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[0606] Step 4:

[0607] The server sends the analysis results back to the interface, where the user can view the photos.

[0608] Example 1

[0609] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0610] In modern society, digital data about individuals' lives and activities is scattered across a wide range of devices and online accounts, creating a need for effective collection, storage, and analysis of this data to provide future predictions and advice tailored to individual needs.However, current systems often collect data manually, and data analysis and future predictions are not standardized, making it difficult to provide users with information that is sufficiently useful.

[0611] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0612] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in a distributed storage in chronological order, means for analyzing the stored data and classifying and tagging it using natural language processing and image recognition technology, means for using artificial intelligence to provide personal future predictions and advice based on the analyzed data, and display means for managing dialogue with the user and providing answers from the artificial intelligence to the user. This integrates a series of processes from automatic data collection to analysis, future predictions, and the provision of advice, allowing users to effectively utilize their own data and obtain more useful information.

[0613] "Personal devices" refers to various types of electronic devices owned by users, including smartphones, tablets, and PCs.

[0614] "Online account" refers to various service accounts that users use on the Internet, including accounts for email, cloud storage, social media, photo sharing services, etc.

[0615] "Data" refers to information collected from personal devices and online accounts, including photos, videos, text documents, audio files, and interaction history.

[0616] "Collection Methods" refers to features or systems that automatically collect data from personal devices or online accounts. These collection methods use API calls and authentication information to obtain data.

[0617] "Means for storing data in distributed storage in chronological order" refers to a method or system for organizing collected data in chronological order and storing it in a distributed data storage system such as cloud storage.

[0618] "Natural language processing" refers to the technology of performing semantic analysis on text data to extract themes, emotions, keywords, etc. It is used to structure unstructured text data.

[0619] "Image recognition technology" refers to the technology of analyzing visual data such as photographs and videos to detect and classify specific people, objects, scenes, etc.

[0620] "Artificial intelligence" refers to machine learning algorithms and models for data analysis and future prediction, including generative AI models, which make predictions and recommendations based on analyzed data.

[0621] "Display means" refers to the interface or application that allows users to view the answers and analysis results from the generated AI. Specifically, this applies to web apps and mobile apps.

[0622] "Schedule management function" refers to a system or function that manages the time schedule for automatic and regular data collection, so that data collection can be carried out continuously at the appropriate time.

[0623] A "specific period" refers to a specific time range specified by the user, including, for example, a specific past day, week, month, or year.

[0624] "Visualization" refers to the process of displaying data in a form that is easy for a user to understand, and includes the use of charts, graphs, photographs, etc.

[0625] The present invention is a system that provides past information, current information, and future predictions for an individual. It utilizes generative AI and a distributed cloud platform to collect and analyze data from an individual's devices and online accounts, and provides future predictions and advice based on the collected data. An embodiment of this system is described in detail below.

[0626] Data collection

[0627] The device collects data from the user's various electronic devices and online accounts. This data collection is automated, for example, by periodically obtaining data using an API. A specific example is the process of downloading photos and videos from the user's cloud storage (online account). This requires OAuth authentication and obtaining an API key. The server uses these credentials to access the user's devices and online services and collect data.

[0628] Data storage

[0629] The server stores the collected data in cloud storage. Specifically, it uses a distributed storage system (e.g., Amazon S3 or Google Cloud Storage) to organize and store the data in chronological order. This allows the data to be managed as a time-series database, efficiently recording the user's past actions and events.

[0630] Data analysis

[0631] The server uses natural language processing (NLP) and image recognition technologies to analyze the stored data. Specific technologies include Google Cloud Natural Language and Amazon Comprehend for semantic analysis of text data, and AWS Rekognition and Google Cloud Vision for content analysis of images and videos. This analysis process classifies the data and assigns appropriate tags, making it easier to search and reference.

[0632] Providing future predictions and advice

[0633] The server uses a generative AI model to provide future predictions and advice to users based on the analyzed data. The system uses models such as OpenAI's GPT series and Google's BERT to learn user behavior and patterns and generate optimal future predictions and advice. For example, if a user enters a question such as "I'm looking for advice on my future career path" into the interface, the server will suggest the optimal career path taking into account past experience and current situation. An example of a prompt sentence in this case would be "I'm looking for advice on my future career path."

[0634] Results display

[0635] The server implements an interface to provide users with future predictions and advice generated by the generative AI. This interface is designed as a web or mobile app and can be built using, for example, React or Flutter. Users can enter questions through this interface and view answers from the generative AI. They can also visualize past data and check future predictions through this interface.

[0636] Specific examples

[0637] Specific examples of the present invention are shown below.

[0638] Travel photo search

[0639] For example, consider the case where a user asks, "I'd like to see photos from a trip I took in the summer five years ago."

[0640] 1. Submit a question

[0641] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[0642] 2. Database Queries

[0643] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[0644] 3. Analysis of Generative AI

[0645] The server analyzes the captured photo data and tags it with appropriate metadata (e.g., travel destination, event name).

[0646] 4. Results display

[0647] The server returns the analysis results to the interface, where the user can view the photos.

[0648] Career path advice

[0649] Let us also consider the case where a user asks, "I'm looking for advice on my future career path."

[0650] 1. Submit a question

[0651] Through the interface, users type in, "I'm looking for advice on my future career path."

[0652] 2. Database Queries

[0653] The server retrieves past career-related data from a time-series database.

[0654] 3. Analysis and advice generation by generative AI

[0655] The generative AI model uses the acquired data to generate optimal career path advice, taking into account the user's skills, experience, and interests.

[0656] 4. Results display

[0657] The server returns the generated advice to the interface, where the user can view the advice.

[0658] In this way, the present invention can effectively collect, store, and analyze user data to provide useful predictions and advice for the future.

[0659] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0660] Step 1: Data collection

[0661] The server uses the user's authentication information (input) to automatically collect data from the user's device and online account. This process involves obtaining data such as photos, videos, text documents, and audio files (output) via APIs. For example, the server periodically obtains the user's photos and videos using the Google Photos API or Dropbox API. The server downloads this data and temporarily stores it in local storage.

[0662] Step 2: Save data

[0663] The server transfers the collected data to distributed cloud storage. In this process, the data is uploaded to cloud storage services such as Amazon S3 and Google Cloud Storage (input). The uploaded data (output) is organized in chronological order and recorded in a user database. This allows the collected data to be managed efficiently, making it easier to search and analyze later.

[0664] Step 3: Data analysis

[0665] The server analyzes the stored data. The input here is data read from cloud storage. The server analyzes the text data using Google Cloud Natural Language and AWS Comprehend to extract key topics and sentiment (output). It also analyzes image and video data using AWS Rekognition and Google Cloud Vision to tag specific objects and scenes (input to output). This categorizes the content of the photos and videos and stores them in a database.

[0666] Step 4: Generate future predictions and advice

[0667] The user submits a question or request for advice through the interface (input). For example, they may enter a prompt such as "I'm looking for advice on my future career path." The server retrieves the user's past data from a time-series database (input) and inputs this data into a generative AI model (e.g., GPT-4). The generative AI model generates future predictions and advice from the retrieved data (output). The output results may include, for example, specific career path suggestions or recommended skill sets.

[0668] Step 5: View the results

[0669] The server returns the generated future predictions and advice to the user's interface. The input here is the output result from the generative AI model. The user can view the displayed results through a web app or mobile app. For example, the results are displayed in an interface built using React or Flutter. This allows the user to easily visualize past data and check future predictions.

[0670] Through the above processing steps, the system of the present invention can effectively collect, store, and analyze user data, and provide future predictions and advice.

[0671] (Application example 1)

[0672] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0673] In modern online shopping, it is important for users to receive product recommendations based on their past purchase and browsing history. However, current systems do not adequately provide accurate predictions or personalized product recommendations based on individual users' behavioral patterns. Therefore, there is a need for more accurate recommendation systems that can help users find products of interest and improve their satisfaction.

[0674] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0675] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, interface means for managing dialogue with the user and providing the user with answers from the generation AI, means for analyzing past purchase history and browsing history to predict future purchasing trends and generate personalized product recommendations, and means for visually presenting the generated product recommendations to the user. This enables highly accurate product recommendations based on the user's individual behavioral patterns.

[0676] "Generative AI" is an artificial intelligence system that learns patterns from large datasets and makes predictions and generation.

[0677] "Personal device" refers to an information processing device owned and used by an individual, such as a smartphone, tablet, or PC.

[0678] "Online Account" refers to a collection of personal authentication information and related data for accessing services on the Internet.

[0679] "Data collection methods" refers to the mechanisms and methods used to obtain the required information from an individual's device or online account.

[0680] A "time series database" refers to a database system for organizing and storing data in chronological order.

[0681] "Large-scale cloud storage" refers to a remote server storage system for storing and managing large amounts of data via the Internet.

[0682] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[0683] "Image recognition technology" refers to the technology that allows a computer to extract, analyze, and classify information from image data.

[0684] "Interface means" refers to the means or method for exchanging information between a user and a computer system.

[0685] "Purchase history" refers to a record of products and services purchased by a user in the past.

[0686] "Browsing history" refers to a record of pages and content that a user has previously accessed on the Internet.

[0687] "Personalized product recommendations" refers to suggesting products that have specific benefits or interests to a user based on their individual purchasing history and behavioral patterns.

[0688] "Visual presentation means" refers to a method or device for visually displaying information to a user.

[0689] MODE FOR CARRYING OUT THE INVENTION

[0690] The present invention is a personalized product recommendation system using generative AI to improve user experience in online shopping. The system includes the following main means:

[0691] Data collection methods

[0692] Personal devices collect purchase and browsing history from users' devices and online accounts. This data is collected automatically and periodically and stored in cloud storage. Data collection is done using communication protocols such as external APIs.

[0693] Large-scale cloud storage and time series databases

[0694] The server stores the collected data in a time-series database in cloud storage. This organizes the data in chronological order, making it easy to refer to the user's behavioral history. Cloud storage services capable of managing large amounts of data (such as Amazon S3 or Google Cloud Storage) are used.

[0695] Data Analysis Methods

[0696] The server uses natural language processing (NLP) and image recognition technology to analyze the stored data. Purchase history and browsing history data are classified and tagged using NLP technology, which allows the interests and concerns of individual users to be clarified.

[0697] Generative AI for future prediction and advice

[0698] The server uses generative AI to predict the user's future purchasing trends based on the analyzed data. The generative AI model used is a deep learning model (e.g., Transformer or GPT model). This model learns from past data and generates individually customized product recommendations.

[0699] Interface Means

[0700] Users can interact with the generative AI through a smartphone app. This interface allows users to visually browse personalized product recommendations. The interface is user-friendly and easy to use.

[0701] Specific use cases

[0702] For example, consider a case where a user wants to predict the next product they are likely to purchase based on their past purchase history of books and electronic devices. The system operates as follows.

[0703] 1. Data collection step: The user's past purchase history and browsing history are collected from the device to cloud storage.

[0704] 2. Data storage step: The collected data is stored in cloud storage as a time series database.

[0705] 3. Data analysis step: The stored data is analyzed using NLP and image recognition technologies to clarify interests and concerns.

[0706] 4. Generative AI step: Based on the analyzed data, the generative AI model predicts the user's future purchasing trends and generates product recommendations.

[0707] 5. Interface step: The user visually browses the generated product recommendations through the smartphone app and makes a purchase decision.

[0708] Examples of prompt statements

[0709] User ID: user1234

[0710] Purchase History:

[0711] 1. Product name: Book A, Purchase date: 2021-10-01, Price: 1,500 yen

[0712] 2. Product name: Home appliance B, Purchase date: 2021-11-15, Price: 30,000 yen

[0713] 3. Product Name: Cosme C, Purchase Date: 2022-02-10, Price: 2,500 yen

[0714] question:

[0715] Predict what this user is likely to purchase this month."

[0716] The above is an embodiment of the present invention, and the system allows users to receive personalized and highly accurate product recommendations, improving their shopping experience.

[0717] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0718] Step 1: Data collection

[0719] The device collects the user's purchase history and browsing history. Specifically, it accesses the user's online account via API and obtains data on previously purchased products and viewed pages. The input is the user ID and authentication information, and the output is a set of purchase history and browsing history data.

[0720] Step 2: Save data

[0721] The server organizes the collected data in chronological order and stores it in large-scale cloud storage. The input is the collected purchase history and browsing history data, and the output is the data stored in cloud storage. Amazon S3 and Google Cloud Storage are used as cloud storage.

[0722] Step 3: Data analysis

[0723] The server analyzes the stored data using NLP and image recognition technology. The input is purchase history and browsing history data retrieved from cloud storage, and the output is the analyzed data (e.g., tagged product information). NLP models such as BERT are used.

[0724] Step 4: Generate future predictions

[0725] The server uses a generative AI model to predict the user's future purchasing trends from the analyzed data. The input is the analyzed data, and the output is personalized product recommendations. Transformer and GPT are used as generative AI models.

[0726] Step 5: Providing recommendations

[0727] The server sends the generated product recommendations to a smartphone app and provides them visually to the user. The input is the product recommendations generated by the generative AI model, and the output is the recommendation results displayed on the user's smartphone app. The user interface is designed so that users can easily check the product recommendations.

[0728] Specific examples

[0729] For example, consider a case where a product that a user is likely to purchase next is predicted based on the purchase history of books and electronic devices that the user has purchased in the past.

[0730] 1. Step 1:

[0731] Input: User ID and authentication information. Purchase history and browsing history are also obtained through API.

[0732] Output: A set of purchase and browsing history data.

[0733] Specific behavior: Sends an API request and receives data in JSON format.

[0734] 2. Step 2:

[0735] Input: Collected purchasing and browsing history data.

[0736] Output: Data stored in cloud storage.

[0737] Specific operation: Organize the data in chronological order and upload it to Amazon S3.

[0738] 3. Step 3:

[0739] Input: Purchase and browsing history data retrieved from cloud storage.

[0740] Output: Parsed data (e.g. tagged product information).

[0741] Specific operation: Extracts and tags product names and categories using NLP technology.

[0742] 4. Step 4:

[0743] Input: Parsed data.

[0744] Output: Personalized product recommendations.

[0745] What it does: Uses generative AI models to predict future purchasing trends and generate product lists.

[0746] 5. Step 5:

[0747] Input: Product recommendations generated by a generative AI model.

[0748] Output: Recommendation results displayed on the user's smartphone app.

[0749] What it does: Sends recommendations to the app and displays them for the user to review.

[0750] These are the specific processing steps of the system. Each step works closely together to improve the user experience, achieving personalized and highly accurate product recommendations.

[0751] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0752] Overall system overview

[0753] This invention is a system that combines a generative AI and an emotion engine to provide a person's past information, current information, and future predictions. The system collects data from a person's devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. The system also uses the emotion engine to recognize the user's emotions and adjusts the advice provided by the generative AI based on that information.

[0754] Data collection methods

[0755] The terminal collects data from the user's devices and online accounts, including photos, videos, text documents, audio files, and interaction history with the generative AI. The server accesses the online accounts based on the user's authentication information and periodically retrieves the data. The collected data is then stored in large-scale cloud storage.

[0756] Large-scale cloud storage and time series databases

[0757] The server stores the collected data in chronological order in large-scale cloud storage. The data is organized in chronological order and constructed as a time-series database. This data records the user's past actions and thoughts in detail and is updated regularly.

[0758] Data Analysis Methods

[0759] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. The results of this analysis are then fed into generative AI and emotion engines.

[0760] Emotion Engine Means

[0761] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes text and voice data to identify the user's emotional state. Changes in emotions are tracked based on the user's interaction history, and the AI ​​uses this data to improve the quality of advice.

[0762] Generative AI for future prediction and advice

[0763] The server provides future predictions and advice based on the data analyzed using the generative AI and the emotional data obtained from the emotion engine. For example, if a user requests advice on their future career path, the generative AI will consider their past experience, current situation, and emotional data to generate optimal advice.

[0764] Interface Means

[0765] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. The interface is also designed to visualize past data and future prediction results so that users can easily check them.

[0766] Specific examples

[0767] User questions and future predictions

[0768] 1. Submit a question

[0769] The user types into the interface, "I'm looking for advice on my future career path."

[0770] 2. Database Queries

[0771] The server retrieves past career-related data and emotion data from a time-series database.

[0772] 3. Analysis and advice generation by generative AI

[0773] Based on the acquired data and data from the emotion engine, the generative AI generates optimal career path advice, taking into account the user's skills, experience, interests, and emotional state.

[0774] 4. Results display

[0775] The server returns the generated advice to the interface, where the user can view the advice.

[0776] The process of searching for travel photos

[0777] 1. Submit a question

[0778] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[0779] 2. Database Queries

[0780] The server searches the time-series database and retrieves photo data that falls within the specified period.

[0781] 3. Analysis of Generative AI

[0782] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[0783] 4. Results display

[0784] The server returns the analysis results to the interface, where the user can view the photos.

[0785] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[0786] The processing flow will be explained below.

[0787] Processing steps from data collection to future prediction and emotion engine

[0788] Data collection

[0789] Step 1:

[0790] A user logs in through the interface and enters authentication information (e.g., username and password).

[0791] Step 2:

[0792] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[0793] Step 3:

[0794] Users register information about the devices they use and their online accounts through the interface.

[0795] Step 4:

[0796] The server receives the registration information and requests access permissions for each device and account.

[0797] Step 5:

[0798] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[0799] Step 6:

[0800] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[0801] Data storage

[0802] Step 7:

[0803] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[0804] Step 8:

[0805] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[0806] Data analysis

[0807] Step 9:

[0808] The server downloads the data stored in the cloud storage and begins analysis.

[0809] Step 10:

[0810] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[0811] Emotion recognition by emotion engine

[0812] Step 11:

[0813] The server acquires the user's text data and voice data and analyzes it using an emotion engine.

[0814] Step 12:

[0815] The emotion engine identifies the user's emotional state based on the captured text and voice data.

[0816] Step 13:

[0817] The emotion engine tracks changes in emotions based on the user's interaction history and provides that data to the generative AI.

[0818] Generative AI for future predictions and advice

[0819] Step 14:

[0820] Users input their questions and inquiries through the interface.

[0821] Step 15:

[0822] The server receives the user's question and passes it to the generation AI for analysis.

[0823] Step 16:

[0824] The generative AI references data provided by the time series database and emotion engine to generate answers to questions.

[0825] Step 17:

[0826] The server sends the generated AI's answer back to the interface.

[0827] Providing an interface

[0828] Step 18:

[0829] Through the interface, users can view past data and advice provided by the generated AI.

[0830] Step 19:

[0831] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[0832] Example: The process of searching for travel photos

[0833] Step 1:

[0834] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[0835] Step 2:

[0836] The server receives the user's query and searches the time-series database for relevant photo data.

[0837] Step 3:

[0838] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[0839] Step 4:

[0840] The server sends the analysis results back to the interface, where the user can view the photos.

[0841] Example: Advice on future career paths

[0842] Step 1:

[0843] The user types into the interface, "I'm looking for advice on my future career path."

[0844] Step 2:

[0845] The server retrieves past career-related data and emotion data from a time-series database.

[0846] Step 3:

[0847] The generative AI uses the acquired data and data from the emotion engine to generate optimal career path advice taking into account the user's skills, experience, interests, and emotional state.

[0848] Step 4:

[0849] The server returns the generated advice to the interface, where the user can view the advice.

[0850] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[0851] Example 2

[0852] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0853] In modern society, there is a wide range of data on individuals' lives and activities, and it is a major challenge to effectively collect and analyze this data, and to provide future predictions and appropriate advice. Furthermore, there is a demand for providing appropriate advice that takes into account the user's emotional state, but a system to achieve this has not yet been fully established.

[0854] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0855] In this invention, the server includes means for collecting data from personal communication devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for inputting the analysis results into an emotion engine and identifying the individual's emotional state, means for using a generative AI to provide future predictions and advice for the individual based on the analyzed data and emotional data, and interface means for managing dialogue with the user and providing the user with answers from the generative AI. This makes it possible to comprehensively collect and analyze a variety of user data and provide highly accurate future predictions and advice that also take the user's emotional state into consideration.

[0856] "Personal communication terminal" is a general term for devices owned by users, such as mobile phones, smartphones, tablets, and personal computers.

[0857] "Online account" is a general term for accounts for various services (social media, cloud storage, email, etc.) that users use on the Internet.

[0858] "Data collection methods" refers to technologies and processes used to automatically obtain data from personal communication devices and online accounts.

[0859] "Means for storing data in large-scale cloud storage in chronological order" refers to a method for organizing and storing collected data in a cloud storage system in chronological order.

[0860] "Large-scale cloud storage" refers to a cloud-based storage system that allows for efficient storage, management, and access of large amounts of data.

[0861] "Means of analyzing data" refers to the techniques and processes used to properly classify, tag, and index collected data.

[0862] "Natural language processing" refers to the technology of using computers to process the natural language that humans use on a daily basis.

[0863] "Image recognition technology" refers to technology that uses computer vision technology to recognize objects, patterns, characters, etc. from images and videos.

[0864] An "emotion engine" refers to technology that analyzes text and voice data to identify a user's emotional state.

[0865] "Generative AI" refers to artificial intelligence that uses machine learning and artificial intelligence techniques to generate new information and advice from data.

[0866] "Means for providing future predictions and advice" refers to technologies and processes that present future scenarios and specific advice to users based on analyzed data and emotional data.

[0867] "Interface means" refers to a user interface through which a user interacts with a system and inputs and outputs information.

[0868] "Schedule management means" refers to the techniques and processes for managing the schedule for regular data collection.

[0869] "Means for searching and displaying events and image data" refers to techniques and processes for searching for events and image data related to a specific period and visually presenting them to a user.

[0870] Overall system overview

[0871] This invention is a system that combines a generative AI and an emotion engine to provide an individual's past information, current information, and future predictions. This system collects data from an individual's communication devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. It also uses the emotion engine to recognize the user's emotions and adjusts the content of the advice provided by the generative AI based on that information.

[0872] Hardware and software used

[0873] Devices and Hardware

[0874] Personal communication devices (smartphones, tablets, computers)

[0875] Cloud servers (Amazon Web Services, Google Cloud Platform, etc.)

[0876] Software and Libraries

[0877] Data collection: API (Google Photos API, Dropbox API, etc.)

[0878] Cloud storage: Amazon S3, Google Cloud Storage

[0879] Time series databases: InfluxDB, TimescaleDB

[0880] Natural Language Processing (NLP): spaCy, NLTK

[0881] Image Recognition: TensorFlow, PyTorch

[0882] Sentiment analysis library: Affectiva, IBM Watson API

[0883] Generative AI model: GPT-3, Transformer model

[0884] Front-end frameworks: React, Vue.js

[0885] Data collection

[0886] The server collects data from users' communication devices and online accounts. For example, data collected from smartphones and PCs includes photos, videos, text documents, audio files, and conversation history with the AI. Data collected using APIs is stored in cloud storage.

[0887] Data storage and time series database construction

[0888] The collected data is stored in chronological order in cloud storage by the server, using Amazon S3 or Google Cloud Storage. The stored data is then constructed into a time-series database using InfluxDB or TimescaleDB. The data is indexed in chronological order and used for subsequent analysis and search.

[0889] Data analysis

[0890] The server analyzes the stored data using natural language processing and image recognition techniques. Text data is tokenized, tagged with parts of speech, and analyzed semantically using NLP libraries (e.g., spaCy and NLTK). Photos and videos are subjected to object and face recognition using TensorFlow and PyTorch. The results of these analyses are stored in a database and fed to the generative AI and emotion engine.

[0891] Emotion Engine

[0892] The server uses text and voice data to identify the user's emotional state. Using an emotion analysis library (such as Affectiva or IBM Watson API), it classifies the user's emotions into categories such as "happiness," "sadness," and "surprise." This emotion data is then used by the generative AI to provide advice.

[0893] Generative AI for future predictions and advice

[0894] The server uses generative AI to provide future predictions and advice based on the analyzed data and emotional data. It uses GPT-3 and Transformer models as generative AI models to generate future scenarios taking into account the user's past experiences and current situation. For example, if a user requests advice on their career path, it will suggest "what type of job would be suitable as their next career step."

[0895] Interface

[0896] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app and is built using front-end frameworks such as React or Vue.js. Users can enter questions or inquiries through the interface and receive answers from the generative AI. The interface is also designed to visualize past data and future predictions so that users can easily check them.

[0897] Specific examples

[0898] 1. Career Advice Prompts

[0899] The user types into the interface, "I'm looking for advice on my next career path."

[0900] 2. Photo search prompt

[0901] The user types into the interface, "I want to see photos from my summer trip five years ago."

[0902] Through the overall configuration and specific processing flow of this system, users can comprehensively collect and analyze a variety of data, and receive future predictions and advice that take into account their emotional state.

[0903] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0904] Step 1: Data collection

[0905] Specific operations: The device collects data from the user's communication devices and online accounts. For example, it automatically retrieves photos, videos, text documents, audio files, and interaction history with the generating AI from smartphones and PCs.

[0906] Input: User authentication and online account information

[0907] Output: Various collected data (photos, videos, text documents, audio files, conversation history)

[0908] Step 2: Save data

[0909] How it works: The server stores the collected data in chronological order in large-scale cloud storage, using Amazon S3 or Google Cloud Storage.

[0910] Input: Various collected data

[0911] Output: Data organized in chronological order is saved on cloud storage.

[0912] Step 3: Building a time series database

[0913] What it does: The server builds a time-series database from the stored data, indexing the data in chronological order using a database like InfluxDB or TimescaleDB.

[0914] Input: Time-series data stored on cloud storage

[0915] Output: Data indexed into a time series database

[0916] Step 4: Data analysis (natural language processing)

[0917] What it does: The server analyzes the stored text data using natural language processing techniques (e.g., spaCy or NLTK), including tokenization, part-of-speech tagging, and semantic analysis.

[0918] Input: Text data in a time series database

[0919] Output: Analysis results (tokenization, part-of-speech tags, semantic analysis)

[0920] Step 5: Data analysis (image recognition)

[0921] Specific operation: The server analyzes the stored photos and videos using image recognition technology (TensorFlow or PyTorch), performs object recognition and facial recognition, and saves the results as metadata.

[0922] Input: Photo and video data in a time series database

[0923] Output: Image recognition results (object recognition, face recognition, etc.)

[0924] Step 6: Sentiment Analysis

[0925] Specific operation: The server analyzes text and audio data using an emotion engine to identify the user's emotional state. It uses emotion analysis libraries (Affectiva and IBM Watson API).

[0926] Input: Natural Language Processing and Audio Data

[0927] Output: Emotion data (labels such as "happy", "sad", "surprise" etc.)

[0928] Step 7: Future prediction and advice generation

[0929] Specific operation: The server uses a generative AI model (such as GPT-3 or a Transformer model) to provide future predictions and advice based on the analyzed data and emotional data.

[0930] Input: Parsed data and sentiment data

[0931] Output: Future prediction and advice (scenario based on prompt)

[0932] Step 8: User Interface

[0933] Specific operation: The server provides an interface for users to interact with the generative AI and emotion engine. Web and mobile apps are built using front-end frameworks such as React and Vue.js. Based on user input, past data is visualized and future prediction results are displayed.

[0934] Input: Questions and inquiries from users

[0935] Output: Answers from the generative AI, visualized historical data, and future prediction results

[0936] (Application example 2)

[0937] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0938] While conventional generative AI systems often predict the future based on an individual's past and current information, they have not yet been able to provide advice that takes into account the user's emotional state or purchasing history, making it difficult to provide specific, individually customized suggestions.In addition, there has been a lack of systems that automatically make optimal suggestions based on the user's emotions and consumption trends in electronic payments and promotions.

[0939] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in large-scale cloud storage in chronological order, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, emotion analysis means for recognizing the user's emotions, analyzing the data, and reflecting the data in the advice provided by the generation AI, means for suggesting optimal payment methods and promotions based on the purchase history and emotion data collected from the personal devices and online accounts, and means for predicting future consumption trends based on the personal purchasing patterns and emotion data. This enables users to receive optimal advice that takes into account their emotional state and purchasing activities.

[0940] "Generative AI" is a technology that uses artificial intelligence to learn from data and generate new information and predictions.

[0941] "Emotion analysis" is a method of recognizing a user's emotional state from data such as text and voice, and quantitatively evaluating that emotion.

[0942] A "time series database" is a database for organizing and storing collected data in chronological order.

[0943] "Natural language processing" is a technology that allows computers to process and understand human language.

[0944] "Image recognition" is a technology for analyzing image data and recognizing its content and characteristics.

[0945] An "interface" is the means by which a user interacts with or obtains information from a system.

[0946] "Cloud storage" is a storage service for storing and managing large amounts of data via the Internet.

[0947] "Purchase history" is a record of purchases made by a user in the past.

[0948] "Promotion" is a marketing activity to promote the sale of a product or service.

[0949] "Consumption trends" refers to patterns and tendencies of users' purchasing behavior.

[0950] "Schedule management" is a management method for collecting and updating data on a regular basis.

[0951] Embodiments of the present invention will be described in detail below.

[0952] System Overview

[0953] This system combines generative AI and a sentiment analysis engine to provide information on a user's past, present, and future, and to suggest optimal payment methods and promotions for individual electronic payment services.

[0954] Data collection

[0955] The server collects data from users' devices and online accounts, including their past purchase history, spending patterns, current financial situation, and emotional state. The data is collected automatically and periodically, and stored in chronological order in large-scale cloud storage. Specific hardware used includes smartphones and cloud storage servers (e.g., AWS, Google Cloud).

[0956] Data analysis

[0957] The stored data is analyzed by the server using natural language processing (NLP) and image recognition techniques to categorize, tag, and index the data. TensorFlow is used for image recognition, and the results are fed into a generative AI and sentiment analysis engine.

[0958] Emotion analysis

[0959] The server analyzes the user's text data and dialogue history and uses an emotion analysis engine to identify their emotional state. Specifically, it uses the NLTK library to track changes in emotion, and the generated AI uses that data.

[0960] Future predictions and advice

[0961] The server provides future predictions and advice based on the data analyzed using generative AI. For example, it suggests optimal payment methods and promotions based on the user's purchasing history and emotional data. It uses a generative AI model (e.g., GPT-3) to generate advice customized for each user.

[0962] Interface

[0963] The interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. It is also designed to visualize past data and future predictions so that users can easily check them. For example, if a user inputs, "I'm worried about my recent spending. Please tell me which credit card I should use," the server will display advice generated by the generative AI based on the appropriate data.

[0964] Examples and prompts

[0965] For example, if a user types, "I'm worried about my recent spending. Can you tell me which credit card I should use?", the following prompt is sent to the generating AI:

[0966] "I'm worried about my recent spending. Which credit card should I use?"

[0967] Purchase History: All purchase data from last year

[0968] Sentiment data: Anxiety score for recent spending

[0969] Based on this, the AI ​​generates advice such as, "Since your expenses have increased recently, we recommend that you use a low-interest credit card. Also, there are currently promotions available that offer cashback and points." and provides this to the user.

[0970] The above is an embodiment of the present invention, which allows a user to receive optimal advice that takes into account their emotional state and purchasing behavior.

[0971] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0972] Step 1:

[0973] Users access the system via a smartphone or web interface and input their questions or concerns. An example of input could be text data such as, "I'm worried about my recent spending. Please tell me which credit card I should use."

[0974] Input: User question text

[0975] Output: The user's question is sent to the server

[0976] Step 2:

[0977] The server automatically collects data from users' devices and online accounts, including their purchasing history, spending patterns, and financial status, and stores the collected data in a time-series database.

[0978] Input: User credentials and online account data

[0979] Output: Purchase history and spending patterns recorded in a time series database

[0980] Step 3:

[0981] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technologies. Text data is semantically analyzed using NLP technology, and image data is analyzed using image recognition.

[0982] Input: Purchase history and spending pattern data obtained from a time series database

[0983] Output: A classified and tagged dataset

[0984] Step 4:

[0985] The server analyzes the acquired text data and dialogue history using an emotion analysis engine to identify the user's emotional state, and calculates an emotion score using the NLTK library.

[0986] Input: User text data and interaction history

[0987] Output: Sentiment score

[0988] Step 5:

[0989] The server uses a generative AI model (such as GPT-3) to generate future predictions and optimal advice based on the emotion data obtained from the emotion analysis engine and the stored purchase history data, and sends specific prompts to the generative AI.

[0990] Input: Emotion data and purchase history data

[0991] Output: Future prediction results and advice from generative AI

[0992] Step 6:

[0993] The server returns the generated advice to the user and visualizes it through an interface, allowing the user to easily browse the advice.

[0994] Input: Generated advice data

[0995] Output: Advice displayed on the interface

[0996] The above are the specific processing steps of the system that realizes the application example.

[0997] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0998] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0999] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1000] [Third embodiment]

[1001] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1002] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1003] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1004] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1005] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1006] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1007] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1008] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1009] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1010] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1011] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1012] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1013] Overall system overview

[1014] The present invention is a system that provides past information, current information, and future predictions for an individual. This system utilizes generative AI and a large-scale cloud infrastructure to collect data from an individual's devices and online accounts, analyze the collected data, and use generative AI to provide future predictions and advice. The system includes the following main means:

[1015] Data collection methods

[1016] The terminal collects data from the user's devices and online accounts. This data includes photos, videos, text documents, audio files, and interaction history with the generating AI. The server uses the user's authentication information to access the online accounts and periodically retrieve the data. This collection process is automated and designed to minimize user interaction.

[1017] Large-scale cloud storage and time series databases

[1018] The server stores the collected data in large-scale cloud storage. The data is organized chronologically and constructed as a time-series database. This data is a detailed record of the user's past actions and thoughts and is updated regularly. Data is saved and updated automatically at regular intervals.

[1019] Data Analysis Methods

[1020] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. This analysis allows the data to be managed efficiently and made easier to search.

[1021] Generative AI for future prediction and advice

[1022] The server provides the user with future predictions and advice based on the data analyzed using the generative AI. For example, if a user requests advice on their future career path, the generative AI will take into account their past experiences and current situation to generate optimal advice. By learning from the user's past data, the generative AI can make individually customized suggestions.

[1023] Interface Means

[1024] The server provides an interface for users to interact with the generative AI. This interface is implemented as a web or mobile app, through which users can input questions or inquiries and receive answers from the generative AI. The interface is designed to allow users to visualize data and easily check past events and future predictions.

[1025] Specific examples

[1026] User questions and future predictions

[1027] For example, a case will be described where a user asks, "I want to see photos from a trip I took in the summer five years ago."

[1028] 1. Submit a question

[1029] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[1030] 2. Database Queries

[1031] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[1032] 3. Analysis of Generative AI

[1033] The server uses generative AI to analyze the captured photo data and tag it with appropriate metadata (e.g., travel destination, event name).

[1034] 4. Results display

[1035] The server returns the analysis results to the interface, where the user can view the photos.

[1036] Advice on future career paths

[1037] Next, a case where a user "seeks advice on future career paths" will be described.

[1038] 1. Submit a question

[1039] Through the interface, users type in, "I'm looking for advice on my future career path."

[1040] 2. Database Queries

[1041] The server retrieves past career-related data from a time-series database.

[1042] 3. Analysis and advice generation by generative AI

[1043] Based on the acquired data, the generative AI generates optimal career path advice, taking into account the user's skills, experience, and interests.

[1044] 4. Results display

[1045] The server returns the generated advice to the interface, where the user can view the advice.

[1046] The above is an embodiment of the present invention. This system allows users to efficiently utilize past data and gain useful insights into the future. Furthermore, by including means for automatic data collection and search / display, user convenience can be improved.

[1047] The processing flow will be explained below.

[1048] Processing steps from data collection to future prediction

[1049] Data collection

[1050] Step 1:

[1051] A user logs in through the interface and enters authentication information (e.g., username and password).

[1052] Step 2:

[1053] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[1054] Step 3:

[1055] Users register information about the devices they use and their online accounts through the interface.

[1056] Step 4:

[1057] The server receives the registration information and requests access permissions for each device and account.

[1058] Step 5:

[1059] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[1060] Step 6:

[1061] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[1062] Data storage

[1063] Step 7:

[1064] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[1065] Step 8:

[1066] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[1067] Data analysis

[1068] Step 9:

[1069] The server downloads the data stored in the cloud storage and begins analysis.

[1070] Step 10:

[1071] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[1072] Generative AI for future predictions and advice

[1073] Step 11:

[1074] Users input their questions and inquiries through the interface.

[1075] Step 12:

[1076] The server receives the user's question and passes it to the generation AI for analysis.

[1077] Step 13:

[1078] The generative AI references a time-series database and generates answers to questions based on the data obtained.

[1079] Step 14:

[1080] The server sends the generated AI's answer back to the interface.

[1081] Providing an interface

[1082] Step 15:

[1083] Through the interface, users can view past data and advice provided by the generated AI.

[1084] Step 16:

[1085] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[1086] Example: The process of searching for travel photos

[1087] Step 1:

[1088] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[1089] Step 2:

[1090] The server receives the user's query and searches the time-series database for relevant photo data.

[1091] Step 3:

[1092] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[1093] Step 4:

[1094] The server sends the analysis results back to the interface, where the user can view the photos.

[1095] Example 1

[1096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1097] In modern society, digital data about individuals' lives and activities is scattered across a wide range of devices and online accounts, creating a need for effective collection, storage, and analysis of this data to provide future predictions and advice tailored to individual needs.However, current systems often collect data manually, and data analysis and future predictions are not standardized, making it difficult to provide users with information that is sufficiently useful.

[1098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1099] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in a distributed storage in chronological order, means for analyzing the stored data and classifying and tagging it using natural language processing and image recognition technology, means for using artificial intelligence to provide personal future predictions and advice based on the analyzed data, and display means for managing dialogue with the user and providing answers from the artificial intelligence to the user. This integrates a series of processes from automatic data collection to analysis, future predictions, and the provision of advice, allowing users to effectively utilize their own data and obtain more useful information.

[1100] "Personal devices" refers to various types of electronic devices owned by users, including smartphones, tablets, and PCs.

[1101] "Online account" refers to various service accounts that users use on the Internet, including accounts for email, cloud storage, social media, photo sharing services, etc.

[1102] "Data" refers to information collected from personal devices and online accounts, including photos, videos, text documents, audio files, and interaction history.

[1103] "Collection Methods" refers to features or systems that automatically collect data from personal devices or online accounts. These collection methods use API calls and authentication information to obtain data.

[1104] "Means for storing data in distributed storage in chronological order" refers to a method or system for organizing collected data in chronological order and storing it in a distributed data storage system such as cloud storage.

[1105] "Natural language processing" refers to the technology of performing semantic analysis on text data to extract themes, emotions, keywords, etc. It is used to structure unstructured text data.

[1106] "Image recognition technology" refers to the technology of analyzing visual data such as photographs and videos to detect and classify specific people, objects, scenes, etc.

[1107] "Artificial intelligence" refers to machine learning algorithms and models for data analysis and future prediction, including generative AI models, which make predictions and recommendations based on analyzed data.

[1108] "Display means" refers to the interface or application that allows users to view the answers and analysis results from the generated AI. Specifically, this applies to web apps and mobile apps.

[1109] "Schedule management function" refers to a system or function that manages the time schedule for automatic and regular data collection, so that data collection can be carried out continuously at the appropriate time.

[1110] A "specific period" refers to a specific time range specified by the user, including, for example, a specific past day, week, month, or year.

[1111] "Visualization" refers to the process of displaying data in a form that is easy for a user to understand, and includes the use of charts, graphs, photographs, etc.

[1112] The present invention is a system that provides past information, current information, and future predictions for an individual. It utilizes generative AI and a distributed cloud platform to collect and analyze data from an individual's devices and online accounts, and provides future predictions and advice based on the collected data. An embodiment of this system is described in detail below.

[1113] Data collection

[1114] The device collects data from the user's various electronic devices and online accounts. This data collection is automated, for example, by periodically obtaining data using an API. A specific example is the process of downloading photos and videos from the user's cloud storage (online account). This requires OAuth authentication and obtaining an API key. The server uses these credentials to access the user's devices and online services and collect data.

[1115] Data storage

[1116] The server stores the collected data in cloud storage. Specifically, it uses a distributed storage system (e.g., Amazon S3 or Google Cloud Storage) to organize and store the data in chronological order. This allows the data to be managed as a time-series database, efficiently recording the user's past actions and events.

[1117] Data analysis

[1118] The server uses natural language processing (NLP) and image recognition technologies to analyze the stored data. Specific technologies include Google Cloud Natural Language and Amazon Comprehend for semantic analysis of text data, and AWS Rekognition and Google Cloud Vision for content analysis of images and videos. This analysis process classifies the data and assigns appropriate tags, making it easier to search and reference.

[1119] Providing future predictions and advice

[1120] The server uses a generative AI model to provide future predictions and advice to users based on the analyzed data. The system uses models such as OpenAI's GPT series and Google's BERT to learn user behavior and patterns and generate optimal future predictions and advice. For example, if a user enters a question such as "I'm looking for advice on my future career path" into the interface, the server will suggest the optimal career path taking into account past experience and current situation. An example of a prompt sentence in this case would be "I'm looking for advice on my future career path."

[1121] Results display

[1122] The server implements an interface to provide users with future predictions and advice generated by the generative AI. This interface is designed as a web or mobile app and can be built using, for example, React or Flutter. Users can enter questions through this interface and view answers from the generative AI. They can also visualize past data and check future predictions through this interface.

[1123] Specific examples

[1124] Specific examples of the present invention are shown below.

[1125] Travel photo search

[1126] For example, consider the case where a user asks, "I'd like to see photos from a trip I took in the summer five years ago."

[1127] 1. Submit a question

[1128] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[1129] 2. Database Queries

[1130] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[1131] 3. Analysis of Generative AI

[1132] The server analyzes the captured photo data and tags it with appropriate metadata (e.g., travel destination, event name).

[1133] 4. Results display

[1134] The server returns the analysis results to the interface, where the user can view the photos.

[1135] Career path advice

[1136] Let us also consider the case where a user asks, "I'm looking for advice on my future career path."

[1137] 1. Submit a question

[1138] Through the interface, users type in, "I'm looking for advice on my future career path."

[1139] 2. Database Queries

[1140] The server retrieves past career-related data from a time-series database.

[1141] 3. Analysis and advice generation by generative AI

[1142] The generative AI model uses the acquired data to generate optimal career path advice, taking into account the user's skills, experience, and interests.

[1143] 4. Results display

[1144] The server returns the generated advice to the interface, where the user can view the advice.

[1145] In this way, the present invention can effectively collect, store, and analyze user data to provide useful predictions and advice for the future.

[1146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1147] Step 1: Data collection

[1148] The server uses the user's authentication information (input) to automatically collect data from the user's device and online account. This process involves obtaining data such as photos, videos, text documents, and audio files (output) via APIs. For example, the server periodically obtains the user's photos and videos using the Google Photos API or Dropbox API. The server downloads this data and temporarily stores it in local storage.

[1149] Step 2: Save data

[1150] The server transfers the collected data to distributed cloud storage. In this process, the data is uploaded to cloud storage services such as Amazon S3 and Google Cloud Storage (input). The uploaded data (output) is organized in chronological order and recorded in a user database. This allows the collected data to be managed efficiently, making it easier to search and analyze later.

[1151] Step 3: Data analysis

[1152] The server analyzes the stored data. The input here is data read from cloud storage. The server analyzes the text data using Google Cloud Natural Language and AWS Comprehend to extract key topics and sentiment (output). It also analyzes image and video data using AWS Rekognition and Google Cloud Vision to tag specific objects and scenes (input to output). This categorizes the content of the photos and videos and stores them in a database.

[1153] Step 4: Generate future predictions and advice

[1154] The user submits a question or request for advice through the interface (input). For example, they may enter a prompt such as "I'm looking for advice on my future career path." The server retrieves the user's past data from a time-series database (input) and inputs this data into a generative AI model (e.g., GPT-4). The generative AI model generates future predictions and advice from the retrieved data (output). The output results may include, for example, specific career path suggestions or recommended skill sets.

[1155] Step 5: View the results

[1156] The server returns the generated future predictions and advice to the user's interface. The input here is the output result from the generative AI model. The user can view the displayed results through a web app or mobile app. For example, the results are displayed in an interface built using React or Flutter. This allows the user to easily visualize past data and check future predictions.

[1157] Through the above processing steps, the system of the present invention can effectively collect, store, and analyze user data, and provide future predictions and advice.

[1158] (Application example 1)

[1159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1160] In modern online shopping, it is important for users to receive product recommendations based on their past purchase and browsing history. However, current systems do not adequately provide accurate predictions or personalized product recommendations based on individual users' behavioral patterns. Therefore, there is a need for more accurate recommendation systems that can help users find products of interest and improve their satisfaction.

[1161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1162] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, interface means for managing dialogue with the user and providing the user with answers from the generation AI, means for analyzing past purchase history and browsing history to predict future purchasing trends and generate personalized product recommendations, and means for visually presenting the generated product recommendations to the user. This enables highly accurate product recommendations based on the user's individual behavioral patterns.

[1163] "Generative AI" is an artificial intelligence system that learns patterns from large datasets and makes predictions and generation.

[1164] "Personal device" refers to an information processing device owned and used by an individual, such as a smartphone, tablet, or PC.

[1165] "Online Account" refers to a collection of personal authentication information and related data for accessing services on the Internet.

[1166] "Data collection methods" refers to the mechanisms and methods used to obtain the required information from an individual's device or online account.

[1167] A "time series database" refers to a database system for organizing and storing data in chronological order.

[1168] "Large-scale cloud storage" refers to a remote server storage system for storing and managing large amounts of data via the Internet.

[1169] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[1170] "Image recognition technology" refers to the technology that allows a computer to extract, analyze, and classify information from image data.

[1171] "Interface means" refers to the means or method for exchanging information between a user and a computer system.

[1172] "Purchase history" refers to a record of products and services purchased by a user in the past.

[1173] "Browsing history" refers to a record of pages and content that a user has previously accessed on the Internet.

[1174] "Personalized product recommendations" refers to suggesting products that have specific benefits or interests to a user based on their individual purchasing history and behavioral patterns.

[1175] "Visual presentation means" refers to a method or device for visually displaying information to a user.

[1176] MODE FOR CARRYING OUT THE INVENTION

[1177] The present invention is a personalized product recommendation system using generative AI to improve user experience in online shopping. The system includes the following main means:

[1178] Data collection methods

[1179] Personal devices collect purchase and browsing history from users' devices and online accounts. This data is collected automatically and periodically and stored in cloud storage. Data collection is done using communication protocols such as external APIs.

[1180] Large-scale cloud storage and time series databases

[1181] The server stores the collected data in a time-series database in cloud storage. This organizes the data in chronological order, making it easy to refer to the user's behavioral history. Cloud storage services capable of managing large amounts of data (such as Amazon S3 or Google Cloud Storage) are used.

[1182] Data Analysis Methods

[1183] The server uses natural language processing (NLP) and image recognition technology to analyze the stored data. Purchase history and browsing history data are classified and tagged using NLP technology, which allows the interests and concerns of individual users to be clarified.

[1184] Generative AI for future prediction and advice

[1185] The server uses generative AI to predict the user's future purchasing trends based on the analyzed data. The generative AI model used is a deep learning model (e.g., Transformer or GPT model). This model learns from past data and generates individually customized product recommendations.

[1186] Interface Means

[1187] Users can interact with the generative AI through a smartphone app. This interface allows users to visually browse personalized product recommendations. The interface is user-friendly and easy to use.

[1188] Specific use cases

[1189] For example, consider a case where a user wants to predict the next product they are likely to purchase based on their past purchase history of books and electronic devices. The system operates as follows.

[1190] 1. Data collection step: The user's past purchase history and browsing history are collected from the device to cloud storage.

[1191] 2. Data storage step: The collected data is stored in cloud storage as a time series database.

[1192] 3. Data analysis step: The stored data is analyzed using NLP and image recognition technologies to clarify interests and concerns.

[1193] 4. Generative AI step: Based on the analyzed data, the generative AI model predicts the user's future purchasing trends and generates product recommendations.

[1194] 5. Interface step: The user visually browses the generated product recommendations through the smartphone app and makes a purchase decision.

[1195] Examples of prompt statements

[1196] User ID: user1234

[1197] Purchase History:

[1198] 1. Product name: Book A, Purchase date: 2021-10-01, Price: 1,500 yen

[1199] 2. Product name: Home appliance B, Purchase date: 2021-11-15, Price: 30,000 yen

[1200] 3. Product Name: Cosme C, Purchase Date: 2022-02-10, Price: 2,500 yen

[1201] question:

[1202] Predict what this user is likely to purchase this month."

[1203] The above is an embodiment of the present invention, and the system allows users to receive personalized and highly accurate product recommendations, improving their shopping experience.

[1204] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1205] Step 1: Data collection

[1206] The device collects the user's purchase history and browsing history. Specifically, it accesses the user's online account via API and obtains data on previously purchased products and viewed pages. The input is the user ID and authentication information, and the output is a set of purchase history and browsing history data.

[1207] Step 2: Save data

[1208] The server organizes the collected data in chronological order and stores it in large-scale cloud storage. The input is the collected purchase history and browsing history data, and the output is the data stored in cloud storage. Amazon S3 and Google Cloud Storage are used as cloud storage.

[1209] Step 3: Data analysis

[1210] The server analyzes the stored data using NLP and image recognition technology. The input is purchase history and browsing history data retrieved from cloud storage, and the output is the analyzed data (e.g., tagged product information). NLP models such as BERT are used.

[1211] Step 4: Generate future predictions

[1212] The server uses a generative AI model to predict the user's future purchasing trends from the analyzed data. The input is the analyzed data, and the output is personalized product recommendations. Transformer and GPT are used as generative AI models.

[1213] Step 5: Providing recommendations

[1214] The server sends the generated product recommendations to a smartphone app and provides them visually to the user. The input is the product recommendations generated by the generative AI model, and the output is the recommendation results displayed on the user's smartphone app. The user interface is designed so that users can easily check the product recommendations.

[1215] Specific examples

[1216] For example, consider a case where a product that a user is likely to purchase next is predicted based on the purchase history of books and electronic devices that the user has purchased in the past.

[1217] 1. Step 1:

[1218] Input: User ID and authentication information. Purchase history and browsing history are also obtained through API.

[1219] Output: A set of purchase and browsing history data.

[1220] Specific behavior: Sends an API request and receives data in JSON format.

[1221] 2. Step 2:

[1222] Input: Collected purchasing and browsing history data.

[1223] Output: Data stored in cloud storage.

[1224] Specific operation: Organize the data in chronological order and upload it to Amazon S3.

[1225] 3. Step 3:

[1226] Input: Purchase and browsing history data retrieved from cloud storage.

[1227] Output: Parsed data (e.g. tagged product information).

[1228] Specific operation: Extracts and tags product names and categories using NLP technology.

[1229] 4. Step 4:

[1230] Input: Parsed data.

[1231] Output: Personalized product recommendations.

[1232] What it does: Uses generative AI models to predict future purchasing trends and generate product lists.

[1233] 5. Step 5:

[1234] Input: Product recommendations generated by a generative AI model.

[1235] Output: Recommendation results displayed on the user's smartphone app.

[1236] What it does: Sends recommendations to the app and displays them for the user to review.

[1237] These are the specific processing steps of the system. Each step works closely together to improve the user experience, achieving personalized and highly accurate product recommendations.

[1238] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1239] Overall system overview

[1240] This invention is a system that combines a generative AI and an emotion engine to provide a person's past information, current information, and future predictions. The system collects data from a person's devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. The system also uses the emotion engine to recognize the user's emotions and adjusts the advice provided by the generative AI based on that information.

[1241] Data collection methods

[1242] The terminal collects data from the user's devices and online accounts, including photos, videos, text documents, audio files, and interaction history with the generative AI. The server accesses the online accounts based on the user's authentication information and periodically retrieves the data. The collected data is then stored in large-scale cloud storage.

[1243] Large-scale cloud storage and time series databases

[1244] The server stores the collected data in chronological order in large-scale cloud storage. The data is organized in chronological order and constructed as a time-series database. This data records the user's past actions and thoughts in detail and is updated regularly.

[1245] Data Analysis Methods

[1246] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. The results of this analysis are then fed into generative AI and emotion engines.

[1247] Emotion Engine Means

[1248] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes text and voice data to identify the user's emotional state. Changes in emotions are tracked based on the user's interaction history, and the AI ​​uses this data to improve the quality of advice.

[1249] Generative AI for future prediction and advice

[1250] The server provides future predictions and advice based on the data analyzed using the generative AI and the emotional data obtained from the emotion engine. For example, if a user requests advice on their future career path, the generative AI will consider their past experience, current situation, and emotional data to generate optimal advice.

[1251] Interface Means

[1252] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. The interface is also designed to visualize past data and future prediction results so that users can easily check them.

[1253] Specific examples

[1254] User questions and future predictions

[1255] 1. Submit a question

[1256] The user types into the interface, "I'm looking for advice on my future career path."

[1257] 2. Database Queries

[1258] The server retrieves past career-related data and emotion data from a time-series database.

[1259] 3. Analysis and advice generation by generative AI

[1260] Based on the acquired data and data from the emotion engine, the generative AI generates optimal career path advice, taking into account the user's skills, experience, interests, and emotional state.

[1261] 4. Results display

[1262] The server returns the generated advice to the interface, where the user can view the advice.

[1263] The process of searching for travel photos

[1264] 1. Submit a question

[1265] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[1266] 2. Database Queries

[1267] The server searches the time-series database and retrieves photo data that falls within the specified period.

[1268] 3. Analysis of Generative AI

[1269] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[1270] 4. Results display

[1271] The server returns the analysis results to the interface, where the user can view the photos.

[1272] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[1273] The processing flow will be explained below.

[1274] Processing steps from data collection to future prediction and emotion engine

[1275] Data collection

[1276] Step 1:

[1277] A user logs in through the interface and enters authentication information (e.g., username and password).

[1278] Step 2:

[1279] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[1280] Step 3:

[1281] Users register information about the devices they use and their online accounts through the interface.

[1282] Step 4:

[1283] The server receives the registration information and requests access permissions for each device and account.

[1284] Step 5:

[1285] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[1286] Step 6:

[1287] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[1288] Data storage

[1289] Step 7:

[1290] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[1291] Step 8:

[1292] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[1293] Data analysis

[1294] Step 9:

[1295] The server downloads the data stored in the cloud storage and begins analysis.

[1296] Step 10:

[1297] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[1298] Emotion recognition by emotion engine

[1299] Step 11:

[1300] The server acquires the user's text data and voice data and analyzes it using an emotion engine.

[1301] Step 12:

[1302] The emotion engine identifies the user's emotional state based on the captured text and voice data.

[1303] Step 13:

[1304] The emotion engine tracks changes in emotions based on the user's interaction history and provides that data to the generative AI.

[1305] Generative AI for future predictions and advice

[1306] Step 14:

[1307] Users input their questions and inquiries through the interface.

[1308] Step 15:

[1309] The server receives the user's question and passes it to the generation AI for analysis.

[1310] Step 16:

[1311] The generative AI references data provided by the time series database and emotion engine to generate answers to questions.

[1312] Step 17:

[1313] The server sends the generated AI's answer back to the interface.

[1314] Providing an interface

[1315] Step 18:

[1316] Through the interface, users can view past data and advice provided by the generated AI.

[1317] Step 19:

[1318] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[1319] Example: The process of searching for travel photos

[1320] Step 1:

[1321] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[1322] Step 2:

[1323] The server receives the user's query and searches the time-series database for relevant photo data.

[1324] Step 3:

[1325] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[1326] Step 4:

[1327] The server sends the analysis results back to the interface, where the user can view the photos.

[1328] Example: Advice on future career paths

[1329] Step 1:

[1330] The user types into the interface, "I'm looking for advice on my future career path."

[1331] Step 2:

[1332] The server retrieves past career-related data and emotion data from a time-series database.

[1333] Step 3:

[1334] The generative AI uses the acquired data and data from the emotion engine to generate optimal career path advice taking into account the user's skills, experience, interests, and emotional state.

[1335] Step 4:

[1336] The server returns the generated advice to the interface, where the user can view the advice.

[1337] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[1338] Example 2

[1339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1340] In modern society, there is a wide range of data on individuals' lives and activities, and it is a major challenge to effectively collect and analyze this data, and to provide future predictions and appropriate advice. Furthermore, there is a demand for providing appropriate advice that takes into account the user's emotional state, but a system to achieve this has not yet been fully established.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1342] In this invention, the server includes means for collecting data from personal communication devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for inputting the analysis results into an emotion engine and identifying the individual's emotional state, means for using a generative AI to provide future predictions and advice for the individual based on the analyzed data and emotional data, and interface means for managing dialogue with the user and providing the user with answers from the generative AI. This makes it possible to comprehensively collect and analyze a variety of user data and provide highly accurate future predictions and advice that also take the user's emotional state into consideration.

[1343] "Personal communication terminal" is a general term for devices owned by users, such as mobile phones, smartphones, tablets, and personal computers.

[1344] "Online account" is a general term for accounts for various services (social media, cloud storage, email, etc.) that users use on the Internet.

[1345] "Data collection methods" refers to technologies and processes used to automatically obtain data from personal communication devices and online accounts.

[1346] "Means for storing data in large-scale cloud storage in chronological order" refers to a method for organizing and storing collected data in a cloud storage system in chronological order.

[1347] "Large-scale cloud storage" refers to a cloud-based storage system that allows for efficient storage, management, and access of large amounts of data.

[1348] "Means of analyzing data" refers to the techniques and processes used to properly classify, tag, and index collected data.

[1349] "Natural language processing" refers to the technology of using computers to process the natural language that humans use on a daily basis.

[1350] "Image recognition technology" refers to technology that uses computer vision technology to recognize objects, patterns, characters, etc. from images and videos.

[1351] An "emotion engine" refers to technology that analyzes text and voice data to identify a user's emotional state.

[1352] "Generative AI" refers to artificial intelligence that uses machine learning and artificial intelligence techniques to generate new information and advice from data.

[1353] "Means for providing future predictions and advice" refers to technologies and processes that present future scenarios and specific advice to users based on analyzed data and emotional data.

[1354] "Interface means" refers to a user interface through which a user interacts with a system and inputs and outputs information.

[1355] "Schedule management means" refers to the techniques and processes for managing the schedule for regular data collection.

[1356] "Means for searching and displaying events and image data" refers to techniques and processes for searching for events and image data related to a specific period and visually presenting them to a user.

[1357] Overall system overview

[1358] This invention is a system that combines a generative AI and an emotion engine to provide an individual's past information, current information, and future predictions. This system collects data from an individual's communication devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. It also uses the emotion engine to recognize the user's emotions and adjusts the content of the advice provided by the generative AI based on that information.

[1359] Hardware and software used

[1360] Devices and Hardware

[1361] Personal communication devices (smartphones, tablets, computers)

[1362] Cloud servers (Amazon Web Services, Google Cloud Platform, etc.)

[1363] Software and Libraries

[1364] Data collection: API (Google Photos API, Dropbox API, etc.)

[1365] Cloud storage: Amazon S3, Google Cloud Storage

[1366] Time series databases: InfluxDB, TimescaleDB

[1367] Natural Language Processing (NLP): spaCy, NLTK

[1368] Image Recognition: TensorFlow, PyTorch

[1369] Sentiment analysis library: Affectiva, IBM Watson API

[1370] Generative AI model: GPT-3, Transformer model

[1371] Front-end frameworks: React, Vue.js

[1372] Data collection

[1373] The server collects data from users' communication devices and online accounts. For example, data collected from smartphones and PCs includes photos, videos, text documents, audio files, and conversation history with the AI. Data collected using APIs is stored in cloud storage.

[1374] Data storage and time series database construction

[1375] The collected data is stored in chronological order in cloud storage by the server, using Amazon S3 or Google Cloud Storage. The stored data is then constructed into a time-series database using InfluxDB or TimescaleDB. The data is indexed in chronological order and used for subsequent analysis and search.

[1376] Data analysis

[1377] The server analyzes the stored data using natural language processing and image recognition techniques. Text data is tokenized, tagged with parts of speech, and analyzed semantically using NLP libraries (e.g., spaCy and NLTK). Photos and videos are subjected to object and face recognition using TensorFlow and PyTorch. The results of these analyses are stored in a database and fed to the generative AI and emotion engine.

[1378] Emotion Engine

[1379] The server uses text and voice data to identify the user's emotional state. Using an emotion analysis library (such as Affectiva or IBM Watson API), it classifies the user's emotions into categories such as "happiness," "sadness," and "surprise." This emotion data is then used by the generative AI to provide advice.

[1380] Generative AI for future predictions and advice

[1381] The server uses generative AI to provide future predictions and advice based on the analyzed data and emotional data. It uses GPT-3 and Transformer models as generative AI models to generate future scenarios taking into account the user's past experiences and current situation. For example, if a user requests advice on their career path, it will suggest "what type of job would be suitable as their next career step."

[1382] Interface

[1383] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app and is built using front-end frameworks such as React or Vue.js. Users can enter questions or inquiries through the interface and receive answers from the generative AI. The interface is also designed to visualize past data and future predictions so that users can easily check them.

[1384] Specific examples

[1385] 1. Career Advice Prompts

[1386] The user types into the interface, "I'm looking for advice on my next career path."

[1387] 2. Photo search prompt

[1388] The user types into the interface, "I want to see photos from my summer trip five years ago."

[1389] Through the overall configuration and specific processing flow of this system, users can comprehensively collect and analyze a variety of data, and receive future predictions and advice that take into account their emotional state.

[1390] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1391] Step 1: Data collection

[1392] Specific operations: The device collects data from the user's communication devices and online accounts. For example, it automatically retrieves photos, videos, text documents, audio files, and interaction history with the generating AI from smartphones and PCs.

[1393] Input: User authentication and online account information

[1394] Output: Various collected data (photos, videos, text documents, audio files, conversation history)

[1395] Step 2: Save data

[1396] How it works: The server stores the collected data in chronological order in large-scale cloud storage, using Amazon S3 or Google Cloud Storage.

[1397] Input: Various collected data

[1398] Output: Data organized in chronological order is saved on cloud storage.

[1399] Step 3: Building a time series database

[1400] What it does: The server builds a time-series database from the stored data, indexing the data in chronological order using a database like InfluxDB or TimescaleDB.

[1401] Input: Time-series data stored on cloud storage

[1402] Output: Data indexed into a time series database

[1403] Step 4: Data analysis (natural language processing)

[1404] What it does: The server analyzes the stored text data using natural language processing techniques (e.g., spaCy or NLTK), including tokenization, part-of-speech tagging, and semantic analysis.

[1405] Input: Text data in a time series database

[1406] Output: Analysis results (tokenization, part-of-speech tags, semantic analysis)

[1407] Step 5: Data analysis (image recognition)

[1408] Specific operation: The server analyzes the stored photos and videos using image recognition technology (TensorFlow or PyTorch), performs object recognition and facial recognition, and saves the results as metadata.

[1409] Input: Photo and video data in a time series database

[1410] Output: Image recognition results (object recognition, face recognition, etc.)

[1411] Step 6: Sentiment Analysis

[1412] Specific operation: The server analyzes text and audio data using an emotion engine to identify the user's emotional state. It uses emotion analysis libraries (Affectiva and IBM Watson API).

[1413] Input: Natural Language Processing and Audio Data

[1414] Output: Emotion data (labels such as "happy", "sad", "surprise" etc.)

[1415] Step 7: Future prediction and advice generation

[1416] Specific operation: The server uses a generative AI model (such as GPT-3 or a Transformer model) to provide future predictions and advice based on the analyzed data and emotional data.

[1417] Input: Parsed data and sentiment data

[1418] Output: Future prediction and advice (scenario based on prompt)

[1419] Step 8: User Interface

[1420] Specific operation: The server provides an interface for users to interact with the generative AI and emotion engine. Web and mobile apps are built using front-end frameworks such as React and Vue.js. Based on user input, past data is visualized and future prediction results are displayed.

[1421] Input: Questions and inquiries from users

[1422] Output: Answers from the generative AI, visualized historical data, and future prediction results

[1423] (Application example 2)

[1424] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1425] While conventional generative AI systems often predict the future based on an individual's past and current information, they have not yet been able to provide advice that takes into account the user's emotional state or purchasing history, making it difficult to provide specific, individually customized suggestions.In addition, there has been a lack of systems that automatically make optimal suggestions based on the user's emotions and consumption trends in electronic payments and promotions.

[1426] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in large-scale cloud storage in chronological order, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, emotion analysis means for recognizing the user's emotions, analyzing the data, and reflecting the data in the advice provided by the generation AI, means for suggesting optimal payment methods and promotions based on the purchase history and emotion data collected from the personal devices and online accounts, and means for predicting future consumption trends based on the personal purchasing patterns and emotion data. This enables users to receive optimal advice that takes into account their emotional state and purchasing activities.

[1427] "Generative AI" is a technology that uses artificial intelligence to learn from data and generate new information and predictions.

[1428] "Emotion analysis" is a method of recognizing a user's emotional state from data such as text and voice, and quantitatively evaluating that emotion.

[1429] A "time series database" is a database for organizing and storing collected data in chronological order.

[1430] "Natural language processing" is a technology that allows computers to process and understand human language.

[1431] "Image recognition" is a technology for analyzing image data and recognizing its content and characteristics.

[1432] An "interface" is the means by which a user interacts with or obtains information from a system.

[1433] "Cloud storage" is a storage service for storing and managing large amounts of data via the Internet.

[1434] "Purchase history" is a record of purchases made by a user in the past.

[1435] "Promotion" is a marketing activity to promote the sale of a product or service.

[1436] "Consumption trends" refers to patterns and tendencies of users' purchasing behavior.

[1437] "Schedule management" is a management method for collecting and updating data on a regular basis.

[1438] Embodiments of the present invention will be described in detail below.

[1439] System Overview

[1440] This system combines generative AI and a sentiment analysis engine to provide information on a user's past, present, and future, and to suggest optimal payment methods and promotions for individual electronic payment services.

[1441] Data collection

[1442] The server collects data from users' devices and online accounts, including their past purchase history, spending patterns, current financial situation, and emotional state. The data is collected automatically and periodically, and stored in chronological order in large-scale cloud storage. Specific hardware used includes smartphones and cloud storage servers (e.g., AWS, Google Cloud).

[1443] Data analysis

[1444] The stored data is analyzed by the server using natural language processing (NLP) and image recognition techniques to categorize, tag, and index the data. TensorFlow is used for image recognition, and the results are fed into a generative AI and sentiment analysis engine.

[1445] Emotion analysis

[1446] The server analyzes the user's text data and dialogue history and uses an emotion analysis engine to identify their emotional state. Specifically, it uses the NLTK library to track changes in emotion, and the generated AI uses that data.

[1447] Future predictions and advice

[1448] The server provides future predictions and advice based on the data analyzed using generative AI. For example, it suggests optimal payment methods and promotions based on the user's purchasing history and emotional data. It uses a generative AI model (e.g., GPT-3) to generate advice customized for each user.

[1449] Interface

[1450] The interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. It is also designed to visualize past data and future predictions so that users can easily check them. For example, if a user inputs, "I'm worried about my recent spending. Please tell me which credit card I should use," the server will display advice generated by the generative AI based on the appropriate data.

[1451] Examples and prompts

[1452] For example, if a user types, "I'm worried about my recent spending. Can you tell me which credit card I should use?", the following prompt is sent to the generating AI:

[1453] "I'm worried about my recent spending. Which credit card should I use?"

[1454] Purchase History: All purchase data from last year

[1455] Sentiment data: Anxiety score for recent spending

[1456] Based on this, the AI ​​generates advice such as, "Since your expenses have increased recently, we recommend that you use a low-interest credit card. Also, there are currently promotions available that offer cashback and points." and provides this to the user.

[1457] The above is an embodiment of the present invention, which allows a user to receive optimal advice that takes into account their emotional state and purchasing behavior.

[1458] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1459] Step 1:

[1460] Users access the system via a smartphone or web interface and input their questions or concerns. An example of input could be text data such as, "I'm worried about my recent spending. Please tell me which credit card I should use."

[1461] Input: User question text

[1462] Output: The user's question is sent to the server

[1463] Step 2:

[1464] The server automatically collects data from users' devices and online accounts, including their purchasing history, spending patterns, and financial status, and stores the collected data in a time-series database.

[1465] Input: User credentials and online account data

[1466] Output: Purchase history and spending patterns recorded in a time series database

[1467] Step 3:

[1468] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technologies. Text data is semantically analyzed using NLP technology, and image data is analyzed using image recognition.

[1469] Input: Purchase history and spending pattern data obtained from a time series database

[1470] Output: A classified and tagged dataset

[1471] Step 4:

[1472] The server analyzes the acquired text data and dialogue history using an emotion analysis engine to identify the user's emotional state, and calculates an emotion score using the NLTK library.

[1473] Input: User text data and interaction history

[1474] Output: Sentiment score

[1475] Step 5:

[1476] The server uses a generative AI model (such as GPT-3) to generate future predictions and optimal advice based on the emotion data obtained from the emotion analysis engine and the stored purchase history data, and sends specific prompts to the generative AI.

[1477] Input: Emotion data and purchase history data

[1478] Output: Future prediction results and advice from generative AI

[1479] Step 6:

[1480] The server returns the generated advice to the user and visualizes it through an interface, allowing the user to easily browse the advice.

[1481] Input: Generated advice data

[1482] Output: Advice displayed on the interface

[1483] The above are the specific processing steps of the system that realizes the application example.

[1484] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1485] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1486] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1487] [Fourth embodiment]

[1488] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1489] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1490] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1491] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1492] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1493] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1494] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1495] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1496] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1497] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1498] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1499] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1500] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1501] Overall system overview

[1502] The present invention is a system that provides past information, current information, and future predictions for an individual. This system utilizes generative AI and a large-scale cloud infrastructure to collect data from an individual's devices and online accounts, analyze the collected data, and use generative AI to provide future predictions and advice. The system includes the following main means:

[1503] Data collection methods

[1504] The terminal collects data from the user's devices and online accounts. This data includes photos, videos, text documents, audio files, and interaction history with the generating AI. The server uses the user's authentication information to access the online accounts and periodically retrieve the data. This collection process is automated and designed to minimize user interaction.

[1505] Large-scale cloud storage and time series databases

[1506] The server stores the collected data in large-scale cloud storage. The data is organized chronologically and constructed as a time-series database. This data is a detailed record of the user's past actions and thoughts and is updated regularly. Data is saved and updated automatically at regular intervals.

[1507] Data Analysis Methods

[1508] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. This analysis allows the data to be managed efficiently and made easier to search.

[1509] Generative AI for future prediction and advice

[1510] The server provides the user with future predictions and advice based on the data analyzed using the generative AI. For example, if a user requests advice on their future career path, the generative AI will take into account their past experiences and current situation to generate optimal advice. By learning from the user's past data, the generative AI can make individually customized suggestions.

[1511] Interface Means

[1512] The server provides an interface for users to interact with the generative AI. This interface is implemented as a web or mobile app, through which users can input questions or inquiries and receive answers from the generative AI. The interface is designed to allow users to visualize data and easily check past events and future predictions.

[1513] Specific examples

[1514] User questions and future predictions

[1515] For example, a case will be described where a user asks, "I want to see photos from a trip I took in the summer five years ago."

[1516] 1. Submit a question

[1517] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[1518] 2. Database Queries

[1519] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[1520] 3. Analysis of Generative AI

[1521] The server uses generative AI to analyze the captured photo data and tag it with appropriate metadata (e.g., travel destination, event name).

[1522] 4. Results display

[1523] The server returns the analysis results to the interface, where the user can view the photos.

[1524] Advice on future career paths

[1525] Next, a case where a user "seeks advice on future career paths" will be described.

[1526] 1. Submit a question

[1527] Through the interface, users type in, "I'm looking for advice on my future career path."

[1528] 2. Database Queries

[1529] The server retrieves past career-related data from a time-series database.

[1530] 3. Analysis and advice generation by generative AI

[1531] Based on the acquired data, the generative AI generates optimal career path advice, taking into account the user's skills, experience, and interests.

[1532] 4. Results display

[1533] The server returns the generated advice to the interface, where the user can view the advice.

[1534] The above is an embodiment of the present invention. This system allows users to efficiently utilize past data and gain useful insights into the future. Furthermore, by including means for automatic data collection and search / display, user convenience can be improved.

[1535] The processing flow will be explained below.

[1536] Processing steps from data collection to future prediction

[1537] Data collection

[1538] Step 1:

[1539] A user logs in through the interface and enters authentication information (e.g., username and password).

[1540] Step 2:

[1541] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[1542] Step 3:

[1543] Users register information about the devices they use and their online accounts through the interface.

[1544] Step 4:

[1545] The server receives the registration information and requests access permissions for each device and account.

[1546] Step 5:

[1547] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[1548] Step 6:

[1549] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[1550] Data storage

[1551] Step 7:

[1552] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[1553] Step 8:

[1554] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[1555] Data analysis

[1556] Step 9:

[1557] The server downloads the data stored in the cloud storage and begins analysis.

[1558] Step 10:

[1559] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[1560] Generative AI for future predictions and advice

[1561] Step 11:

[1562] Users input their questions and inquiries through the interface.

[1563] Step 12:

[1564] The server receives the user's question and passes it to the generation AI for analysis.

[1565] Step 13:

[1566] The generative AI references a time-series database and generates answers to questions based on the data obtained.

[1567] Step 14:

[1568] The server sends the generated AI's answer back to the interface.

[1569] Providing an interface

[1570] Step 15:

[1571] Through the interface, users can view past data and advice provided by the generated AI.

[1572] Step 16:

[1573] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[1574] Example: The process of searching for travel photos

[1575] Step 1:

[1576] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[1577] Step 2:

[1578] The server receives the user's query and searches the time-series database for relevant photo data.

[1579] Step 3:

[1580] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[1581] Step 4:

[1582] The server sends the analysis results back to the interface, where the user can view the photos.

[1583] Example 1

[1584] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1585] In modern society, digital data about individuals' lives and activities is scattered across a wide range of devices and online accounts, creating a need for effective collection, storage, and analysis of this data to provide future predictions and advice tailored to individual needs.However, current systems often collect data manually, and data analysis and future predictions are not standardized, making it difficult to provide users with information that is sufficiently useful.

[1586] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1587] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in a distributed storage in chronological order, means for analyzing the stored data and classifying and tagging it using natural language processing and image recognition technology, means for using artificial intelligence to provide personal future predictions and advice based on the analyzed data, and display means for managing dialogue with the user and providing answers from the artificial intelligence to the user. This integrates a series of processes from automatic data collection to analysis, future predictions, and the provision of advice, allowing users to effectively utilize their own data and obtain more useful information.

[1588] "Personal devices" refers to various types of electronic devices owned by users, including smartphones, tablets, and PCs.

[1589] "Online account" refers to various service accounts that users use on the Internet, including accounts for email, cloud storage, social media, photo sharing services, etc.

[1590] "Data" refers to information collected from personal devices and online accounts, including photos, videos, text documents, audio files, and interaction history.

[1591] "Collection Methods" refers to features or systems that automatically collect data from personal devices or online accounts. These collection methods use API calls and authentication information to obtain data.

[1592] "Means for storing data in distributed storage in chronological order" refers to a method or system for organizing collected data in chronological order and storing it in a distributed data storage system such as cloud storage.

[1593] "Natural language processing" refers to the technology of performing semantic analysis on text data to extract themes, emotions, keywords, etc. It is used to structure unstructured text data.

[1594] "Image recognition technology" refers to the technology of analyzing visual data such as photographs and videos to detect and classify specific people, objects, scenes, etc.

[1595] "Artificial intelligence" refers to machine learning algorithms and models for data analysis and future prediction, including generative AI models, which make predictions and recommendations based on analyzed data.

[1596] "Display means" refers to the interface or application that allows users to view the answers and analysis results from the generated AI. Specifically, this applies to web apps and mobile apps.

[1597] "Schedule management function" refers to a system or function that manages the time schedule for automatic and regular data collection, so that data collection can be carried out continuously at the appropriate time.

[1598] A "specific period" refers to a specific time range specified by the user, including, for example, a specific past day, week, month, or year.

[1599] "Visualization" refers to the process of displaying data in a form that is easy for a user to understand, and includes the use of charts, graphs, photographs, etc.

[1600] The present invention is a system that provides past information, current information, and future predictions for an individual. It utilizes generative AI and a distributed cloud platform to collect and analyze data from an individual's devices and online accounts, and provides future predictions and advice based on the collected data. An embodiment of this system is described in detail below.

[1601] Data collection

[1602] The device collects data from the user's various electronic devices and online accounts. This data collection is automated, for example, by periodically obtaining data using an API. A specific example is the process of downloading photos and videos from the user's cloud storage (online account). This requires OAuth authentication and obtaining an API key. The server uses these credentials to access the user's devices and online services and collect data.

[1603] Data storage

[1604] The server stores the collected data in cloud storage. Specifically, it uses a distributed storage system (e.g., Amazon S3 or Google Cloud Storage) to organize and store the data in chronological order. This allows the data to be managed as a time-series database, efficiently recording the user's past actions and events.

[1605] Data analysis

[1606] The server uses natural language processing (NLP) and image recognition technologies to analyze the stored data. Specific technologies include Google Cloud Natural Language and Amazon Comprehend for semantic analysis of text data, and AWS Rekognition and Google Cloud Vision for content analysis of images and videos. This analysis process classifies the data and assigns appropriate tags, making it easier to search and reference.

[1607] Providing future predictions and advice

[1608] The server uses a generative AI model to provide future predictions and advice to users based on the analyzed data. The system uses models such as OpenAI's GPT series and Google's BERT to learn user behavior and patterns and generate optimal future predictions and advice. For example, if a user enters a question such as "I'm looking for advice on my future career path" into the interface, the server will suggest the optimal career path taking into account past experience and current situation. An example of a prompt sentence in this case would be "I'm looking for advice on my future career path."

[1609] Results display

[1610] The server implements an interface to provide users with future predictions and advice generated by the generative AI. This interface is designed as a web or mobile app and can be built using, for example, React or Flutter. Users can enter questions through this interface and view answers from the generative AI. They can also visualize past data and check future predictions through this interface.

[1611] Specific examples

[1612] Specific examples of the present invention are shown below.

[1613] Travel photo search

[1614] For example, consider the case where a user asks, "I'd like to see photos from a trip I took in the summer five years ago."

[1615] 1. Submit a question

[1616] Through the interface, the user types, "I'd like to see photos from a trip I took five summers ago."

[1617] 2. Database Queries

[1618] The server searches the time-series database and retrieves photo data that corresponds to the specified period.

[1619] 3. Analysis of Generative AI

[1620] The server analyzes the captured photo data and tags it with appropriate metadata (e.g., travel destination, event name).

[1621] 4. Results display

[1622] The server returns the analysis results to the interface, where the user can view the photos.

[1623] Career path advice

[1624] Let us also consider the case where a user asks, "I'm looking for advice on my future career path."

[1625] 1. Submit a question

[1626] Through the interface, users type in, "I'm looking for advice on my future career path."

[1627] 2. Database Queries

[1628] The server retrieves past career-related data from a time-series database.

[1629] 3. Analysis and advice generation by generative AI

[1630] The generative AI model uses the acquired data to generate optimal career path advice, taking into account the user's skills, experience, and interests.

[1631] 4. Results display

[1632] The server returns the generated advice to the interface, where the user can view the advice.

[1633] In this way, the present invention can effectively collect, store, and analyze user data to provide useful predictions and advice for the future.

[1634] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1635] Step 1: Data collection

[1636] The server uses the user's authentication information (input) to automatically collect data from the user's device and online account. This process involves obtaining data such as photos, videos, text documents, and audio files (output) via APIs. For example, the server periodically obtains the user's photos and videos using the Google Photos API or Dropbox API. The server downloads this data and temporarily stores it in local storage.

[1637] Step 2: Save data

[1638] The server transfers the collected data to distributed cloud storage. In this process, the data is uploaded to cloud storage services such as Amazon S3 and Google Cloud Storage (input). The uploaded data (output) is organized in chronological order and recorded in a user database. This allows the collected data to be managed efficiently, making it easier to search and analyze later.

[1639] Step 3: Data analysis

[1640] The server analyzes the stored data. The input here is data read from cloud storage. The server analyzes the text data using Google Cloud Natural Language and AWS Comprehend to extract key topics and sentiment (output). It also analyzes image and video data using AWS Rekognition and Google Cloud Vision to tag specific objects and scenes (input to output). This categorizes the content of the photos and videos and stores them in a database.

[1641] Step 4: Generate future predictions and advice

[1642] The user submits a question or request for advice through the interface (input). For example, they may enter a prompt such as "I'm looking for advice on my future career path." The server retrieves the user's past data from a time-series database (input) and inputs this data into a generative AI model (e.g., GPT-4). The generative AI model generates future predictions and advice from the retrieved data (output). The output results may include, for example, specific career path suggestions or recommended skill sets.

[1643] Step 5: View the results

[1644] The server returns the generated future predictions and advice to the user's interface. The input here is the output result from the generative AI model. The user can view the displayed results through a web app or mobile app. For example, the results are displayed in an interface built using React or Flutter. This allows the user to easily visualize past data and check future predictions.

[1645] Through the above processing steps, the system of the present invention can effectively collect, store, and analyze user data, and provide future predictions and advice.

[1646] (Application example 1)

[1647] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1648] In modern online shopping, it is important for users to receive product recommendations based on their past purchase and browsing history. However, current systems do not adequately provide accurate predictions or personalized product recommendations based on individual users' behavioral patterns. Therefore, there is a need for more accurate recommendation systems that can help users find products of interest and improve their satisfaction.

[1649] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1650] In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, interface means for managing dialogue with the user and providing the user with answers from the generation AI, means for analyzing past purchase history and browsing history to predict future purchasing trends and generate personalized product recommendations, and means for visually presenting the generated product recommendations to the user. This enables highly accurate product recommendations based on the user's individual behavioral patterns.

[1651] "Generative AI" is an artificial intelligence system that learns patterns from large datasets and makes predictions and generation.

[1652] "Personal device" refers to an information processing device owned and used by an individual, such as a smartphone, tablet, or PC.

[1653] "Online Account" refers to a collection of personal authentication information and related data for accessing services on the Internet.

[1654] "Data collection methods" refers to the mechanisms and methods used to obtain the required information from an individual's device or online account.

[1655] A "time series database" refers to a database system for organizing and storing data in chronological order.

[1656] "Large-scale cloud storage" refers to a remote server storage system for storing and managing large amounts of data via the Internet.

[1657] "Natural language processing" refers to the technology that enables computers to understand, interpret, and generate human language.

[1658] "Image recognition technology" refers to the technology that allows a computer to extract, analyze, and classify information from image data.

[1659] "Interface means" refers to the means or method for exchanging information between a user and a computer system.

[1660] "Purchase history" refers to a record of products and services purchased by a user in the past.

[1661] "Browsing history" refers to a record of pages and content that a user has previously accessed on the Internet.

[1662] "Personalized product recommendations" refers to suggesting products that have specific benefits or interests to a user based on their individual purchasing history and behavioral patterns.

[1663] "Visual presentation means" refers to a method or device for visually displaying information to a user.

[1664] MODE FOR CARRYING OUT THE INVENTION

[1665] The present invention is a personalized product recommendation system using generative AI to improve user experience in online shopping. The system includes the following main means:

[1666] Data collection methods

[1667] Personal devices collect purchase and browsing history from users' devices and online accounts. This data is collected automatically and periodically and stored in cloud storage. Data collection is done using communication protocols such as external APIs.

[1668] Large-scale cloud storage and time series databases

[1669] The server stores the collected data in a time-series database in cloud storage. This organizes the data in chronological order, making it easy to refer to the user's behavioral history. Cloud storage services capable of managing large amounts of data (such as Amazon S3 or Google Cloud Storage) are used.

[1670] Data Analysis Methods

[1671] The server uses natural language processing (NLP) and image recognition technology to analyze the stored data. Purchase history and browsing history data are classified and tagged using NLP technology, which allows the interests and concerns of individual users to be clarified.

[1672] Generative AI for future prediction and advice

[1673] The server uses generative AI to predict the user's future purchasing trends based on the analyzed data. The generative AI model used is a deep learning model (e.g., Transformer or GPT model). This model learns from past data and generates individually customized product recommendations.

[1674] Interface Means

[1675] Users can interact with the generative AI through a smartphone app. This interface allows users to visually browse personalized product recommendations. The interface is user-friendly and easy to use.

[1676] Specific use cases

[1677] For example, consider a case where a user wants to predict the next product they are likely to purchase based on their past purchase history of books and electronic devices. The system operates as follows.

[1678] 1. Data collection step: The user's past purchase history and browsing history are collected from the device to cloud storage.

[1679] 2. Data storage step: The collected data is stored in cloud storage as a time series database.

[1680] 3. Data analysis step: The stored data is analyzed using NLP and image recognition technologies to clarify interests and concerns.

[1681] 4. Generative AI step: Based on the analyzed data, the generative AI model predicts the user's future purchasing trends and generates product recommendations.

[1682] 5. Interface step: The user visually browses the generated product recommendations through the smartphone app and makes a purchase decision.

[1683] Examples of prompt statements

[1684] User ID: user1234

[1685] Purchase History:

[1686] 1. Product name: Book A, Purchase date: 2021-10-01, Price: 1,500 yen

[1687] 2. Product name: Home appliance B, Purchase date: 2021-11-15, Price: 30,000 yen

[1688] 3. Product Name: Cosme C, Purchase Date: 2022-02-10, Price: 2,500 yen

[1689] question:

[1690] Predict what this user is likely to purchase this month."

[1691] The above is an embodiment of the present invention, and the system allows users to receive personalized and highly accurate product recommendations, improving their shopping experience.

[1692] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1693] Step 1: Data collection

[1694] The device collects the user's purchase history and browsing history. Specifically, it accesses the user's online account via API and obtains data on previously purchased products and viewed pages. The input is the user ID and authentication information, and the output is a set of purchase history and browsing history data.

[1695] Step 2: Save data

[1696] The server organizes the collected data in chronological order and stores it in large-scale cloud storage. The input is the collected purchase history and browsing history data, and the output is the data stored in cloud storage. Amazon S3 and Google Cloud Storage are used as cloud storage.

[1697] Step 3: Data analysis

[1698] The server analyzes the stored data using NLP and image recognition technology. The input is purchase history and browsing history data retrieved from cloud storage, and the output is the analyzed data (e.g., tagged product information). NLP models such as BERT are used.

[1699] Step 4: Generate future predictions

[1700] The server uses a generative AI model to predict the user's future purchasing trends from the analyzed data. The input is the analyzed data, and the output is personalized product recommendations. Transformer and GPT are used as generative AI models.

[1701] Step 5: Providing recommendations

[1702] The server sends the generated product recommendations to a smartphone app and provides them visually to the user. The input is the product recommendations generated by the generative AI model, and the output is the recommendation results displayed on the user's smartphone app. The user interface is designed so that users can easily check the product recommendations.

[1703] Specific examples

[1704] For example, consider a case where a product that a user is likely to purchase next is predicted based on the purchase history of books and electronic devices that the user has purchased in the past.

[1705] 1. Step 1:

[1706] Input: User ID and authentication information. Purchase history and browsing history are also obtained through API.

[1707] Output: A set of purchase and browsing history data.

[1708] Specific behavior: Sends an API request and receives data in JSON format.

[1709] 2. Step 2:

[1710] Input: Collected purchasing and browsing history data.

[1711] Output: Data stored in cloud storage.

[1712] Specific operation: Organize the data in chronological order and upload it to Amazon S3.

[1713] 3. Step 3:

[1714] Input: Purchase and browsing history data retrieved from cloud storage.

[1715] Output: Parsed data (e.g. tagged product information).

[1716] Specific operation: Extracts and tags product names and categories using NLP technology.

[1717] 4. Step 4:

[1718] Input: Parsed data.

[1719] Output: Personalized product recommendations.

[1720] What it does: Uses generative AI models to predict future purchasing trends and generate product lists.

[1721] 5. Step 5:

[1722] Input: Product recommendations generated by a generative AI model.

[1723] Output: Recommendation results displayed on the user's smartphone app.

[1724] What it does: Sends recommendations to the app and displays them for the user to review.

[1725] These are the specific processing steps of the system. Each step works closely together to improve the user experience, achieving personalized and highly accurate product recommendations.

[1726] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1727] Overall system overview

[1728] This invention is a system that combines a generative AI and an emotion engine to provide a person's past information, current information, and future predictions. The system collects data from a person's devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. The system also uses the emotion engine to recognize the user's emotions and adjusts the advice provided by the generative AI based on that information.

[1729] Data collection methods

[1730] The terminal collects data from the user's devices and online accounts, including photos, videos, text documents, audio files, and interaction history with the generative AI. The server accesses the online accounts based on the user's authentication information and periodically retrieves the data. The collected data is then stored in large-scale cloud storage.

[1731] Large-scale cloud storage and time series databases

[1732] The server stores the collected data in chronological order in large-scale cloud storage. The data is organized in chronological order and constructed as a time-series database. This data records the user's past actions and thoughts in detail and is updated regularly.

[1733] Data Analysis Methods

[1734] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technology. For example, photos and videos are analyzed for content using image recognition technology, while text data is analyzed for semantics using NLP technology. The results of this analysis are then fed into generative AI and emotion engines.

[1735] Emotion Engine Means

[1736] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes text and voice data to identify the user's emotional state. Changes in emotions are tracked based on the user's interaction history, and the AI ​​uses this data to improve the quality of advice.

[1737] Generative AI for future prediction and advice

[1738] The server provides future predictions and advice based on the data analyzed using the generative AI and the emotional data obtained from the emotion engine. For example, if a user requests advice on their future career path, the generative AI will consider their past experience, current situation, and emotional data to generate optimal advice.

[1739] Interface Means

[1740] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. The interface is also designed to visualize past data and future prediction results so that users can easily check them.

[1741] Specific examples

[1742] User questions and future predictions

[1743] 1. Submit a question

[1744] The user types into the interface, "I'm looking for advice on my future career path."

[1745] 2. Database Queries

[1746] The server retrieves past career-related data and emotion data from a time-series database.

[1747] 3. Analysis and advice generation by generative AI

[1748] Based on the acquired data and data from the emotion engine, the generative AI generates optimal career path advice, taking into account the user's skills, experience, interests, and emotional state.

[1749] 4. Results display

[1750] The server returns the generated advice to the interface, where the user can view the advice.

[1751] The process of searching for travel photos

[1752] 1. Submit a question

[1753] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[1754] 2. Database Queries

[1755] The server searches the time-series database and retrieves photo data that falls within the specified period.

[1756] 3. Analysis of Generative AI

[1757] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[1758] 4. Results display

[1759] The server returns the analysis results to the interface, where the user can view the photos.

[1760] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[1761] The processing flow will be explained below.

[1762] Processing steps from data collection to future prediction and emotion engine

[1763] Data collection

[1764] Step 1:

[1765] A user logs in through the interface and enters authentication information (e.g., username and password).

[1766] Step 2:

[1767] The server receives the user's authentication information, checks it against a database, and if authentication is successful, allows the user to log in.

[1768] Step 3:

[1769] Users register information about the devices they use and their online accounts through the interface.

[1770] Step 4:

[1771] The server receives the registration information and requests access permissions for each device and account.

[1772] Step 5:

[1773] The server sets an automated schedule to collect data from users' devices and online accounts at regular intervals.

[1774] Step 6:

[1775] The server retrieves data via APIs or the file system of each device and stores it in temporary storage.

[1776] Data storage

[1777] Step 7:

[1778] The server uploads the data stored in the temporary storage to a large-scale cloud storage.

[1779] Step 8:

[1780] The server indexes the metadata (time, location, type, etc.) corresponding to the data.

[1781] Data analysis

[1782] Step 9:

[1783] The server downloads the data stored in the cloud storage and begins analysis.

[1784] Step 10:

[1785] The server uses natural language processing (NLP) and image recognition techniques to classify, tag, and index the data.

[1786] Emotion recognition by emotion engine

[1787] Step 11:

[1788] The server acquires the user's text data and voice data and analyzes it using an emotion engine.

[1789] Step 12:

[1790] The emotion engine identifies the user's emotional state based on the captured text and voice data.

[1791] Step 13:

[1792] The emotion engine tracks changes in emotions based on the user's interaction history and provides that data to the generative AI.

[1793] Generative AI for future predictions and advice

[1794] Step 14:

[1795] Users input their questions and inquiries through the interface.

[1796] Step 15:

[1797] The server receives the user's question and passes it to the generation AI for analysis.

[1798] Step 16:

[1799] The generative AI references data provided by the time series database and emotion engine to generate answers to questions.

[1800] Step 17:

[1801] The server sends the generated AI's answer back to the interface.

[1802] Providing an interface

[1803] Step 18:

[1804] Through the interface, users can view past data and advice provided by the generated AI.

[1805] Step 19:

[1806] Users can use the interface to interact with the generated AI in real time, asking questions and providing advice.

[1807] Example: The process of searching for travel photos

[1808] Step 1:

[1809] The user types into the interface, "I want to see photos from a trip I took five summers ago."

[1810] Step 2:

[1811] The server receives the user's query and searches the time-series database for relevant photo data.

[1812] Step 3:

[1813] The server uses generative AI to analyze the retrieved photo data and generate appropriate metadata.

[1814] Step 4:

[1815] The server sends the analysis results back to the interface, where the user can view the photos.

[1816] Example: Advice on future career paths

[1817] Step 1:

[1818] The user types into the interface, "I'm looking for advice on my future career path."

[1819] Step 2:

[1820] The server retrieves past career-related data and emotion data from a time-series database.

[1821] Step 3:

[1822] The generative AI uses the acquired data and data from the emotion engine to generate optimal career path advice taking into account the user's skills, experience, interests, and emotional state.

[1823] Step 4:

[1824] The server returns the generated advice to the interface, where the user can view the advice.

[1825] The above is an embodiment of the present invention that combines an emotion engine. This system allows users to efficiently utilize past data and gain useful insights into the future. It also provides individually customized advice based on the user's emotional state, achieving a better user experience.

[1826] Example 2

[1827] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1828] In modern society, there is a wide range of data on individuals' lives and activities, and it is a major challenge to effectively collect and analyze this data, and to provide future predictions and appropriate advice. Furthermore, there is a demand for providing appropriate advice that takes into account the user's emotional state, but a system to achieve this has not yet been fully established.

[1829] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1830] In this invention, the server includes means for collecting data from personal communication devices and online accounts, means for storing the collected data in chronological order in large-scale cloud storage, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for inputting the analysis results into an emotion engine and identifying the individual's emotional state, means for using a generative AI to provide future predictions and advice for the individual based on the analyzed data and emotional data, and interface means for managing dialogue with the user and providing the user with answers from the generative AI. This makes it possible to comprehensively collect and analyze a variety of user data and provide highly accurate future predictions and advice that also take the user's emotional state into consideration.

[1831] "Personal communication terminal" is a general term for devices owned by users, such as mobile phones, smartphones, tablets, and personal computers.

[1832] "Online account" is a general term for accounts for various services (social media, cloud storage, email, etc.) that users use on the Internet.

[1833] "Data collection methods" refers to technologies and processes used to automatically obtain data from personal communication devices and online accounts.

[1834] "Means for storing data in large-scale cloud storage in chronological order" refers to a method for organizing and storing collected data in a cloud storage system in chronological order.

[1835] "Large-scale cloud storage" refers to a cloud-based storage system that allows for efficient storage, management, and access of large amounts of data.

[1836] "Means of analyzing data" refers to the techniques and processes used to properly classify, tag, and index collected data.

[1837] "Natural language processing" refers to the technology of using computers to process the natural language that humans use on a daily basis.

[1838] "Image recognition technology" refers to technology that uses computer vision technology to recognize objects, patterns, characters, etc. from images and videos.

[1839] An "emotion engine" refers to technology that analyzes text and voice data to identify a user's emotional state.

[1840] "Generative AI" refers to artificial intelligence that uses machine learning and artificial intelligence techniques to generate new information and advice from data.

[1841] "Means for providing future predictions and advice" refers to technologies and processes that present future scenarios and specific advice to users based on analyzed data and emotional data.

[1842] "Interface means" refers to a user interface through which a user interacts with a system and inputs and outputs information.

[1843] "Schedule management means" refers to the techniques and processes for managing the schedule for regular data collection.

[1844] "Means for searching and displaying events and image data" refers to techniques and processes for searching for events and image data related to a specific period and visually presenting them to a user.

[1845] Overall system overview

[1846] This invention is a system that combines a generative AI and an emotion engine to provide an individual's past information, current information, and future predictions. This system collects data from an individual's communication devices and online accounts, analyzes the collected data, and uses a generative AI to provide future predictions and advice. It also uses the emotion engine to recognize the user's emotions and adjusts the content of the advice provided by the generative AI based on that information.

[1847] Hardware and software used

[1848] Devices and Hardware

[1849] Personal communication devices (smartphones, tablets, computers)

[1850] Cloud servers (Amazon Web Services, Google Cloud Platform, etc.)

[1851] Software and Libraries

[1852] Data collection: API (Google Photos API, Dropbox API, etc.)

[1853] Cloud storage: Amazon S3, Google Cloud Storage

[1854] Time series databases: InfluxDB, TimescaleDB

[1855] Natural Language Processing (NLP): spaCy, NLTK

[1856] Image Recognition: TensorFlow, PyTorch

[1857] Sentiment analysis library: Affectiva, IBM Watson API

[1858] Generative AI model: GPT-3, Transformer model

[1859] Front-end frameworks: React, Vue.js

[1860] Data collection

[1861] The server collects data from users' communication devices and online accounts. For example, data collected from smartphones and PCs includes photos, videos, text documents, audio files, and conversation history with the AI. Data collected using APIs is stored in cloud storage.

[1862] Data storage and time series database construction

[1863] The collected data is stored in chronological order in cloud storage by the server, using Amazon S3 or Google Cloud Storage. The stored data is then constructed into a time-series database using InfluxDB or TimescaleDB. The data is indexed in chronological order and used for subsequent analysis and search.

[1864] Data analysis

[1865] The server analyzes the stored data using natural language processing and image recognition techniques. Text data is tokenized, tagged with parts of speech, and analyzed semantically using NLP libraries (e.g., spaCy and NLTK). Photos and videos are subjected to object and face recognition using TensorFlow and PyTorch. The results of these analyses are stored in a database and fed to the generative AI and emotion engine.

[1866] Emotion Engine

[1867] The server uses text and voice data to identify the user's emotional state. Using an emotion analysis library (such as Affectiva or IBM Watson API), it classifies the user's emotions into categories such as "happiness," "sadness," and "surprise." This emotion data is then used by the generative AI to provide advice.

[1868] Generative AI for future predictions and advice

[1869] The server uses generative AI to provide future predictions and advice based on the analyzed data and emotional data. It uses GPT-3 and Transformer models as generative AI models to generate future scenarios taking into account the user's past experiences and current situation. For example, if a user requests advice on their career path, it will suggest "what type of job would be suitable as their next career step."

[1870] Interface

[1871] The server provides an interface for users to interact with the generative AI and emotion engine. This interface is implemented as a web or mobile app and is built using front-end frameworks such as React or Vue.js. Users can enter questions or inquiries through the interface and receive answers from the generative AI. The interface is also designed to visualize past data and future predictions so that users can easily check them.

[1872] Specific examples

[1873] 1. Career Advice Prompts

[1874] The user types into the interface, "I'm looking for advice on my next career path."

[1875] 2. Photo search prompt

[1876] The user types into the interface, "I want to see photos from my summer trip five years ago."

[1877] Through the overall configuration and specific processing flow of this system, users can comprehensively collect and analyze a variety of data, and receive future predictions and advice that take into account their emotional state.

[1878] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1879] Step 1: Data collection

[1880] Specific operations: The device collects data from the user's communication devices and online accounts. For example, it automatically retrieves photos, videos, text documents, audio files, and interaction history with the generating AI from smartphones and PCs.

[1881] Input: User authentication and online account information

[1882] Output: Various collected data (photos, videos, text documents, audio files, conversation history)

[1883] Step 2: Save data

[1884] How it works: The server stores the collected data in chronological order in large-scale cloud storage, using Amazon S3 or Google Cloud Storage.

[1885] Input: Various collected data

[1886] Output: Data organized in chronological order is saved on cloud storage.

[1887] Step 3: Building a time series database

[1888] What it does: The server builds a time-series database from the stored data, indexing the data in chronological order using a database like InfluxDB or TimescaleDB.

[1889] Input: Time-series data stored on cloud storage

[1890] Output: Data indexed into a time series database

[1891] Step 4: Data analysis (natural language processing)

[1892] What it does: The server analyzes the stored text data using natural language processing techniques (e.g., spaCy or NLTK), including tokenization, part-of-speech tagging, and semantic analysis.

[1893] Input: Text data in a time series database

[1894] Output: Analysis results (tokenization, part-of-speech tags, semantic analysis)

[1895] Step 5: Data analysis (image recognition)

[1896] Specific operation: The server analyzes the stored photos and videos using image recognition technology (TensorFlow or PyTorch), performs object recognition and facial recognition, and saves the results as metadata.

[1897] Input: Photo and video data in a time series database

[1898] Output: Image recognition results (object recognition, face recognition, etc.)

[1899] Step 6: Sentiment Analysis

[1900] Specific operation: The server analyzes text and audio data using an emotion engine to identify the user's emotional state. It uses emotion analysis libraries (Affectiva and IBM Watson API).

[1901] Input: Natural Language Processing and Audio Data

[1902] Output: Emotion data (labels such as "happy", "sad", "surprise" etc.)

[1903] Step 7: Future prediction and advice generation

[1904] Specific operation: The server uses a generative AI model (such as GPT-3 or a Transformer model) to provide future predictions and advice based on the analyzed data and emotional data.

[1905] Input: Parsed data and sentiment data

[1906] Output: Future prediction and advice (scenario based on prompt)

[1907] Step 8: User Interface

[1908] Specific operation: The server provides an interface for users to interact with the generative AI and emotion engine. Web and mobile apps are built using front-end frameworks such as React and Vue.js. Based on user input, past data is visualized and future prediction results are displayed.

[1909] Input: Questions and inquiries from users

[1910] Output: Answers from the generative AI, visualized historical data, and future prediction results

[1911] (Application example 2)

[1912] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1913] While conventional generative AI systems often predict the future based on an individual's past and current information, they have not yet been able to provide advice that takes into account the user's emotional state or purchasing history, making it difficult to provide specific, individually customized suggestions.In addition, there has been a lack of systems that automatically make optimal suggestions based on the user's emotions and consumption trends in electronic payments and promotions.

[1914] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from personal devices and online accounts, means for storing the collected data in large-scale cloud storage in chronological order, means for analyzing the stored data and classifying it using natural language processing and image recognition technology, means for using a generation AI to provide personal future predictions and advice based on the analyzed data, emotion analysis means for recognizing the user's emotions, analyzing the data, and reflecting the data in the advice provided by the generation AI, means for suggesting optimal payment methods and promotions based on the purchase history and emotion data collected from the personal devices and online accounts, and means for predicting future consumption trends based on the personal purchasing patterns and emotion data. This enables users to receive optimal advice that takes into account their emotional state and purchasing activities.

[1915] "Generative AI" is a technology that uses artificial intelligence to learn from data and generate new information and predictions.

[1916] "Emotion analysis" is a method of recognizing a user's emotional state from data such as text and voice, and quantitatively evaluating that emotion.

[1917] A "time series database" is a database for organizing and storing collected data in chronological order.

[1918] "Natural language processing" is a technology that allows computers to process and understand human language.

[1919] "Image recognition" is a technology for analyzing image data and recognizing its content and characteristics.

[1920] An "interface" is the means by which a user interacts with or obtains information from a system.

[1921] "Cloud storage" is a storage service for storing and managing large amounts of data via the Internet.

[1922] "Purchase history" is a record of purchases made by a user in the past.

[1923] "Promotion" is a marketing activity to promote the sale of a product or service.

[1924] "Consumption trends" refers to patterns and tendencies of users' purchasing behavior.

[1925] "Schedule management" is a management method for collecting and updating data on a regular basis.

[1926] Embodiments of the present invention will be described in detail below.

[1927] System Overview

[1928] This system combines generative AI and a sentiment analysis engine to provide information on a user's past, present, and future, and to suggest optimal payment methods and promotions for individual electronic payment services.

[1929] Data collection

[1930] The server collects data from users' devices and online accounts, including their past purchase history, spending patterns, current financial situation, and emotional state. The data is collected automatically and periodically, and stored in chronological order in large-scale cloud storage. Specific hardware used includes smartphones and cloud storage servers (e.g., AWS, Google Cloud).

[1931] Data analysis

[1932] The stored data is analyzed by the server using natural language processing (NLP) and image recognition techniques to categorize, tag, and index the data. TensorFlow is used for image recognition, and the results are fed into a generative AI and sentiment analysis engine.

[1933] Emotion analysis

[1934] The server analyzes the user's text data and dialogue history and uses an emotion analysis engine to identify their emotional state. Specifically, it uses the NLTK library to track changes in emotion, and the generated AI uses that data.

[1935] Future predictions and advice

[1936] The server provides future predictions and advice based on the data analyzed using generative AI. For example, it suggests optimal payment methods and promotions based on the user's purchasing history and emotional data. It uses a generative AI model (e.g., GPT-3) to generate advice customized for each user.

[1937] Interface

[1938] The interface is implemented as a web or mobile app, allowing users to input questions or inquiries and receive answers from the generative AI. It is also designed to visualize past data and future predictions so that users can easily check them. For example, if a user inputs, "I'm worried about my recent spending. Please tell me which credit card I should use," the server will display advice generated by the generative AI based on the appropriate data.

[1939] Examples and prompts

[1940] For example, if a user types, "I'm worried about my recent spending. Can you tell me which credit card I should use?", the following prompt is sent to the generating AI:

[1941] "I'm worried about my recent spending. Which credit card should I use?"

[1942] Purchase History: All purchase data from last year

[1943] Sentiment data: Anxiety score for recent spending

[1944] Based on this, the AI ​​generates advice such as, "Since your expenses have increased recently, we recommend that you use a low-interest credit card. Also, there are currently promotions available that offer cashback and points." and provides this to the user.

[1945] The above is an embodiment of the present invention, which allows a user to receive optimal advice that takes into account their emotional state and purchasing behavior.

[1946] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1947] Step 1:

[1948] Users access the system via a smartphone or web interface and input their questions or concerns. An example of input could be text data such as, "I'm worried about my recent spending. Please tell me which credit card I should use."

[1949] Input: User question text

[1950] Output: The user's question is sent to the server

[1951] Step 2:

[1952] The server automatically collects data from users' devices and online accounts, including their purchasing history, spending patterns, and financial status, and stores the collected data in a time-series database.

[1953] Input: User credentials and online account data

[1954] Output: Purchase history and spending patterns recorded in a time series database

[1955] Step 3:

[1956] The server analyzes the stored data and categorizes, tags, and indexes it using natural language processing (NLP) and image recognition technologies. Text data is semantically analyzed using NLP technology, and image data is analyzed using image recognition.

[1957] Input: Purchase history and spending pattern data obtained from a time series database

[1958] Output: A classified and tagged dataset

[1959] Step 4:

[1960] The server analyzes the acquired text data and dialogue history using an emotion analysis engine to identify the user's emotional state, and calculates an emotion score using the NLTK library.

[1961] Input: User text data and interaction history

[1962] Output: Sentiment score

[1963] Step 5:

[1964] The server uses a generative AI model (such as GPT-3) to generate future predictions and optimal advice based on the emotion data obtained from the emotion analysis engine and the stored purchase history data, and sends specific prompts to the generative AI.

[1965] Input: Emotion data and purchase history data

[1966] Output: Future prediction results and advice from generative AI

[1967] Step 6:

[1968] The server returns the generated advice to the user and visualizes it through an interface, allowing the user to easily browse the advice.

[1969] Input: Generated advice data

[1970] Output: Advice displayed on the interface

[1971] The above are the specific processing steps of the system that realizes the application example.

[1972] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1973] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1974] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1975] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1976] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1977] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1978] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1979] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1980] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1981] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1982] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1983] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1984] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1985] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1986] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1987] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1988] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1989] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1990] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1991] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1992] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1993] The following is further disclosed regarding the above embodiment.

[1994] (Claim 1)

[1995] A system that uses generative AI to generate information about an individual's past, present, and future, and to predict their future.

[1996] The means by which data is collected from personal devices and online accounts;

[1997] A means of storing the collected data in chronological order in large-scale cloud storage;

[1998] means for analyzing and classifying the stored data using natural language processing and image recognition techniques;

[1999] A means of using generative AI to provide personalized future predictions and advice based on the analyzed data;

[2000] A system including an interface means for managing a dialogue with a user and providing the user with answers from the generative AI.

[2001] (Claim 2)

[2002] 10. The system of claim 1,

[2003] A system that includes a scheduler for automatically and periodically collecting data from personal devices and online accounts.

[2004] (Claim 3)

[2005] 10. The system of claim 1,

[2006] The system includes a means for searching and displaying events and photos for a specific period based on user data.

[2007] "Example 1"

[2008] (Claim 1)

[2009] A system for obtaining past information, current information, and future predictions for an individual, comprising:

[2010] The means by which data is collected from personal devices and online accounts;

[2011] a means for storing the collected data in a distributed storage in chronological order;

[2012] means for analyzing the stored data and categorizing and tagging it using natural language processing and image recognition techniques;

[2013] A means for providing personalized future predictions and advice based on the analyzed data using artificial intelligence;

[2014] A system including a display means for managing a dialogue with a user and providing answers from an artificial intelligence to the user.

[2015] (Claim 2)

[2016] 2. The system of claim 1, wherein the data collection means includes a schedule management function that automatically and periodically collects data from personal devices and online accounts.

[2017] (Claim 3)

[2018] 10. The system of claim 1, wherein the display means includes a function for searching and visualizing events and images for a specific period based on user data.

[2019] "Application Example 1"

[2020] (Claim 1)

[2021] A system that uses generative AI to generate information about an individual's past, present, and future, and to predict their future.

[2022] The means by which data is collected from personal devices and online accounts;

[2023] A means of storing the collected data in chronological order in large-scale cloud storage;

[2024] means for analyzing and classifying the stored data using natural language processing and image recognition techniques;

[2025] A means of using generative AI to provide personalized future predictions and advice based on the analyzed data;

[2026] an interface means for managing a dialogue with a user and providing the user with answers from the generating AI;

[2027] A means for analyzing past purchase history and browsing history, predicting future purchase trends, and generating personalized product recommendations;

[2028] a means for visually presenting the generated product recommendations to a user;

[2029] ...

[2030] A system including:

[2031] (Claim 2)

[2032] 10. The system of claim 1, further comprising a schedule management means for automatically and periodically collecting data from personal devices and online accounts.

[2033] (Claim 3)

[2034] 10. The system of claim 1, further comprising means for searching for events and photos for a specific period based on user data and displaying visualized results.

[2035] "Example 2: Combining Emotion Engines"

[2036] (Claim 1)

[2037] A system that uses generative AI to generate information about an individual's past, present, and future, and to predict their future.

[2038] The means by which data is collected from personal communication devices and online accounts;

[2039] A means of storing the collected data in chronological order in large-scale cloud storage;

[2040] means for analyzing and classifying the stored data using natural language processing and image recognition techniques;

[2041] a means for inputting the analysis results into an emotion engine to identify the emotional state of the individual;

[2042] A means for using generative AI to provide personalized future predictions and advice based on the analyzed data and sentiment data;

[2043] A system including an interface means for managing a dialogue with a user and providing the user with answers from the generative AI.

[2044] (Claim 2)

[2045] 10. The system of claim 1, further comprising a schedule management means for automatically and periodically collecting data from personal communication devices and online accounts.

[2046] (Claim 3)

[2047] 10. The system of claim 1, further comprising means for searching and displaying events and image data for a specific period based on user data.

[2048] "Application example 2 when combining emotion engines"

[2049] (Claim 1)

[2050] A system that uses generative AI to generate information about an individual's past, present, and future, and to predict their future.

[2051] The means by which data is collected from personal devices and online accounts;

[2052] A means of storing the collected data in chronological order in large-scale cloud storage;

[2053] means for analyzing and classifying the stored data using natural language processing and image recognition techniques;

[2054] A means of using generative AI to provide personalized future predictions and advice based on the analyzed data;

[2055] an interface means for managing a dialogue with a user and providing the user with answers from the generating AI;

[2056] An emotion analysis method that recognizes the user's emotions, analyzes the data, and reflects it in the advice provided by the generation AI;

[2057] A way to suggest optimal payment methods and promotions based on purchase history and sentiment data collected from personal devices and online accounts;

[2058] A system that includes a means of predicting future consumption trends based on an individual's purchasing patterns and sentiment data.

[2059] (Claim 2)

[2060] 10. The system of claim 1, further comprising a schedule management means for automatically and periodically collecting data from personal devices and online accounts.

[2061] (Claim 3)

[2062] 10. The system of claim 1, further comprising means for searching and displaying events and photos for a specific period based on user data. [Explanation of symbols]

[2063] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A system that uses generative AI to generate information about an individual's past, present, and future, and to predict their future. The means by which data is collected from personal devices and online accounts; A means of storing the collected data in chronological order in large-scale cloud storage; means for analyzing and classifying the stored data using natural language processing and image recognition techniques; A means of using generative AI to provide personalized future predictions and advice based on the analyzed data; A system including an interface means for managing a dialogue with a user and providing the user with answers from the generative AI.

2. 10. The system of claim 1, A system that includes a scheduler for automatically and periodically collecting data from personal devices and online accounts.

3. 10. The system of claim 1, The system includes a means for searching and displaying events and photos for a specific period based on user data.

Images