System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting past data of a user from a cloud storage; means for analyzing the collected data; and means for providing a service in a plurality of modes according to a selection of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people generate large amounts of data in their daily lives in the form of photos, audio, memos, emails, and so on. However, this data is simply stored, and once it has disappeared from memory, there are few opportunities to relive those memories. In addition, individual pieces of data are scattered, making it difficult to systematically extract and share specific memories. Furthermore, people who do not have anyone to talk to, such as the elderly or those living alone, feel lonely because they have no one to talk to about their memories.

[0005] On the other hand, it is technically difficult to generate documents that reflect one's own thought patterns and personality, such as when one wants to show others a portrait or copy of oneself, or when one is required to express oneself in a specific situation. This has led to a demand for documents that reflect individual characteristics and personality, for example, when creating speech manuscripts or messages to be left after death. [Means for solving the problem]

[0006] The present invention provides a system that effectively utilizes users' past data stored in cloud storage to provide services tailored to each user's purpose.The present invention solves the above-mentioned problems by the following means.

[0007] First, it acquires permission from the user and provides a means to collect past data such as photos, audio, notes, and emails from cloud storage. Second, it analyzes the collected data and allows the user to choose between a memory generation mode and a personality reproduction mode.

[0008] In the reminiscence generation mode, the app generates reminiscences based on the user's past events and memories based on the collected data. Specifically, it analyzes data such as photos and notes, and generates related reminiscences in real time using a natural language generation model.

[0009] In the personality reproduction mode, the system analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. For example, it has a means to generate sentences that reflect the user's characteristics, such as speeches and messages.

[0010] This allows users to share past memories and generate text that reflects their own characteristics, according to their needs at the time, making it possible to bring out and share individual memories and personalities even more.

[0011] "Cloud storage" is a remote server for storing and managing data over the Internet.

[0012] "User" refers to a person who uses this system and provides the data.

[0013] "Data collection means" refers to the function of collecting information such as photos, audio, notes, and emails from cloud storage.

[0014] "Data analysis means" refers to the function of analyzing collected data and extracting and organizing specific information.

[0015] "Memories Generation Mode" refers to the function that generates memories based on the user's past photos, audio, notes, and emails.

[0016] "Personality reproduction mode" refers to a function that analyzes the user's thought patterns, speaking style, and writing style, and generates sentences based on the results.

[0017] A "natural language generation model" is an algorithm or machine learning model for generating human language based on input text data.

[0018] "Analyzed Data" refers to information collected by the data collection means and analyzed by the data analysis means.

[0019] A "machine learning model" is an algorithm or artificial intelligence technique that uses large amounts of data to learn specific patterns or rules.

[0020] "Real-time" means that processing is nearly instantaneous and results are provided immediately.

[0021] A "question" is text that a user enters into a system to request specific information or an answer.

[0022] "Authorization" is the act of a user giving permission to the system for a specific operation or use of data. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] This invention is a system that collects data such as past photos, audio, memos, and emails stored in a user's cloud storage, analyzes the data, and provides a service that recreates memories and the user's personality in various modes.

[0045] System configuration

[0046] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0047] 1. Data Collection Methods

[0048] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[0049] 2. Data analysis methods

[0050] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[0051] 3. Memories Generation Mode

[0052] The server generates a story based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server uses a natural language generation model to generate an answer about the photo's location, date, and time, as well as related episodes. The device then displays the generated answer to the user.

[0053] 4. Personality Reproduction Mode

[0054] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on these. For example, if a user requests "Create a retirement speech," the server learns the user's style from past emails and notes and generates a speech based on that. The generated speech is sent to the user's device and displayed.

[0055] Specific examples

[0056] 1. Example of the reminiscence generation mode

[0057] The user starts the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the metadata of the corresponding photo from cloud storage, generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were very beautiful," sends the answer to the device, and displays it to the user.

[0058] 2. Specific examples of personality reproduction mode

[0059] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[0060] User Interface and Operation

[0061] User Interface

[0062] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question entry screen, and a results display screen.

[0063] Operating Procedure

[0064] 1. The user launches the app and logs in.

[0065] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0066] 3. The user enters and submits a question or request.

[0067] 4. The server parses the data and generates the results.

[0068] 5. The results are displayed on the device and feedback is given to the user.

[0069] In this way, users can enjoy reminiscing about the past and generate sentences that reflect their own personalities. This system allows users to effectively utilize past data and enjoy richer life experiences.

[0070] The processing flow will be explained below.

[0071] Processing steps of data collection instruments

[0072] Step 1:

[0073] The user taps on the app icon to launch it, and the app is launched.

[0074] Step 2:

[0075] The user enters their ID and password on the login screen and presses the "Login" button.

[0076] Step 3:

[0077] The server checks the entered authentication information against a database and performs authentication.

[0078] Step 4:

[0079] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[0080] Step 5:

[0081] The device displays a permission dialog to the user requesting permission to collect data.

[0082] Step 6:

[0083] The user selects "Allow" in the permission dialog.

[0084] Step 7:

[0085] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[0086] Step 8:

[0087] The server stores the acquired data in storage and prepares it for analysis.

[0088] Processing steps of data analysis method

[0089] Step 1:

[0090] The server retrieves the collected data for analysis.

[0091] Step 2:

[0092] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[0093] Step 3:

[0094] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[0095] Step 4:

[0096] The server classifies and organizes the extracted information and stores it in a database so that it can be used to generate memories and recreate personalities.

[0097] Processing steps for reminiscence generation mode

[0098] Step 1:

[0099] The user selects "reminiscence mode" on the mode selection screen.

[0100] Step 2:

[0101] The user enters a question or request into the question input screen and presses the "Submit" button.

[0102] Step 3:

[0103] The terminal sends the user's question to the server.

[0104] Step 4:

[0105] The server analyzes the user's question using a natural language processing model.

[0106] Step 5:

[0107] Based on the analysis results, the server searches the database for relevant memories.

[0108] Step 6:

[0109] The server uses the reminiscence generation model to generate reminiscences in response to questions from the user.

[0110] Step 7:

[0111] The terminal displays the reminiscences received from the server to the user.

[0112] Personality Reproduction Mode Processing Steps

[0113] Step 1:

[0114] The user selects "personality reproduction mode" on the mode selection screen.

[0115] Step 2:

[0116] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[0117] Step 3:

[0118] The terminal transmits the user's request to the server.

[0119] Step 4:

[0120] The server retrieves relevant data to analyze the user's thinking patterns, writing style, and speaking style.

[0121] Step 5:

[0122] The server uses machine learning models to generate sentences based on the user's style.

[0123] Step 6:

[0124] The server sends the generated text to the terminal.

[0125] Step 7:

[0126] The terminal displays the generated text to the user.

[0127] Example 1

[0128] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0129] In today's digitalized society, users store vast amounts of data (photos, audio, notes, emails, etc.) in cloud storage. However, this data simply functions as a storage location and is not used for specific services. It is also difficult to effectively utilize past data to reminisce or recreate a user's writing or speaking style. This prevents users from fully utilizing the potential value of their data.

[0130] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0131] In this invention, the server includes means for collecting data from cloud storage based on the user's access permission, means for analyzing the collected data and extracting metadata and content, means for generating and providing information from the analyzed data in response to the user's request, means for using a natural language generation model to generate reminiscences, and means for learning the user's writing and speaking style and generating text, thereby enabling the user to effectively utilize past data and enjoy a richer digital experience.

[0132] "User" means any person or entity that uses the System.

[0133] "Access permission" refers to the act of authentication by a user to allow the server to access their cloud storage.

[0134] "Cloud storage" refers to an online storage service for storing user data via the Internet.

[0135] "Data" refers to information such as photos, audio, notes, and emails stored in the user's cloud storage.

[0136] "Server" refers to a computer and related hardware and software that executes various processes such as collection, analysis, and generation.

[0137] "Means" refers to the processes, functions, or devices necessary to achieve a certain purpose.

[0138] "Collection" refers to the act of obtaining specific data from cloud storage.

[0139] "Analysis" refers to the process of understanding the content of collected data and extracting metadata and important information.

[0140] "Metadata" refers to additional information about the data (e.g., the date and location of a photo).

[0141] "Information generation" refers to the act of creating new information based on analyzed data.

[0142] "Memories generation mode" refers to a mode that provides the function of generating memories based on past data.

[0143] A "natural language generation model" refers to a machine learning model that generates natural language expressions based on input text information.

[0144] "Learning how to write and speak" refers to the process of analyzing the user's past writing and voice data and learning their characteristics.

[0145] "Text generation" refers to the act of creating new text based on learned features.

[0146] This invention is a system that effectively utilizes past data stored in a user's cloud storage to provide a service that recreates reminiscences and the user's writing and speaking styles. The system is composed of a data collection means, a data analysis means, a server that includes a reminiscence generation mode and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0147] Data collection methods

[0148] The server obtains the user's permission to collect data from cloud storage. To collect data, it uses common cloud storage APIs such as Google Drive API and Dropbox API. Specifically, after the user launches the app and logs in, the server securely downloads the data by granting permission to access the cloud storage.

[0149] Data Analysis Methods

[0150] The server uses machine learning models (e.g. TensorFlow or PyTorch) to analyze the collected data. The following steps are performed:

[0151] Photo data analysis: The server analyzes the photo data and extracts metadata and content using image recognition algorithms (e.g., OpenCV or TensorFlow models), specifically face detection and scene recognition to determine whether a particular face was captured in a particular location.

[0152] Analyzing the audio data: When the server analyzes the audio data, it uses speech recognition technologies such as Google Cloud Speech-to-Text API or Amazon Transcribe to transcribe the content and extract key keywords and phrases.

[0153] Memo / Email Analysis: The server analyzes text data (memos, emails) and extracts important information using natural language processing techniques (e.g., SpaCy or NLTK).

[0154] Memories generation mode

[0155] When a user selects the reminiscence mode, the server generates a reminiscence based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server generates an answer using a natural language generation model (e.g., GPT-3) based on the metadata of the corresponding photo and related anecdotes. The generated answer is sent to the device and displayed to the user.

[0156] Specific examples

[0157] Example of memory generation mode

[0158] The user launches the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the photo's metadata and generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken as part of a family trip," which is sent to the device and displayed to the user.

[0159] Example prompt for a generative AI model:

[0160] "Generate an episode about a photo taken by a user on a family trip in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Tell them that the lavender fields were very beautiful."

[0161] Personality Reproduction Mode

[0162] When a user selects the personality mode, the server learns the user's writing and speaking style based on the analyzed memo and email data. For example, if a user requests to "write a birthday message for a friend," the server analyzes past emails and memos and generates a message based on the user's style.

[0163] Specific examples

[0164] Specific examples of personality reproduction mode

[0165] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[0166] Example prompt for a generative AI model:

[0167] "We've learned about the user's writing and speaking style from their past emails and notes. Generate the following birthday message for a friend. This user prefers to write in a friendly, warm style."

[0168] User Interface and Operation

[0169] The user interface includes a login screen, a mode selection screen, a question input screen, and a result display screen. The user interacts with the system through these screens. The operation procedure is as follows:

[0170] 1. The user launches the app and logs in.

[0171] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0172] 3. The user enters and submits a question or request.

[0173] 4. The server parses the data and generates the results.

[0174] 5. The results are displayed on the device and feedback is given to the user.

[0175] In this way, users can utilize past data, reminisce, and generate texts that suit their own style. This system allows users to effectively utilize data stored in cloud storage and enjoy a richer digital experience.

[0176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0177] Step 1: User launches the app and logs in.

[0178] Input: User login information (username, password)

[0179] Data processing / calculation: The server verifies and authenticates the login information, allowing the user to access cloud storage.

[0180] Output: Authentication success message and app home screen

[0181] Step 2: The user grants permission to access the cloud storage.

[0182] Input: A dialog where the user asks for permission to access a cloud storage service

[0183] Data processing / calculation: The server authenticates using OAuth2.0 and establishes a secure connection to the user's cloud storage.

[0184] Output: Obtain cloud storage access permission and display the app mode selection screen.

[0185] Step 3: The user selects "reminiscence mode" or "personality reproduction mode."

[0186] Input: User mode selection (reminiscence mode or personality mode)

[0187] Data processing / calculation: The server sends instructions to the device to display the UI according to the selected mode.

[0188] Output: Screen display according to selected mode

[0189] Step 4: The user enters and submits a question or request.

[0190] Input: A user-typed question or request (e.g., "Where was this photo taken?")

[0191] Data processing / calculation: The server analyzes the user's input and performs appropriate data collection and analysis processing.

[0192] Output: Analysis results of the question or request

[0193] Step 5: The server collects the data from the cloud storage.

[0194] Input: User cloud storage access permissions, data information to be collected

[0195] Data processing / calculation: The server uses the Google Drive API or Dropbox API to download data such as photos, audio, notes, and emails.

[0196] Output: Collected data (photos, audio, notes, emails)

[0197] Step 6: The server analyzes the collected data.

[0198] Input: Collected data

[0199] Data processing / calculation: The server uses machine learning models (e.g., TensorFlow, PyTorch) to perform the following analysis:

[0200] Photo data analysis: Extract metadata and content using image recognition algorithms

[0201] Analysis of audio data: Transcription and content extraction using speech recognition technology

[0202] Memo and email analysis: Extracting important information using natural language processing techniques

[0203] Output: Analyzed data (metadata, transcription results, extracted key information)

[0204] Step 7: The server generates the reminiscence and text.

[0205] Input: Analyzed data, user questions and requests

[0206] Data processing / computation: Generate reminiscences and text using a natural language generation model (e.g., GPT-3). This involves feeding prompts into the generative AI model.

[0207] Specific Action: Example: "Generate an episode about a photo of a family trip taken by a user in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Please tell us that the lavender fields were very beautiful."

[0208] Output: Generated reminiscences and text

[0209] Step 8: Provide the generated results to the user.

[0210] Input: Generated reminiscences and text

[0211] Data processing / calculation: The server sends the generated data to the terminal and displays it on the screen.

[0212] Output: Reminiscences and text displayed on the user's terminal

[0213] In this way, by clarifying the specific operations and the associated data input and output at each processing step, users can make full use of past data through the system, enjoy reminiscing, and generate individual sentences.

[0214] (Application example 1)

[0215] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0216] In recent years, the amount of data stored in cloud storage has increased, but methods for effectively utilizing this data are limited. There is also a need for methods to provide personalized services using users' past data and systems that generate accurate responses to user requests. In particular, for food delivery services, functions such as recommendations based on past order history and message generation for special occasions are effective. The objective of this invention is to realize a system that analyzes and utilizes data in cloud storage to provide users with reminiscences, personalized recommendations, and message generation based on personality reproduction.

[0217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0218] In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, and means for making individualized recommendations based on the data analysis, thereby enabling personalized recommendations based on the user's past data, the generation of stories of memories, and the generation of messages for special occasions.

[0219] "Cloud storage" is an online storage service for storing and managing data via the Internet.

[0220] "Data collection means" refers to a means for obtaining a user's past data from cloud storage.

[0221] "Data analysis means" refers to means for analyzing collected data and extracting necessary information.

[0222] The "means for providing a service" is a means for providing a service in a plurality of modes in response to a user's selection.

[0223] "Personalized recommendations" means making personalized recommendations to users based on data analysis.

[0224] The "mode for generating reminiscences" is a mode for generating reminiscences based on the user's past photos, voice, memos and emails.

[0225] "Personality reproduction mode" is a mode that reproduces the user's thought patterns, writing style, and speaking style based on their past data.

[0226] This invention is a system that collects and analyzes data stored in a user's cloud storage, and provides reminiscences, personalized recommendations, and message generation that recreates the user's personality. Specific embodiments are described below.

[0227] System configuration

[0228] This system is composed of a server including a data collection means, a data analysis means, a service provision means, and a personalized recommendation means, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0229] Data collection methods

[0230] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. In this process, protecting the user's privacy and consent to data collection are important.

[0231] Data Analysis Methods

[0232] The server analyzes the collected data and uses machine learning models to extract the necessary information. This allows it to accurately analyze photo metadata, audio content, and note content. It uses the ImageAI library (ResNet model) for image classification and textgenrnn for text analysis.

[0233] Service delivery and personalized recommendation methods

[0234] The server provides services in three modes depending on the user's selection: memory generation mode, personality reproduction mode, and personalized recommendation mode, allowing the user to analyze past order data and messages to obtain a personalized customer experience.

[0235] Memories generation mode

[0236] When a user asks about a specific photo, "Where was this photo taken?", the server analyzes the photo's metadata from cloud storage and generates a story about the photo, such as "This photo was taken in Biei, Hokkaido in the summer of 2018. It was taken during a family trip, and the lavender fields were beautiful," which is then displayed on the device.

[0237] Personality Reproduction Mode

[0238] When a user requests a retirement speech, the server analyzes past emails and memos to learn the user's writing and speaking style. Based on this, it generates a speech such as, "Thank you all for joining us today. I am truly grateful to have had the opportunity to work with you all," and displays it on the device.

[0239] Personalized recommendation mode

[0240] Based on data on the food a user has ordered in the past, the system makes personalized recommendations such as, "Did you like the pizza you ordered for your birthday last year?" or "How about this new menu item?" The server analyzes the user's past order history and preferences, generates appropriate recommendations, and displays them on the device.

[0241] User Interface and Operation

[0242] Users interact with the system through terminal applications, which include a login screen, a mode selection screen, a question entry screen, and a result display screen.

[0243] Operating Procedure

[0244] 1. The user launches the app and logs in.

[0245] 2. The user selects "Memories mode," "Personality reproduction mode," or "Individualized recommendation mode" on the mode selection screen.

[0246] 3. The user enters and submits a question or request.

[0247] 4. The server analyzes the data and generates results.

[0248] 5. The generated results are displayed on the device and feedback is provided to the user.

[0249] Prompt Sentence Examples

[0250] When a user uploads a photo and asks, "Tell me what memories you have of this photo," the server analyzes the photo and tells them, "This photo is of the pizza I ordered for my birthday last year, and it was delicious."

[0251] In this way, users can enjoy past memories and receive personalized recommendations and thoughtful messages. This system allows users to effectively utilize past data and enjoy a richer experience.

[0252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0253] Step 1:

[0254] The user starts the application on the terminal and logs in.

[0255] Input: User login information (ID, password).

[0256] Output: Notifies the success or failure of user authentication.

[0257] Specific operation: The user enters their ID and password into the login screen, and the server receives this and performs authentication.

[0258] Step 2:

[0259] The user selects "reminiscence mode," "personality reproduction mode," or "individualized recommendation mode" on the mode selection screen.

[0260] Input: User mode selection.

[0261] Output: The screen of the selected mode.

[0262] Specific operation: The user selects a mode according to the purpose, and the server prepares for data collection and analysis accordingly.

[0263] Step 3:

[0264] The user enters and submits a question or request.

[0265] Input: The user's question or request (e.g., "Tell me what you remember about this photo").

[0266] Output: Success or failure of sending data to the server.

[0267] Specific operation: The application sends the question or request entered by the user to the server.

[0268] Step 4:

[0269] The server collects the user's data from the cloud storage.

[0270] Input: User permissions and cloud storage access information.

[0271] Output: Collected data such as photos, audio, notes, emails, etc.

[0272] Specific operation: The server retrieves the required data using the cloud storage API.

[0273] Step 5:

[0274] The server analyzes the collected data.

[0275] Input: Data collected in step 4.

[0276] Output: Information based on the parsed metadata and content.

[0277] Specific operation: The ResNet model from the ImageAI library is used for image classification, and textgenrnn is used for text analysis, analyzing photo metadata, audio content, note contents, etc.

[0278] Step 6:

[0279] The server generates a service based on the analysis results.

[0280] Input: Analyzed data and user questions or requests.

[0281] Output: Generated memories, personality reenactments, and personalized recommendations.

[0282] Specific behavior: For example, in the reminiscence mode, it generates a narrative such as, "This photo was taken in Biei, Hokkaido in the summer of 2018."

[0283] Step 7:

[0284] The server generates a response that is sent to the terminal and displayed to the user.

[0285] Input: Generated reminiscences, personality reenactments, and personalized recommendations.

[0286] Output: The results that are displayed to the user on the application screen.

[0287] Specific operation: The server sends the generated content to the terminal, and the application displays it.

[0288] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0289] This invention combines an emotion engine with a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. In particular, the aim is to improve the quality and user experience of the memory generation mode and personality reproduction mode by recognizing the user's emotions.

[0290] System configuration

[0291] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, a personality reproduction mode, and an emotion engine, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0292] 1. Data Collection Methods

[0293] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[0294] 2. Data analysis methods

[0295] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[0296] 3. Emotion Engine

[0297] The server is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their current emotional state and generate an appropriate response.

[0298] 4. Memories Generation Mode

[0299] The server generates a reminiscence based on the analyzed data. It also takes into account the data from the emotion engine and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[0300] 5. Personality Reproduction Mode

[0301] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[0302] Specific examples

[0303] 1. Example of the reminiscence generation mode

[0304] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[0305] 2. Specific examples of personality reproduction mode

[0306] The user launches the app, logs in, and selects "Personality Reproduction Mode." The emotion engine recognizes that the user's tone of voice indicates nervousness. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, the server generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for everything you've done. I'd especially like to thank Tanaka-san, who helped me with Project X. I will continue to walk a new path, keeping the time we spent together in my heart. I'm a little nervous, but I truly appreciate it." The speech is then sent to the device and displayed to the user.

[0307] User Interface and Operation

[0308] User Interface

[0309] Users interact with the system through a device app, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. It may also display real-time emotion analysis results from an emotion engine.

[0310] Operating Procedure

[0311] 1. The user launches the app and logs in.

[0312] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0313] 3. The user enters and submits a question or request.

[0314] 4. The server analyzes the data and generates results using the emotion engine.

[0315] 5. The results are displayed on the device and feedback is given to the user.

[0316] In this way, users can share their past memories while feeling emotionally attached to them, and generate sentences that correspond to their emotional state.This system allows users to effectively utilize past data and enjoy richer life experiences.

[0317] The processing flow will be explained below.

[0318] Processing steps of data collection instruments

[0319] Step 1:

[0320] The user taps on the app icon to launch it, and the app is launched.

[0321] Step 2:

[0322] The user enters their ID and password on the login screen and presses the "Login" button.

[0323] Step 3:

[0324] The server checks the entered authentication information against a database and performs authentication.

[0325] Step 4:

[0326] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[0327] Step 5:

[0328] The device displays a permission dialog to the user requesting permission to collect data.

[0329] Step 6:

[0330] The user selects "Allow" in the permission dialog.

[0331] Step 7:

[0332] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[0333] Step 8:

[0334] The server stores the acquired data in storage in preparation for analysis.

[0335] Processing steps of data analysis method

[0336] Step 1:

[0337] The server retrieves the collected data and begins analyzing it.

[0338] Step 2:

[0339] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[0340] Step 3:

[0341] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[0342] Step 4:

[0343] The server classifies the extracted information and stores it in a database.

[0344] Emotion Engine Processing Steps

[0345] Step 1:

[0346] The device captures the user's voice and facial expressions in real time.

[0347] Step 2:

[0348] The device sends the captured data to the server.

[0349] Step 3:

[0350] The server analyzes the received voice and facial expression data.

[0351] Step 4:

[0352] The server uses an emotion engine to recognize the user's emotional state (e.g., joy, sadness, tension, etc.).

[0353] Step 5:

[0354] The emotional data analyzed by the emotion engine is reflected in related processing (memory story generation and personality reproduction).

[0355] Processing steps for reminiscence generation mode

[0356] Step 1:

[0357] The user selects "Memories mode" on the mode selection screen within the app.

[0358] Step 2:

[0359] The user enters a question into the question input screen and presses the "Submit" button.

[0360] Step 3:

[0361] The terminal sends the user's question to the server.

[0362] Step 4:

[0363] The server analyzes the user's question using a natural language processing model.

[0364] Step 5:

[0365] The server searches the database for relevant memories based on the analysis results and emotion engine data.

[0366] Step 6:

[0367] The server uses a reminiscence generation model to generate a reminiscence that is in line with the user's emotions.

[0368] Step 7:

[0369] The terminal displays the reminiscences received from the server to the user.

[0370] Personality Reproduction Mode Processing Steps

[0371] Step 1:

[0372] The user selects "Personality Reproduction Mode" on the mode selection screen within the app.

[0373] Step 2:

[0374] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[0375] Step 3:

[0376] The terminal transmits the user's request to the server.

[0377] Step 4:

[0378] The server retrieves relevant past data and analyzes the user's thought patterns, writing style, and speaking style.

[0379] Step 5:

[0380] The server takes into account the data from the emotion engine and generates appropriate sentences according to the user's emotional state.

[0381] Step 6:

[0382] The server sends the generated text to the terminal.

[0383] Step 7:

[0384] The terminal displays the generated text to the user.

[0385] Example 2

[0386] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] The purpose of this invention is to provide a richer user experience by utilizing the user's past data stored in cloud storage. However, conventional technologies simply display data, making it difficult to provide services that take the user's emotional state into account. Furthermore, there is a problem in that the quality of reminiscences and personality reproduction is low because emotions are not recognized.

[0388] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means for providing an emotion engine for analyzing emotions from the user's voice and facial expressions, means for generating reminiscences based on the user's emotional state, and means for generating sentences based on the user's writing and speaking style. This makes it possible to provide high-quality reminiscences according to the user's emotional state and improve the accuracy of personality reproduction.

[0389] "Cloud storage" is a storage service on a remote server that stores and manages data over the Internet.

[0390] "User's past data" refers to various data that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[0391] "Means of collection" refers to the functionality for obtaining the required data from authorized cloud storage.

[0392] "Means of analysis" refers to the ability to extract useful information from collected data using machine learning models and algorithms.

[0393] "Means for providing services" refers to the function of generating content such as reminiscences and personality reenactments based on analyzed data and providing it to users.

[0394] "Emotion engine" refers to technology that analyzes a user's voice and facial expressions to identify their current emotional state.

[0395] "Means for generating reminiscences" refers to a function for automatically generating reminiscences for a user based on analyzed past data and the user's current emotional state.

[0396] "Means of personality reproduction" refers to a function that analyzes the user's past data and generates sentences based on the user's writing and speaking style.

[0397] This invention is a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. By incorporating an emotion engine into this system, the quality and user experience of the reminiscence generation mode and personality reproduction mode can be improved. Specific embodiments are described below.

[0398] System configuration

[0399] This system consists of a server that performs a series of processes, such as data collection means, data analysis means, memory generation mode, personality reproduction mode, and emotion engine, a terminal that acts as a user interface, and a series of means for interaction with the user.

[0400] Data collection methods

[0401] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from cloud storage (e.g., a general cloud storage service). In this process, it is important to protect the user's privacy and obtain consent for data collection.

[0402] Data Analysis Methods

[0403] The server uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the collected data. This allows it to accurately analyze photo metadata, audio content, and note entries. The analysis results are used to generate reminiscences and recreate personalities.

[0404] Emotion Engine

[0405] The server is equipped with an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) that analyzes emotions from the user's voice and facial expressions, allowing it to recognize the user's current emotional state and generate an appropriate response.

[0406] Memories generation mode

[0407] The server generates a reminiscence based on the analyzed data. It also takes into account the emotion engine data and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[0408] Examples:

[0409] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[0410] Personality Reproduction Mode

[0411] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[0412] Examples:

[0413] The user starts the app, logs in, and selects "Personality Mode." The emotion engine recognizes that the user's tone of voice sounds tense. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, it generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for your support over the years. I'm especially grateful to everyone who helped me on this project. I'm truly grateful, but I'm nervous about taking on this new challenge." The speech is then sent to the device and displayed to the user.

[0414] User Interface and Operation

[0415] User Interface:

[0416] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. In addition, the app may display real-time emotion analysis results from an emotion engine.

[0417] Operation steps:

[0418] 1. The user launches the app and logs in.

[0419] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0420] 3. The user enters and submits a question or request.

[0421] 4. The server analyzes the data and generates results using the emotion engine.

[0422] 5. The results are displayed on the device and feedback is given to the user.

[0423] This process allows users to share their past memories while feeling the emotions involved, and generate text that reflects their emotional state. This system allows users to effectively utilize past data and enjoy richer life experiences.

[0424] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0425] Step 1:

[0426] The user launches the app and logs in. The user enters their username and password on the device's login screen and clicks the login button. The server checks the user's authentication information against the database, and if authentication is successful, it generates a JWT (JSON Web Token) and sends it to the device. This starts the user's session. The output is the home screen with the user logged in.

[0427] Step 2:

[0428] The user selects a mode. The user selects "Memories Mode" or "Personality Reproduction Mode" on the home screen. The device sends the user's selection to the server. The server records the mode selected by the user and prepares the system to proceed to the next step. The input is an instruction to select a mode, and the output is confirmation of the selected mode.

[0429] Step 3:

[0430] The server obtains consent to collect data. The server displays a dialog to the user requesting permission to access cloud storage. If the user agrees, the server obtains the necessary data access rights from the cloud storage using the OAuth 2.0 protocol. The input is consent information, and the output is the acquisition of access rights. The server collects photos, audio, notes, emails, etc. from the cloud storage service and stores them in a temporary storage area.

[0431] Step 4:

[0432] The server analyzes the collected data. The server analyzes the data using machine learning models (e.g., TensorFlow, PyTorch). Specific operations include extracting metadata from photos, transcribing audio data, and analyzing the content of notes. The input is the collected data, and the output is the analysis results. The analysis results are used in the next step.

[0433] Step 5:

[0434] The server performs emotion analysis using an emotion engine. The server inputs the user's voice and facial expression data into an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) to analyze the user's emotional state. The input is the user's voice and facial expression data, and the output is data on the user's emotional state (e.g., joy, sadness, tension). This allows the user's current emotional state to be quantified.

[0435] Step 6:

[0436] The server generates content based on emotions. Based on the analysis results and the emotion engine results, the server generates sentences for reminiscences or personality reenactment. To do this, a prompt sentence is input to a generative AI model (e.g., GPT-3, BERT), which then generates an appropriate sentence. The input is the prompt sentence and the analysis results, and the output is the generated sentence.

[0437] Specifically, in the reminiscence mode, the system uses the prompt "Tell me your memories of this photo" and adjusts the emotional tone based on the generated text. In the personality reproduction mode, the system uses the prompt "Write a retirement speech" and adjusts the tone of the generated speech, such as making it more relaxed.

[0438] Step 7:

[0439] The server sends the generated content to the terminal. The server sends the generated text to the terminal in JSON format. The terminal parses the received data and displays it on the user interface. The user can check the displayed results and send feedback accordingly. The input is the data of the generated content, and the output is the user's display screen.

[0440] Through the above steps, users can enjoy high-quality emotional reminiscences and personality reproductions using past data.

[0441] (Application example 2)

[0442] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0443] Conventional robot assistant systems in factories have limited capabilities for providing work guidance and employee support, making it difficult to provide appropriate support that takes into account the emotional state of employees. Therefore, there has been a demand for a method that can simultaneously promote efficient work and reduce employee stress.

[0444] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of users from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means having an emotion engine that analyzes facial expressions and voice in real time, and means for providing appropriate instructions and support according to the user's emotions. This enables efficient work guidance and support for employees while taking into account their emotional state.

[0445] "Cloud storage" is a technology that provides remote servers for storing data over the Internet.

[0446] "User's past data" refers collectively to information that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[0447] "Means of collection" refers to the processes and tools used to obtain the required data from cloud storage.

[0448] "Means of analysis" are methods and techniques for analyzing collected data and extracting useful information.

[0449] "Multiple modes" refer to different types of services that the system can provide, including, for example, a reminiscence generation mode and a personality reproduction mode.

[0450] The "emotion engine" is a technology that analyzes the user's emotional state from voice and facial expression data.

[0451] "Real-time facial expression and voice analysis" refers to the ability to instantly assess a user's current emotional state.

[0452] "Means for providing instructions and support" refers to mechanisms for providing specific actions or advice to users based on the analysis results.

[0453] The "mode for generating memories" is a function that generates stories that evoke memories based on the user's past data.

[0454] "Personality Reproduction Mode" is a function that imitates the user's personality based on their past actions and documents, and generates responses in real time.

[0455] The "work instruction mode" is a function in which the robot instructs employees on how to do their work in a factory or other environment.

[0456] "Production history analysis mode" is a function that analyzes past data to evaluate employee work performance.

[0457] "Voicing" refers to the act of the robot providing verbal advice or instructions depending on the employee's emotional state.

[0458] This paper describes an embodiment of a system equipped with an emotion engine for factory robots that incorporates a "production history analysis mode" and a "work guidance mode." This system uses a robot assistant in a factory to analyze the emotional state of employees in real time and provide appropriate guidance and support. The main hardware and software configurations and their processing flow are described.

[0459] Hardware Configuration

[0460] 1. Factory robots

[0461] Supports all operations required for operation.

[0462] 2. Camera and microphone

[0463] Capture employees' facial expressions and voices.

[0464] 3. Cloud Storage

[0465] A remote server for storing employee historical data.

[0466] Software Configuration

[0467] 1. Machine Learning Model

[0468] TensorFlow and Google Cloud AI are used for data analysis.

[0469] 2. Emotion Engine

[0470] Affectiva and Microsoft Azure Emotion API are used as technologies to analyze emotions from voice and facial expressions.

[0471] 3. Robot Control Software

[0472] Control the robot's movements using ROS (Robot Operating System).

[0473] 4. User Interface

[0474] Users interact with the system via factory tablets and displays.

[0475] Program processing flow

[0476] 1. Data Collection

[0477] The server collects users' past data from cloud storage, including photos, audio, notes, and emails.

[0478] Cameras and microphones are used to collect employees' facial expressions and voices in real time.

[0479] 2. Data Analysis

[0480] The collected data is analyzed using machine learning models on a cloud server.

[0481] The emotion engine analyzes the employee's emotional state in real time, determining whether they are tired or tense.

[0482] 3. Memories generation mode and personality reproduction mode

[0483] The server utilizes historical data collected and real-time emotional data to generate appropriate topics and tones depending on the user's emotional state.

[0484] For example, if the system detects that a user is depressed, and asks, "Tell me about this photo," the memory-generating mode will respond in a gentle tone with, "This photo was taken in the summer of 2018 of the lavender fields in Biei, Hokkaido."

[0485] 4. Production history analysis mode

[0486] The server analyzes past data and real-time emotional data to evaluate the employee's work performance. For example, if the server determines that the employee is tired, the robot will advise them to "pause your work and take a short break."

[0487] 5. Work Instruction Mode

[0488] The server will guide employees through the appropriate work process based on their emotional state, explaining to a nervous employee in a relaxing voice, "Let's simplify the next step."

[0489] Specific examples

[0490] Prompt Sentence Examples

[0491] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[0492] This will realize a system that provides efficient work guidance and support to employees while taking into account their emotional state. Specifically, if an employee is tired, the system will pause work to encourage them to rest, and it will also simplify work processes to reduce stress.

[0493] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0494] Step 1:

[0495] The server collects users' past data from cloud storage. The input requires the cloud storage API key and user authentication information, and the output includes photos, audio, notes, emails, etc. This data is stored digitally and sent to the next analysis step.

[0496] Step 2:

[0497] The server collects facial expression and voice data of employees in real time using a camera and microphone. Real-time data from the camera and microphone is required as input, and facial expression video data and voice data are obtained as output. This data is sent to the emotion engine to analyze the employee's current emotional state.

[0498] Step 3:

[0499] The server analyzes the collected data. It uses as input the previously collected data (output of step 1) and real-time emotion data (output of step 2). This allows for metadata analysis of photos, keyword detection in speech, and text analysis of notes and emails. The output provides key information and insights that are integrated with the emotion engine's analysis.

[0500] Step 4:

[0501] The server uses an emotion engine to analyze the employee's emotional state. It requires real-time collected facial expression and voice data as input, and outputs an emotional status, such as "tired," "tense," or "relaxed." This status is used to generate appropriate instructions for the user in the next step.

[0502] Step 5:

[0503] The server executes the production history analysis mode based on the analysis results. As input, it requires the past data analysis results (output of step 3) and the real-time emotional status (output of step 4). The server evaluates the employee's work performance and generates appropriate instructions and support according to the employee's emotional state. As output, it obtains a specific action plan.

[0504] One specific action is "suggesting rest." For example, if the server determines that an employee is tired, it will generate instructions such as "We recommend that you pause your work and take a short break," and communicate this to the employee via the robot's voice output system.

[0505] Step 6:

[0506] The server executes the work instruction mode. It uses the analysis results (output of Step 5) and emotional status data as input. The server instructs the work procedure according to the employee's emotional state. For example, if the employee is nervous, the server generates instructions encouraging relaxation, such as "Let's perform the next step using a simplified procedure," and the robot conveys this aloud.

[0507] Step 7:

[0508] The server re-stores the collected and analyzed data in cloud storage. The latest analysis results and generated instruction data are required as input, and the updated employee data history is stored in cloud storage as output. This will further increase the accuracy of future analysis and instruction generation.

[0509] Example prompt

[0510] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[0511] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0512] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0513] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0514] [Second embodiment]

[0515] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0516] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0517] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0518] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0519] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0520] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0521] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0522] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0523] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0524] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0525] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0526] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0527] This invention is a system that collects data such as past photos, audio, memos, and emails stored in a user's cloud storage, analyzes the data, and provides a service that recreates memories and the user's personality in various modes.

[0528] System configuration

[0529] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0530] 1. Data Collection Methods

[0531] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[0532] 2. Data analysis methods

[0533] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[0534] 3. Memories Generation Mode

[0535] The server generates a story based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server uses a natural language generation model to generate an answer about the photo's location, date, and time, as well as related episodes. The device then displays the generated answer to the user.

[0536] 4. Personality Reproduction Mode

[0537] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on these. For example, if a user requests "Create a retirement speech," the server learns the user's style from past emails and notes and generates a speech based on that. The generated speech is sent to the user's device and displayed.

[0538] Specific examples

[0539] 1. Example of the reminiscence generation mode

[0540] The user starts the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the metadata of the corresponding photo from cloud storage, generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were very beautiful," sends the answer to the device, and displays it to the user.

[0541] 2. Specific examples of personality reproduction mode

[0542] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[0543] User Interface and Operation

[0544] User Interface

[0545] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question entry screen, and a results display screen.

[0546] Operating Procedure

[0547] 1. The user launches the app and logs in.

[0548] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0549] 3. The user enters and submits a question or request.

[0550] 4. The server parses the data and generates the results.

[0551] 5. The results are displayed on the device and feedback is given to the user.

[0552] In this way, users can enjoy reminiscing about the past and generate sentences that reflect their own personalities. This system allows users to effectively utilize past data and enjoy richer life experiences.

[0553] The processing flow will be explained below.

[0554] Processing steps of data collection instruments

[0555] Step 1:

[0556] The user taps on the app icon to launch it, and the app is launched.

[0557] Step 2:

[0558] The user enters their ID and password on the login screen and presses the "Login" button.

[0559] Step 3:

[0560] The server checks the entered authentication information against a database and performs authentication.

[0561] Step 4:

[0562] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[0563] Step 5:

[0564] The device displays a permission dialog to the user requesting permission to collect data.

[0565] Step 6:

[0566] The user selects "Allow" in the permission dialog.

[0567] Step 7:

[0568] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[0569] Step 8:

[0570] The server stores the acquired data in storage and prepares it for analysis.

[0571] Processing steps of data analysis method

[0572] Step 1:

[0573] The server retrieves the collected data for analysis.

[0574] Step 2:

[0575] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[0576] Step 3:

[0577] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[0578] Step 4:

[0579] The server classifies and organizes the extracted information and stores it in a database so that it can be used to generate memories and recreate personalities.

[0580] Processing steps for reminiscence generation mode

[0581] Step 1:

[0582] The user selects "reminiscence mode" on the mode selection screen.

[0583] Step 2:

[0584] The user enters a question or request into the question input screen and presses the "Submit" button.

[0585] Step 3:

[0586] The terminal sends the user's question to the server.

[0587] Step 4:

[0588] The server analyzes the user's question using a natural language processing model.

[0589] Step 5:

[0590] Based on the analysis results, the server searches the database for relevant memories.

[0591] Step 6:

[0592] The server uses the reminiscence generation model to generate reminiscences in response to questions from the user.

[0593] Step 7:

[0594] The terminal displays the reminiscences received from the server to the user.

[0595] Personality Reproduction Mode Processing Steps

[0596] Step 1:

[0597] The user selects "personality reproduction mode" on the mode selection screen.

[0598] Step 2:

[0599] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[0600] Step 3:

[0601] The terminal transmits the user's request to the server.

[0602] Step 4:

[0603] The server retrieves relevant data to analyze the user's thinking patterns, writing style, and speaking style.

[0604] Step 5:

[0605] The server uses machine learning models to generate sentences based on the user's style.

[0606] Step 6:

[0607] The server sends the generated text to the terminal.

[0608] Step 7:

[0609] The terminal displays the generated text to the user.

[0610] Example 1

[0611] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0612] In today's digitalized society, users store vast amounts of data (photos, audio, notes, emails, etc.) in cloud storage. However, this data simply functions as a storage location and is not used for specific services. It is also difficult to effectively utilize past data to reminisce or recreate a user's writing or speaking style. This prevents users from fully utilizing the potential value of their data.

[0613] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0614] In this invention, the server includes means for collecting data from cloud storage based on the user's access permission, means for analyzing the collected data and extracting metadata and content, means for generating and providing information from the analyzed data in response to the user's request, means for using a natural language generation model to generate reminiscences, and means for learning the user's writing and speaking style and generating text, thereby enabling the user to effectively utilize past data and enjoy a richer digital experience.

[0615] "User" means any person or entity that uses the System.

[0616] "Access permission" refers to the act of authentication by a user to allow the server to access their cloud storage.

[0617] "Cloud storage" refers to an online storage service for storing user data via the Internet.

[0618] "Data" refers to information such as photos, audio, notes, and emails stored in the user's cloud storage.

[0619] "Server" refers to a computer and related hardware and software that executes various processes such as collection, analysis, and generation.

[0620] "Means" refers to the processes, functions, or devices necessary to achieve a certain purpose.

[0621] "Collection" refers to the act of obtaining specific data from cloud storage.

[0622] "Analysis" refers to the process of understanding the content of collected data and extracting metadata and important information.

[0623] "Metadata" refers to additional information about the data (e.g., the date and location of a photo).

[0624] "Information generation" refers to the act of creating new information based on analyzed data.

[0625] "Memories generation mode" refers to a mode that provides the function of generating memories based on past data.

[0626] A "natural language generation model" refers to a machine learning model that generates natural language expressions based on input text information.

[0627] "Learning how to write and speak" refers to the process of analyzing the user's past writing and voice data and learning their characteristics.

[0628] "Text generation" refers to the act of creating new text based on learned features.

[0629] This invention is a system that effectively utilizes past data stored in a user's cloud storage to provide a service that recreates reminiscences and the user's writing and speaking styles. The system is composed of a data collection means, a data analysis means, a server that includes a reminiscence generation mode and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0630] Data collection methods

[0631] The server obtains the user's permission to collect data from cloud storage. To collect data, it uses common cloud storage APIs such as Google Drive API and Dropbox API. Specifically, after the user launches the app and logs in, the server securely downloads the data by granting permission to access the cloud storage.

[0632] Data Analysis Methods

[0633] The server uses machine learning models (e.g. TensorFlow or PyTorch) to analyze the collected data. The following steps are performed:

[0634] Photo data analysis: The server analyzes the photo data and extracts metadata and content using image recognition algorithms (e.g., OpenCV or TensorFlow models), specifically face detection and scene recognition to determine whether a particular face was captured in a particular location.

[0635] Analyzing the audio data: When the server analyzes the audio data, it uses speech recognition technologies such as Google Cloud Speech-to-Text API or Amazon Transcribe to transcribe the content and extract key keywords and phrases.

[0636] Memo / Email Analysis: The server analyzes text data (memos, emails) and extracts important information using natural language processing techniques (e.g., SpaCy or NLTK).

[0637] Memories generation mode

[0638] When a user selects the reminiscence mode, the server generates a reminiscence based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server generates an answer using a natural language generation model (e.g., GPT-3) based on the metadata of the corresponding photo and related anecdotes. The generated answer is sent to the device and displayed to the user.

[0639] Specific examples

[0640] Example of memory generation mode

[0641] The user launches the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the photo's metadata and generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken as part of a family trip," which is sent to the device and displayed to the user.

[0642] Example prompt for a generative AI model:

[0643] "Generate an episode about a photo taken by a user on a family trip in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Tell them that the lavender fields were very beautiful."

[0644] Personality Reproduction Mode

[0645] When a user selects the personality mode, the server learns the user's writing and speaking style based on the analyzed memo and email data. For example, if a user requests to "write a birthday message for a friend," the server analyzes past emails and memos and generates a message based on the user's style.

[0646] Specific examples

[0647] Specific examples of personality reproduction mode

[0648] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[0649] Example prompt for a generative AI model:

[0650] "We've learned about the user's writing and speaking style from their past emails and notes. Generate the following birthday message for a friend. This user prefers to write in a friendly, warm style."

[0651] User Interface and Operation

[0652] The user interface includes a login screen, a mode selection screen, a question input screen, and a result display screen. The user interacts with the system through these screens. The operation procedure is as follows:

[0653] 1. The user launches the app and logs in.

[0654] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0655] 3. The user enters and submits a question or request.

[0656] 4. The server parses the data and generates the results.

[0657] 5. The results are displayed on the device and feedback is given to the user.

[0658] In this way, users can utilize past data, reminisce, and generate texts that suit their own style. This system allows users to effectively utilize data stored in cloud storage and enjoy a richer digital experience.

[0659] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0660] Step 1: User launches the app and logs in.

[0661] Input: User login information (username, password)

[0662] Data processing / calculation: The server verifies and authenticates the login information, allowing the user to access cloud storage.

[0663] Output: Authentication success message and app home screen

[0664] Step 2: The user grants permission to access the cloud storage.

[0665] Input: A dialog where the user asks for permission to access a cloud storage service

[0666] Data processing / calculation: The server authenticates using OAuth2.0 and establishes a secure connection to the user's cloud storage.

[0667] Output: Obtain cloud storage access permission and display the app mode selection screen.

[0668] Step 3: The user selects "reminiscence mode" or "personality reproduction mode."

[0669] Input: User mode selection (reminiscence mode or personality mode)

[0670] Data processing / calculation: The server sends instructions to the device to display the UI according to the selected mode.

[0671] Output: Screen display according to selected mode

[0672] Step 4: The user enters and submits a question or request.

[0673] Input: A user-typed question or request (e.g., "Where was this photo taken?")

[0674] Data processing / calculation: The server analyzes the user's input and performs appropriate data collection and analysis processing.

[0675] Output: Analysis results of the question or request

[0676] Step 5: The server collects the data from the cloud storage.

[0677] Input: User cloud storage access permissions, data information to be collected

[0678] Data processing / calculation: The server uses the Google Drive API or Dropbox API to download data such as photos, audio, notes, and emails.

[0679] Output: Collected data (photos, audio, notes, emails)

[0680] Step 6: The server analyzes the collected data.

[0681] Input: Collected data

[0682] Data processing / calculation: The server uses machine learning models (e.g., TensorFlow, PyTorch) to perform the following analysis:

[0683] Photo data analysis: Extract metadata and content using image recognition algorithms

[0684] Analysis of audio data: Transcription and content extraction using speech recognition technology

[0685] Memo and email analysis: Extracting important information using natural language processing techniques

[0686] Output: Analyzed data (metadata, transcription results, extracted key information)

[0687] Step 7: The server generates the reminiscence and text.

[0688] Input: Analyzed data, user questions and requests

[0689] Data processing / computation: Generate reminiscences and text using a natural language generation model (e.g., GPT-3). This involves feeding prompts into the generative AI model.

[0690] Specific Action: Example: "Generate an episode about a photo of a family trip taken by a user in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Please tell us that the lavender fields were very beautiful."

[0691] Output: Generated reminiscences and text

[0692] Step 8: Provide the generated results to the user.

[0693] Input: Generated reminiscences and text

[0694] Data processing / calculation: The server sends the generated data to the terminal and displays it on the screen.

[0695] Output: Reminiscences and text displayed on the user's terminal

[0696] In this way, by clarifying the specific operations and the associated data input and output at each processing step, users can make full use of past data through the system, enjoy reminiscing, and generate individual sentences.

[0697] (Application example 1)

[0698] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] In recent years, the amount of data stored in cloud storage has increased, but methods for effectively utilizing this data are limited. There is also a need for methods to provide personalized services using users' past data and systems that generate accurate responses to user requests. In particular, for food delivery services, functions such as recommendations based on past order history and message generation for special occasions are effective. The objective of this invention is to realize a system that analyzes and utilizes data in cloud storage to provide users with reminiscences, personalized recommendations, and message generation based on personality reproduction.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0701] In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, and means for making individualized recommendations based on the data analysis, thereby enabling personalized recommendations based on the user's past data, the generation of stories of memories, and the generation of messages for special occasions.

[0702] "Cloud storage" is an online storage service for storing and managing data via the Internet.

[0703] "Data collection means" refers to a means for obtaining a user's past data from cloud storage.

[0704] "Data analysis means" refers to means for analyzing collected data and extracting necessary information.

[0705] The "means for providing a service" is a means for providing a service in a plurality of modes in response to a user's selection.

[0706] "Personalized recommendations" means making personalized recommendations to users based on data analysis.

[0707] The "mode for generating reminiscences" is a mode for generating reminiscences based on the user's past photos, voice, memos and emails.

[0708] "Personality reproduction mode" is a mode that reproduces the user's thought patterns, writing style, and speaking style based on their past data.

[0709] This invention is a system that collects and analyzes data stored in a user's cloud storage, and provides reminiscences, personalized recommendations, and message generation that recreates the user's personality. Specific embodiments are described below.

[0710] System configuration

[0711] This system is composed of a server including a data collection means, a data analysis means, a service provision means, and a personalized recommendation means, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0712] Data collection methods

[0713] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. In this process, protecting the user's privacy and consent to data collection are important.

[0714] Data Analysis Methods

[0715] The server analyzes the collected data and uses machine learning models to extract the necessary information. This allows it to accurately analyze photo metadata, audio content, and note content. It uses the ImageAI library (ResNet model) for image classification and textgenrnn for text analysis.

[0716] Service delivery and personalized recommendation methods

[0717] The server provides services in three modes depending on the user's selection: memory generation mode, personality reproduction mode, and personalized recommendation mode, allowing the user to analyze past order data and messages to obtain a personalized customer experience.

[0718] Memories generation mode

[0719] When a user asks about a specific photo, "Where was this photo taken?", the server analyzes the photo's metadata from cloud storage and generates a story about the photo, such as "This photo was taken in Biei, Hokkaido in the summer of 2018. It was taken during a family trip, and the lavender fields were beautiful," which is then displayed on the device.

[0720] Personality Reproduction Mode

[0721] When a user requests a retirement speech, the server analyzes past emails and memos to learn the user's writing and speaking style. Based on this, it generates a speech such as, "Thank you all for joining us today. I am truly grateful to have had the opportunity to work with you all," and displays it on the device.

[0722] Personalized recommendation mode

[0723] Based on data on the food a user has ordered in the past, the system makes personalized recommendations such as, "Did you like the pizza you ordered for your birthday last year?" or "How about this new menu item?" The server analyzes the user's past order history and preferences, generates appropriate recommendations, and displays them on the device.

[0724] User Interface and Operation

[0725] Users interact with the system through terminal applications, which include a login screen, a mode selection screen, a question entry screen, and a result display screen.

[0726] Operating Procedure

[0727] 1. The user launches the app and logs in.

[0728] 2. The user selects "Memories mode," "Personality reproduction mode," or "Individualized recommendation mode" on the mode selection screen.

[0729] 3. The user enters and submits a question or request.

[0730] 4. The server analyzes the data and generates results.

[0731] 5. The generated results are displayed on the device and feedback is provided to the user.

[0732] Prompt Sentence Examples

[0733] When a user uploads a photo and asks, "Tell me what memories you have of this photo," the server analyzes the photo and tells them, "This photo is of the pizza I ordered for my birthday last year, and it was delicious."

[0734] In this way, users can enjoy past memories and receive personalized recommendations and thoughtful messages. This system allows users to effectively utilize past data and enjoy a richer experience.

[0735] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0736] Step 1:

[0737] The user starts the application on the terminal and logs in.

[0738] Input: User login information (ID, password).

[0739] Output: Notifies the success or failure of user authentication.

[0740] Specific operation: The user enters their ID and password into the login screen, and the server receives this and performs authentication.

[0741] Step 2:

[0742] The user selects "reminiscence mode," "personality reproduction mode," or "individualized recommendation mode" on the mode selection screen.

[0743] Input: User mode selection.

[0744] Output: The screen of the selected mode.

[0745] Specific operation: The user selects a mode according to the purpose, and the server prepares for data collection and analysis accordingly.

[0746] Step 3:

[0747] The user enters and submits a question or request.

[0748] Input: The user's question or request (e.g., "Tell me what you remember about this photo").

[0749] Output: Success or failure of sending data to the server.

[0750] Specific operation: The application sends the question or request entered by the user to the server.

[0751] Step 4:

[0752] The server collects the user's data from the cloud storage.

[0753] Input: User permissions and cloud storage access information.

[0754] Output: Collected data such as photos, audio, notes, emails, etc.

[0755] Specific operation: The server retrieves the required data using the cloud storage API.

[0756] Step 5:

[0757] The server analyzes the collected data.

[0758] Input: Data collected in step 4.

[0759] Output: Information based on the parsed metadata and content.

[0760] Specific operation: The ResNet model from the ImageAI library is used for image classification, and textgenrnn is used for text analysis, analyzing photo metadata, audio content, note contents, etc.

[0761] Step 6:

[0762] The server generates a service based on the analysis results.

[0763] Input: Analyzed data and user questions or requests.

[0764] Output: Generated memories, personality reenactments, and personalized recommendations.

[0765] Specific behavior: For example, in the reminiscence mode, it generates a narrative such as, "This photo was taken in Biei, Hokkaido in the summer of 2018."

[0766] Step 7:

[0767] The server generates a response that is sent to the terminal and displayed to the user.

[0768] Input: Generated reminiscences, personality reenactments, and personalized recommendations.

[0769] Output: The results that are displayed to the user on the application screen.

[0770] Specific operation: The server sends the generated content to the terminal, and the application displays it.

[0771] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0772] This invention combines an emotion engine with a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. In particular, the aim is to improve the quality and user experience of the memory generation mode and personality reproduction mode by recognizing the user's emotions.

[0773] System configuration

[0774] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, a personality reproduction mode, and an emotion engine, a terminal that acts as a user interface, and a series of means for interacting with the user.

[0775] 1. Data Collection Methods

[0776] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[0777] 2. Data analysis methods

[0778] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[0779] 3. Emotion Engine

[0780] The server is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their current emotional state and generate an appropriate response.

[0781] 4. Memories Generation Mode

[0782] The server generates a reminiscence based on the analyzed data. It also takes into account the data from the emotion engine and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[0783] 5. Personality Reproduction Mode

[0784] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[0785] Specific examples

[0786] 1. Example of the reminiscence generation mode

[0787] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[0788] 2. Specific examples of personality reproduction mode

[0789] The user launches the app, logs in, and selects "Personality Reproduction Mode." The emotion engine recognizes that the user's tone of voice indicates nervousness. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, the server generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for everything you've done. I'd especially like to thank Tanaka-san, who helped me with Project X. I will continue to walk a new path, keeping the time we spent together in my heart. I'm a little nervous, but I truly appreciate it." The speech is then sent to the device and displayed to the user.

[0790] User Interface and Operation

[0791] User Interface

[0792] Users interact with the system through a device app, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. It may also display real-time emotion analysis results from an emotion engine.

[0793] Operating Procedure

[0794] 1. The user launches the app and logs in.

[0795] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0796] 3. The user enters and submits a question or request.

[0797] 4. The server analyzes the data and generates results using the emotion engine.

[0798] 5. The results are displayed on the device and feedback is given to the user.

[0799] In this way, users can share their past memories while feeling emotionally attached to them, and generate sentences that correspond to their emotional state.This system allows users to effectively utilize past data and enjoy richer life experiences.

[0800] The processing flow will be explained below.

[0801] Processing steps of data collection instruments

[0802] Step 1:

[0803] The user taps on the app icon to launch it, and the app is launched.

[0804] Step 2:

[0805] The user enters their ID and password on the login screen and presses the "Login" button.

[0806] Step 3:

[0807] The server checks the entered authentication information against a database and performs authentication.

[0808] Step 4:

[0809] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[0810] Step 5:

[0811] The device displays a permission dialog to the user requesting permission to collect data.

[0812] Step 6:

[0813] The user selects "Allow" in the permission dialog.

[0814] Step 7:

[0815] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[0816] Step 8:

[0817] The server stores the acquired data in storage in preparation for analysis.

[0818] Processing steps of data analysis method

[0819] Step 1:

[0820] The server retrieves the collected data and begins analyzing it.

[0821] Step 2:

[0822] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[0823] Step 3:

[0824] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[0825] Step 4:

[0826] The server classifies the extracted information and stores it in a database.

[0827] Emotion Engine Processing Steps

[0828] Step 1:

[0829] The device captures the user's voice and facial expressions in real time.

[0830] Step 2:

[0831] The device sends the captured data to the server.

[0832] Step 3:

[0833] The server analyzes the received voice and facial expression data.

[0834] Step 4:

[0835] The server uses an emotion engine to recognize the user's emotional state (e.g., joy, sadness, tension, etc.).

[0836] Step 5:

[0837] The emotional data analyzed by the emotion engine is reflected in related processing (memory story generation and personality reproduction).

[0838] Processing steps for reminiscence generation mode

[0839] Step 1:

[0840] The user selects "Memories mode" on the mode selection screen within the app.

[0841] Step 2:

[0842] The user enters a question into the question input screen and presses the "Submit" button.

[0843] Step 3:

[0844] The terminal sends the user's question to the server.

[0845] Step 4:

[0846] The server analyzes the user's question using a natural language processing model.

[0847] Step 5:

[0848] The server searches the database for relevant memories based on the analysis results and emotion engine data.

[0849] Step 6:

[0850] The server uses a reminiscence generation model to generate a reminiscence that is in line with the user's emotions.

[0851] Step 7:

[0852] The terminal displays the reminiscences received from the server to the user.

[0853] Personality Reproduction Mode Processing Steps

[0854] Step 1:

[0855] The user selects "Personality Reproduction Mode" on the mode selection screen within the app.

[0856] Step 2:

[0857] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[0858] Step 3:

[0859] The terminal transmits the user's request to the server.

[0860] Step 4:

[0861] The server retrieves relevant past data and analyzes the user's thought patterns, writing style, and speaking style.

[0862] Step 5:

[0863] The server takes into account the data from the emotion engine and generates appropriate sentences according to the user's emotional state.

[0864] Step 6:

[0865] The server sends the generated text to the terminal.

[0866] Step 7:

[0867] The terminal displays the generated text to the user.

[0868] Example 2

[0869] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0870] The purpose of this invention is to provide a richer user experience by utilizing the user's past data stored in cloud storage. However, conventional technologies simply display data, making it difficult to provide services that take the user's emotional state into account. Furthermore, there is a problem in that the quality of reminiscences and personality reproduction is low because emotions are not recognized.

[0871] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means for providing an emotion engine for analyzing emotions from the user's voice and facial expressions, means for generating reminiscences based on the user's emotional state, and means for generating sentences based on the user's writing and speaking style. This makes it possible to provide high-quality reminiscences according to the user's emotional state and improve the accuracy of personality reproduction.

[0872] "Cloud storage" is a storage service on a remote server that stores and manages data over the Internet.

[0873] "User's past data" refers to various data that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[0874] "Means of collection" refers to the functionality for obtaining the required data from authorized cloud storage.

[0875] "Means of analysis" refers to the ability to extract useful information from collected data using machine learning models and algorithms.

[0876] "Means for providing services" refers to the function of generating content such as reminiscences and personality reenactments based on analyzed data and providing it to users.

[0877] "Emotion engine" refers to technology that analyzes a user's voice and facial expressions to identify their current emotional state.

[0878] "Means for generating reminiscences" refers to a function for automatically generating reminiscences for a user based on analyzed past data and the user's current emotional state.

[0879] "Means of personality reproduction" refers to a function that analyzes the user's past data and generates sentences based on the user's writing and speaking style.

[0880] This invention is a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. By incorporating an emotion engine into this system, the quality and user experience of the reminiscence generation mode and personality reproduction mode can be improved. Specific embodiments are described below.

[0881] System configuration

[0882] This system consists of a server that performs a series of processes, such as data collection means, data analysis means, memory generation mode, personality reproduction mode, and emotion engine, a terminal that acts as a user interface, and a series of means for interaction with the user.

[0883] Data collection methods

[0884] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from cloud storage (e.g., a general cloud storage service). In this process, it is important to protect the user's privacy and obtain consent for data collection.

[0885] Data Analysis Methods

[0886] The server uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the collected data. This allows it to accurately analyze photo metadata, audio content, and note entries. The analysis results are used to generate reminiscences and recreate personalities.

[0887] Emotion Engine

[0888] The server is equipped with an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) that analyzes emotions from the user's voice and facial expressions, allowing it to recognize the user's current emotional state and generate an appropriate response.

[0889] Memories generation mode

[0890] The server generates a reminiscence based on the analyzed data. It also takes into account the emotion engine data and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[0891] Examples:

[0892] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[0893] Personality Reproduction Mode

[0894] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[0895] Examples:

[0896] The user starts the app, logs in, and selects "Personality Mode." The emotion engine recognizes that the user's tone of voice sounds tense. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, it generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for your support over the years. I'm especially grateful to everyone who helped me on this project. I'm truly grateful, but I'm nervous about taking on this new challenge." The speech is then sent to the device and displayed to the user.

[0897] User Interface and Operation

[0898] User Interface:

[0899] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. In addition, the app may display real-time emotion analysis results from an emotion engine.

[0900] Operation steps:

[0901] 1. The user launches the app and logs in.

[0902] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[0903] 3. The user enters and submits a question or request.

[0904] 4. The server analyzes the data and generates results using the emotion engine.

[0905] 5. The results are displayed on the device and feedback is given to the user.

[0906] This process allows users to share their past memories while feeling the emotions involved, and generate text that reflects their emotional state. This system allows users to effectively utilize past data and enjoy richer life experiences.

[0907] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0908] Step 1:

[0909] The user launches the app and logs in. The user enters their username and password on the device's login screen and clicks the login button. The server checks the user's authentication information against the database, and if authentication is successful, it generates a JWT (JSON Web Token) and sends it to the device. This starts the user's session. The output is the home screen with the user logged in.

[0910] Step 2:

[0911] The user selects a mode. The user selects "Memories Mode" or "Personality Reproduction Mode" on the home screen. The device sends the user's selection to the server. The server records the mode selected by the user and prepares the system to proceed to the next step. The input is an instruction to select a mode, and the output is confirmation of the selected mode.

[0912] Step 3:

[0913] The server obtains consent to collect data. The server displays a dialog to the user requesting permission to access cloud storage. If the user agrees, the server obtains the necessary data access rights from the cloud storage using the OAuth 2.0 protocol. The input is consent information, and the output is the acquisition of access rights. The server collects photos, audio, notes, emails, etc. from the cloud storage service and stores them in a temporary storage area.

[0914] Step 4:

[0915] The server analyzes the collected data. The server analyzes the data using machine learning models (e.g., TensorFlow, PyTorch). Specific operations include extracting metadata from photos, transcribing audio data, and analyzing the content of notes. The input is the collected data, and the output is the analysis results. The analysis results are used in the next step.

[0916] Step 5:

[0917] The server performs emotion analysis using an emotion engine. The server inputs the user's voice and facial expression data into an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) to analyze the user's emotional state. The input is the user's voice and facial expression data, and the output is data on the user's emotional state (e.g., joy, sadness, tension). This allows the user's current emotional state to be quantified.

[0918] Step 6:

[0919] The server generates content based on emotions. Based on the analysis results and the emotion engine results, the server generates sentences for reminiscences or personality reenactment. To do this, a prompt sentence is input to a generative AI model (e.g., GPT-3, BERT), which then generates an appropriate sentence. The input is the prompt sentence and the analysis results, and the output is the generated sentence.

[0920] Specifically, in the reminiscence mode, the system uses the prompt "Tell me your memories of this photo" and adjusts the emotional tone based on the generated text. In the personality reproduction mode, the system uses the prompt "Write a retirement speech" and adjusts the tone of the generated speech, such as making it more relaxed.

[0921] Step 7:

[0922] The server sends the generated content to the terminal. The server sends the generated text to the terminal in JSON format. The terminal parses the received data and displays it on the user interface. The user can check the displayed results and send feedback accordingly. The input is the data of the generated content, and the output is the user's display screen.

[0923] Through the above steps, users can enjoy high-quality emotional reminiscences and personality reproductions using past data.

[0924] (Application example 2)

[0925] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0926] Conventional robot assistant systems in factories have limited capabilities for providing work guidance and employee support, making it difficult to provide appropriate support that takes into account the emotional state of employees. Therefore, there has been a demand for a method that can simultaneously promote efficient work and reduce employee stress.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of users from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means having an emotion engine that analyzes facial expressions and voice in real time, and means for providing appropriate instructions and support according to the user's emotions. This enables efficient work guidance and support for employees while taking into account their emotional state.

[0928] "Cloud storage" is a technology that provides remote servers for storing data over the Internet.

[0929] "User's past data" refers collectively to information that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[0930] "Means of collection" refers to the processes and tools used to obtain the required data from cloud storage.

[0931] "Means of analysis" are methods and techniques for analyzing collected data and extracting useful information.

[0932] "Multiple modes" refer to different types of services that the system can provide, including, for example, a reminiscence generation mode and a personality reproduction mode.

[0933] The "emotion engine" is a technology that analyzes the user's emotional state from voice and facial expression data.

[0934] "Real-time facial expression and voice analysis" refers to the ability to instantly assess a user's current emotional state.

[0935] "Means for providing instructions and support" refers to mechanisms for providing specific actions or advice to users based on the analysis results.

[0936] The "mode for generating memories" is a function that generates stories that evoke memories based on the user's past data.

[0937] "Personality Reproduction Mode" is a function that imitates the user's personality based on their past actions and documents, and generates responses in real time.

[0938] The "work instruction mode" is a function in which the robot instructs employees on how to do their work in a factory or other environment.

[0939] "Production history analysis mode" is a function that analyzes past data to evaluate employee work performance.

[0940] "Voicing" refers to the act of the robot providing verbal advice or instructions depending on the employee's emotional state.

[0941] This paper describes an embodiment of a system equipped with an emotion engine for factory robots that incorporates a "production history analysis mode" and a "work guidance mode." This system uses a robot assistant in a factory to analyze the emotional state of employees in real time and provide appropriate guidance and support. The main hardware and software configurations and their processing flow are described.

[0942] Hardware Configuration

[0943] 1. Factory robots

[0944] Supports all operations required for operation.

[0945] 2. Camera and microphone

[0946] Capture employees' facial expressions and voices.

[0947] 3. Cloud Storage

[0948] A remote server for storing employee historical data.

[0949] Software Configuration

[0950] 1. Machine Learning Model

[0951] TensorFlow and Google Cloud AI are used for data analysis.

[0952] 2. Emotion Engine

[0953] Affectiva and Microsoft Azure Emotion API are used as technologies to analyze emotions from voice and facial expressions.

[0954] 3. Robot Control Software

[0955] Control the robot's movements using ROS (Robot Operating System).

[0956] 4. User Interface

[0957] Users interact with the system via factory tablets and displays.

[0958] Program processing flow

[0959] 1. Data Collection

[0960] The server collects users' past data from cloud storage, including photos, audio, notes, and emails.

[0961] Cameras and microphones are used to collect employees' facial expressions and voices in real time.

[0962] 2. Data Analysis

[0963] The collected data is analyzed using machine learning models on a cloud server.

[0964] The emotion engine analyzes the employee's emotional state in real time, determining whether they are tired or tense.

[0965] 3. Memories generation mode and personality reproduction mode

[0966] The server utilizes historical data collected and real-time emotional data to generate appropriate topics and tones depending on the user's emotional state.

[0967] For example, if the user is recognized as depressed, when asked "Tell me about this photo," the memory-generating mode will respond in a gentle tone with, "This photo was taken in the summer of 2018 of the lavender fields in Biei, Hokkaido."

[0968] 4. Production history analysis mode

[0969] The server analyzes past data and real-time emotional data to evaluate the employee's work performance. For example, if the server determines that the employee is tired, the robot will advise them to "pause your work and take a short break."

[0970] 5. Work Instruction Mode

[0971] The server will guide employees through the appropriate work process based on their emotional state, explaining to a nervous employee in a relaxing voice, "Let's simplify the next step."

[0972] Specific examples

[0973] Prompt Sentence Examples

[0974] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[0975] This will realize a system that provides efficient work guidance and support to employees while taking into account their emotional state. Specifically, if an employee is tired, the system will pause work to encourage them to rest, and it will also simplify work processes to reduce stress.

[0976] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0977] Step 1:

[0978] The server collects users' past data from cloud storage. The input requires the cloud storage API key and user authentication information, and the output includes photos, audio, notes, emails, etc. This data is stored digitally and sent to the next analysis step.

[0979] Step 2:

[0980] The server collects facial expression and voice data of employees in real time using a camera and microphone. Real-time data from the camera and microphone is required as input, and facial expression video data and voice data are obtained as output. This data is sent to the emotion engine to analyze the employee's current emotional state.

[0981] Step 3:

[0982] The server analyzes the collected data. It uses as input the previously collected data (output of step 1) and real-time emotion data (output of step 2). This allows for metadata analysis of photos, keyword detection in speech, and text analysis of notes and emails. The output provides key information and insights that are integrated with the emotion engine's analysis.

[0983] Step 4:

[0984] The server uses an emotion engine to analyze the employee's emotional state. It requires real-time collected facial expression and voice data as input, and outputs an emotional status, such as "tired," "tense," or "relaxed." This status is used to generate appropriate instructions for the user in the next step.

[0985] Step 5:

[0986] The server executes the production history analysis mode based on the analysis results. As input, it requires the past data analysis results (output of step 3) and the real-time emotional status (output of step 4). The server evaluates the employee's work performance and generates appropriate instructions and support according to the employee's emotional state. As output, it obtains a specific action plan.

[0987] One specific action is "suggesting rest." For example, if the server determines that an employee is tired, it will generate instructions such as "We recommend that you temporarily stop working and take a short break," and communicate this to the employee via the robot's voice output system.

[0988] Step 6:

[0989] The server executes the work instruction mode. It uses the analysis results (output of Step 5) and emotional status data as input. The server instructs the work procedure according to the employee's emotional state. For example, if the employee is nervous, the server generates instructions encouraging relaxation, such as "Let's perform the next step using a simplified procedure," and the robot conveys this aloud.

[0990] Step 7:

[0991] The server re-stores the collected and analyzed data in cloud storage. The latest analysis results and generated instruction data are required as input, and the updated employee data history is stored in cloud storage as output. This will further increase the accuracy of future analysis and instruction generation.

[0992] Example prompt

[0993] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[0994] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0995] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0996] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0997] [Third embodiment]

[0998] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0999] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1000] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1001] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1002] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1003] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1004] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1005] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1006] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1007] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1008] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1009] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1010] This invention is a system that collects data such as past photos, audio, memos, and emails stored in a user's cloud storage, analyzes the data, and provides a service that recreates memories and the user's personality in various modes.

[1011] System configuration

[1012] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1013] 1. Data Collection Methods

[1014] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[1015] 2. Data analysis methods

[1016] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[1017] 3. Memories Generation Mode

[1018] The server generates a story based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server uses a natural language generation model to generate an answer about the photo's location, date, and time, as well as related episodes. The device then displays the generated answer to the user.

[1019] 4. Personality Reproduction Mode

[1020] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on these. For example, if a user requests "Create a retirement speech," the server learns the user's style from past emails and notes and generates a speech based on that. The generated speech is sent to the user's device and displayed.

[1021] Specific examples

[1022] 1. Example of the reminiscence generation mode

[1023] The user starts the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the metadata of the corresponding photo from cloud storage, generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were very beautiful," sends the answer to the device, and displays it to the user.

[1024] 2. Specific examples of personality reproduction mode

[1025] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[1026] User Interface and Operation

[1027] User Interface

[1028] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question entry screen, and a results display screen.

[1029] Operating Procedure

[1030] 1. The user launches the app and logs in.

[1031] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1032] 3. The user enters and submits a question or request.

[1033] 4. The server parses the data and generates the results.

[1034] 5. The results are displayed on the device and feedback is given to the user.

[1035] In this way, users can enjoy reminiscing about the past and generate sentences that reflect their own personalities. This system allows users to effectively utilize past data and enjoy richer life experiences.

[1036] The processing flow will be explained below.

[1037] Processing steps of data collection instruments

[1038] Step 1:

[1039] The user taps on the app icon to launch it, and the app is launched.

[1040] Step 2:

[1041] The user enters their ID and password on the login screen and presses the "Login" button.

[1042] Step 3:

[1043] The server checks the entered authentication information against a database and performs authentication.

[1044] Step 4:

[1045] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[1046] Step 5:

[1047] The device displays a permission dialog to the user requesting permission to collect data.

[1048] Step 6:

[1049] The user selects "Allow" in the permission dialog.

[1050] Step 7:

[1051] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[1052] Step 8:

[1053] The server stores the acquired data in storage and prepares it for analysis.

[1054] Processing steps of data analysis method

[1055] Step 1:

[1056] The server retrieves the collected data for analysis.

[1057] Step 2:

[1058] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[1059] Step 3:

[1060] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[1061] Step 4:

[1062] The server classifies and organizes the extracted information and stores it in a database so that it can be used to generate memories and recreate personalities.

[1063] Processing steps for reminiscence generation mode

[1064] Step 1:

[1065] The user selects "reminiscence mode" on the mode selection screen.

[1066] Step 2:

[1067] The user enters a question or request into the question input screen and presses the "Submit" button.

[1068] Step 3:

[1069] The terminal sends the user's question to the server.

[1070] Step 4:

[1071] The server analyzes the user's question using a natural language processing model.

[1072] Step 5:

[1073] Based on the analysis results, the server searches the database for relevant memories.

[1074] Step 6:

[1075] The server uses the reminiscence generation model to generate reminiscences in response to questions from the user.

[1076] Step 7:

[1077] The terminal displays the reminiscences received from the server to the user.

[1078] Personality Reproduction Mode Processing Steps

[1079] Step 1:

[1080] The user selects "personality reproduction mode" on the mode selection screen.

[1081] Step 2:

[1082] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[1083] Step 3:

[1084] The terminal transmits the user's request to the server.

[1085] Step 4:

[1086] The server retrieves relevant data to analyze the user's thinking patterns, writing style, and speaking style.

[1087] Step 5:

[1088] The server uses machine learning models to generate sentences based on the user's style.

[1089] Step 6:

[1090] The server sends the generated text to the terminal.

[1091] Step 7:

[1092] The terminal displays the generated text to the user.

[1093] Example 1

[1094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1095] In today's digitalized society, users store vast amounts of data (photos, audio, notes, emails, etc.) in cloud storage. However, this data simply functions as a storage location and is not used for specific services. It is also difficult to effectively utilize past data to reminisce or recreate a user's writing or speaking style. This prevents users from fully utilizing the potential value of their data.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1097] In this invention, the server includes means for collecting data from cloud storage based on the user's access permission, means for analyzing the collected data and extracting metadata and content, means for generating and providing information from the analyzed data in response to the user's request, means for using a natural language generation model to generate reminiscences, and means for learning the user's writing and speaking style and generating text, thereby enabling the user to effectively utilize past data and enjoy a richer digital experience.

[1098] "User" means any person or entity that uses the System.

[1099] "Access permission" refers to the act of authentication by a user to allow the server to access their cloud storage.

[1100] "Cloud storage" refers to an online storage service for storing user data via the Internet.

[1101] "Data" refers to information such as photos, audio, notes, and emails stored in the user's cloud storage.

[1102] "Server" refers to a computer and related hardware and software that executes various processes such as collection, analysis, and generation.

[1103] "Means" refers to the processes, functions, or devices necessary to achieve a certain purpose.

[1104] "Collection" refers to the act of obtaining specific data from cloud storage.

[1105] "Analysis" refers to the process of understanding the content of collected data and extracting metadata and important information.

[1106] "Metadata" refers to additional information about the data (e.g., the date and location of a photo).

[1107] "Information generation" refers to the act of creating new information based on analyzed data.

[1108] "Memories generation mode" refers to a mode that provides the function of generating memories based on past data.

[1109] A "natural language generation model" refers to a machine learning model that generates natural language expressions based on input text information.

[1110] "Learning how to write and speak" refers to the process of analyzing the user's past writing and voice data and learning their characteristics.

[1111] "Text generation" refers to the act of creating new text based on learned features.

[1112] This invention is a system that effectively utilizes past data stored in a user's cloud storage to provide a service that recreates reminiscences and the user's writing and speaking styles. The system is composed of a data collection means, a data analysis means, a server that includes a reminiscence generation mode and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1113] Data collection methods

[1114] The server obtains the user's permission to collect data from cloud storage. To collect data, it uses common cloud storage APIs such as Google Drive API and Dropbox API. Specifically, after the user launches the app and logs in, the server securely downloads the data by granting permission to access the cloud storage.

[1115] Data Analysis Methods

[1116] The server uses machine learning models (e.g. TensorFlow or PyTorch) to analyze the collected data. The following steps are performed:

[1117] Photo data analysis: The server analyzes the photo data and extracts metadata and content using image recognition algorithms (e.g., OpenCV or TensorFlow models), specifically face detection and scene recognition to determine whether a particular face was captured in a particular location.

[1118] Analyzing the audio data: When the server analyzes the audio data, it uses speech recognition technologies such as Google Cloud Speech-to-Text API or Amazon Transcribe to transcribe the content and extract key keywords and phrases.

[1119] Memo / Email Analysis: The server analyzes text data (memos, emails) and extracts important information using natural language processing techniques (e.g., SpaCy or NLTK).

[1120] Memories generation mode

[1121] When a user selects the reminiscence mode, the server generates a reminiscence based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server generates an answer using a natural language generation model (e.g., GPT-3) based on the metadata of the corresponding photo and related anecdotes. The generated answer is sent to the device and displayed to the user.

[1122] Specific examples

[1123] Example of memory generation mode

[1124] The user launches the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the photo's metadata and generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken as part of a family trip," which is sent to the device and displayed to the user.

[1125] Example prompt for a generative AI model:

[1126] "Generate an episode about a photo taken by a user on a family trip in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Tell them that the lavender fields were very beautiful."

[1127] Personality Reproduction Mode

[1128] When a user selects the personality mode, the server learns the user's writing and speaking style based on the analyzed memo and email data. For example, if a user requests to "write a birthday message for a friend," the server analyzes past emails and memos and generates a message based on the user's style.

[1129] Specific examples

[1130] Specific examples of personality reproduction mode

[1131] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[1132] Example prompt for a generative AI model:

[1133] "We've learned about the user's writing and speaking style from their past emails and notes. Generate the following birthday message for a friend. This user prefers to write in a friendly, warm style."

[1134] User Interface and Operation

[1135] The user interface includes a login screen, a mode selection screen, a question input screen, and a result display screen. The user interacts with the system through these screens. The operation procedure is as follows:

[1136] 1. The user launches the app and logs in.

[1137] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1138] 3. The user enters and submits a question or request.

[1139] 4. The server parses the data and generates the results.

[1140] 5. The results are displayed on the device and feedback is given to the user.

[1141] In this way, users can utilize past data, reminisce, and generate texts that suit their own style. This system allows users to effectively utilize data stored in cloud storage and enjoy a richer digital experience.

[1142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1143] Step 1: User launches the app and logs in.

[1144] Input: User login information (username, password)

[1145] Data processing / calculation: The server verifies and authenticates the login information, allowing the user to access cloud storage.

[1146] Output: Authentication success message and app home screen

[1147] Step 2: The user grants permission to access the cloud storage.

[1148] Input: A dialog where the user asks for permission to access a cloud storage service

[1149] Data processing / calculation: The server authenticates using OAuth2.0 and establishes a secure connection to the user's cloud storage.

[1150] Output: Obtain cloud storage access permission and display the app mode selection screen.

[1151] Step 3: The user selects "reminiscence mode" or "personality reproduction mode."

[1152] Input: User mode selection (reminiscence mode or personality mode)

[1153] Data processing / calculation: The server sends instructions to the device to display the UI according to the selected mode.

[1154] Output: Screen display according to selected mode

[1155] Step 4: The user enters and submits a question or request.

[1156] Input: A user-typed question or request (e.g., "Where was this photo taken?")

[1157] Data processing / calculation: The server analyzes the user's input and performs appropriate data collection and analysis processing.

[1158] Output: Analysis results of the question or request

[1159] Step 5: The server collects the data from the cloud storage.

[1160] Input: User cloud storage access permissions, data information to be collected

[1161] Data processing / calculation: The server uses the Google Drive API or Dropbox API to download data such as photos, audio, notes, and emails.

[1162] Output: Collected data (photos, audio, notes, emails)

[1163] Step 6: The server analyzes the collected data.

[1164] Input: Collected data

[1165] Data processing / calculation: The server uses machine learning models (e.g., TensorFlow, PyTorch) to perform the following analysis:

[1166] Photo data analysis: Extract metadata and content using image recognition algorithms

[1167] Analysis of audio data: Transcription and content extraction using speech recognition technology

[1168] Memo and email analysis: Extracting important information using natural language processing techniques

[1169] Output: Analyzed data (metadata, transcription results, extracted key information)

[1170] Step 7: The server generates the reminiscence and text.

[1171] Input: Analyzed data, user questions and requests

[1172] Data processing / computation: Generate reminiscences and text using a natural language generation model (e.g., GPT-3). This involves feeding prompts into the generative AI model.

[1173] Specific Action: Example: "Generate an episode about a photo of a family trip taken by a user in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Please tell us that the lavender fields were very beautiful."

[1174] Output: Generated reminiscences and text

[1175] Step 8: Provide the generated results to the user.

[1176] Input: Generated reminiscences and text

[1177] Data processing / calculation: The server sends the generated data to the terminal and displays it on the screen.

[1178] Output: Reminiscences and text displayed on the user's terminal

[1179] In this way, by clarifying the specific operations and the associated data input and output at each processing step, users can make full use of past data through the system, enjoy reminiscing, and generate individual sentences.

[1180] (Application example 1)

[1181] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1182] In recent years, the amount of data stored in cloud storage has increased, but methods for effectively utilizing this data are limited. There is also a need for methods to provide personalized services using users' past data and systems that generate accurate responses to user requests. In particular, for food delivery services, functions such as recommendations based on past order history and message generation for special occasions are effective. The objective of this invention is to realize a system that analyzes and utilizes data in cloud storage to provide users with reminiscences, personalized recommendations, and message generation based on personality reproduction.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1184] In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, and means for making individualized recommendations based on the data analysis, thereby enabling personalized recommendations based on the user's past data, the generation of stories of memories, and the generation of messages for special occasions.

[1185] "Cloud storage" is an online storage service for storing and managing data via the Internet.

[1186] "Data collection means" refers to a means for obtaining a user's past data from cloud storage.

[1187] "Data analysis means" refers to means for analyzing collected data and extracting necessary information.

[1188] The "means for providing a service" is a means for providing a service in a plurality of modes in response to a user's selection.

[1189] "Personalized recommendations" means making personalized recommendations to users based on data analysis.

[1190] The "mode for generating reminiscences" is a mode for generating reminiscences based on the user's past photos, voice, memos and emails.

[1191] "Personality reproduction mode" is a mode that reproduces the user's thought patterns, writing style, and speaking style based on their past data.

[1192] This invention is a system that collects and analyzes data stored in a user's cloud storage, and provides reminiscences, personalized recommendations, and message generation that recreates the user's personality. Specific embodiments are described below.

[1193] System configuration

[1194] This system is composed of a server including a data collection means, a data analysis means, a service provision means, and a personalized recommendation means, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1195] Data collection methods

[1196] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. In this process, protecting the user's privacy and consent to data collection are important.

[1197] Data Analysis Methods

[1198] The server analyzes the collected data and uses machine learning models to extract the necessary information. This allows it to accurately analyze photo metadata, audio content, and note content. It uses the ImageAI library (ResNet model) for image classification and textgenrnn for text analysis.

[1199] Service delivery and personalized recommendation methods

[1200] The server provides services in three modes depending on the user's selection: memory generation mode, personality reproduction mode, and personalized recommendation mode, allowing the user to analyze past order data and messages to obtain a personalized customer experience.

[1201] Memories generation mode

[1202] When a user asks about a specific photo, "Where was this photo taken?", the server analyzes the photo's metadata from cloud storage and generates a story about the photo, such as "This photo was taken in Biei, Hokkaido in the summer of 2018. It was taken during a family trip, and the lavender fields were beautiful," which is then displayed on the device.

[1203] Personality Reproduction Mode

[1204] When a user requests a retirement speech, the server analyzes past emails and memos to learn the user's writing and speaking style. Based on this, it generates a speech such as, "Thank you all for joining us today. I am truly grateful to have had the opportunity to work with you all," and displays it on the device.

[1205] Personalized recommendation mode

[1206] Based on data on the food a user has ordered in the past, the system makes personalized recommendations such as, "Did you like the pizza you ordered for your birthday last year?" or "How about this new menu item?" The server analyzes the user's past order history and preferences, generates appropriate recommendations, and displays them on the device.

[1207] User Interface and Operation

[1208] Users interact with the system through terminal applications, which include a login screen, a mode selection screen, a question entry screen, and a result display screen.

[1209] Operating Procedure

[1210] 1. The user launches the app and logs in.

[1211] 2. The user selects "Memories mode," "Personality reproduction mode," or "Individualized recommendation mode" on the mode selection screen.

[1212] 3. The user enters and submits a question or request.

[1213] 4. The server analyzes the data and generates results.

[1214] 5. The generated results are displayed on the device and feedback is provided to the user.

[1215] Prompt Sentence Examples

[1216] When a user uploads a photo and asks, "Tell me what memories you have of this photo," the server analyzes the photo and tells them, "This photo is of the pizza I ordered for my birthday last year, and it was delicious."

[1217] In this way, users can enjoy past memories and receive personalized recommendations and thoughtful messages. This system allows users to effectively utilize past data and enjoy a richer experience.

[1218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1219] Step 1:

[1220] The user starts the application on the terminal and logs in.

[1221] Input: User login information (ID, password).

[1222] Output: Notifies the success or failure of user authentication.

[1223] Specific operation: The user enters their ID and password into the login screen, and the server receives this and performs authentication.

[1224] Step 2:

[1225] The user selects "reminiscence mode," "personality reproduction mode," or "individualized recommendation mode" on the mode selection screen.

[1226] Input: User mode selection.

[1227] Output: The screen of the selected mode.

[1228] Specific operation: The user selects a mode according to the purpose, and the server prepares for data collection and analysis accordingly.

[1229] Step 3:

[1230] The user enters and submits a question or request.

[1231] Input: The user's question or request (e.g., "Tell me what you remember about this photo").

[1232] Output: Success or failure of sending data to the server.

[1233] Specific operation: The application sends the question or request entered by the user to the server.

[1234] Step 4:

[1235] The server collects the user's data from the cloud storage.

[1236] Input: User permissions and cloud storage access information.

[1237] Output: Collected data such as photos, audio, notes, emails, etc.

[1238] Specific operation: The server retrieves the required data using the cloud storage API.

[1239] Step 5:

[1240] The server analyzes the collected data.

[1241] Input: Data collected in step 4.

[1242] Output: Information based on the parsed metadata and content.

[1243] Specific operation: The ResNet model from the ImageAI library is used for image classification, and textgenrnn is used for text analysis, analyzing photo metadata, audio content, note contents, etc.

[1244] Step 6:

[1245] The server generates a service based on the analysis results.

[1246] Input: Analyzed data and user questions or requests.

[1247] Output: Generated memories, personality reenactments, and personalized recommendations.

[1248] Specific behavior: For example, in the reminiscence mode, it generates a narrative such as, "This photo was taken in Biei, Hokkaido in the summer of 2018."

[1249] Step 7:

[1250] The server generates a response that is sent to the terminal and displayed to the user.

[1251] Input: Generated reminiscences, personality reenactments, and personalized recommendations.

[1252] Output: The results that are displayed to the user on the application screen.

[1253] Specific operation: The server sends the generated content to the terminal, and the application displays it.

[1254] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1255] This invention combines an emotion engine with a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. In particular, the aim is to improve the quality and user experience of the memory generation mode and personality reproduction mode by recognizing the user's emotions.

[1256] System configuration

[1257] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, a personality reproduction mode, and an emotion engine, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1258] 1. Data Collection Methods

[1259] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[1260] 2. Data analysis methods

[1261] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[1262] 3. Emotion Engine

[1263] The server is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their current emotional state and generate an appropriate response.

[1264] 4. Memories Generation Mode

[1265] The server generates a reminiscence based on the analyzed data. It also takes into account the data from the emotion engine and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[1266] 5. Personality Reproduction Mode

[1267] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[1268] Specific examples

[1269] 1. Example of the reminiscence generation mode

[1270] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[1271] 2. Specific examples of personality reproduction mode

[1272] The user launches the app, logs in, and selects "Personality Reproduction Mode." The emotion engine recognizes that the user's tone of voice indicates nervousness. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, the server generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for everything you've done. I'd especially like to thank Tanaka-san, who helped me with Project X. I will continue to walk a new path, keeping the time we spent together in my heart. I'm a little nervous, but I truly appreciate it." The speech is then sent to the device and displayed to the user.

[1273] User Interface and Operation

[1274] User Interface

[1275] Users interact with the system through a device app, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. It may also display real-time emotion analysis results from an emotion engine.

[1276] Operating Procedure

[1277] 1. The user launches the app and logs in.

[1278] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1279] 3. The user enters and submits a question or request.

[1280] 4. The server analyzes the data and generates results using the emotion engine.

[1281] 5. The results are displayed on the device and feedback is given to the user.

[1282] In this way, users can share their past memories while feeling the emotions involved, and generate sentences that reflect their emotional state. This system allows users to effectively utilize past data and enjoy richer life experiences.

[1283] The processing flow will be explained below.

[1284] Processing steps of data collection instruments

[1285] Step 1:

[1286] The user taps on the app icon to launch it, and the app is launched.

[1287] Step 2:

[1288] The user enters their ID and password on the login screen and presses the "Login" button.

[1289] Step 3:

[1290] The server checks the entered authentication information against a database and performs authentication.

[1291] Step 4:

[1292] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[1293] Step 5:

[1294] The device displays a permission dialog to the user requesting permission to collect data.

[1295] Step 6:

[1296] The user selects "Allow" in the permission dialog.

[1297] Step 7:

[1298] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[1299] Step 8:

[1300] The server stores the acquired data in storage in preparation for analysis.

[1301] Processing steps of data analysis method

[1302] Step 1:

[1303] The server retrieves the collected data and begins analyzing it.

[1304] Step 2:

[1305] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[1306] Step 3:

[1307] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[1308] Step 4:

[1309] The server classifies the extracted information and stores it in a database.

[1310] Emotion Engine Processing Steps

[1311] Step 1:

[1312] The device captures the user's voice and facial expressions in real time.

[1313] Step 2:

[1314] The device sends the captured data to the server.

[1315] Step 3:

[1316] The server analyzes the received voice and facial expression data.

[1317] Step 4:

[1318] The server uses an emotion engine to recognize the user's emotional state (e.g., joy, sadness, tension, etc.).

[1319] Step 5:

[1320] The emotional data analyzed by the emotion engine is reflected in related processing (memory story generation and personality reproduction).

[1321] Processing steps for reminiscence generation mode

[1322] Step 1:

[1323] The user selects "Memories mode" on the mode selection screen within the app.

[1324] Step 2:

[1325] The user enters a question into the question input screen and presses the "Submit" button.

[1326] Step 3:

[1327] The terminal sends the user's question to the server.

[1328] Step 4:

[1329] The server analyzes the user's question using a natural language processing model.

[1330] Step 5:

[1331] The server searches the database for relevant memories based on the analysis results and emotion engine data.

[1332] Step 6:

[1333] The server uses a reminiscence generation model to generate a reminiscence that is in line with the user's emotions.

[1334] Step 7:

[1335] The terminal displays the reminiscences received from the server to the user.

[1336] Personality Reproduction Mode Processing Steps

[1337] Step 1:

[1338] The user selects "Personality Reproduction Mode" on the mode selection screen within the app.

[1339] Step 2:

[1340] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[1341] Step 3:

[1342] The terminal transmits the user's request to the server.

[1343] Step 4:

[1344] The server retrieves relevant past data and analyzes the user's thought patterns, writing style, and speaking style.

[1345] Step 5:

[1346] The server takes into account the data from the emotion engine and generates appropriate sentences according to the user's emotional state.

[1347] Step 6:

[1348] The server sends the generated text to the terminal.

[1349] Step 7:

[1350] The terminal displays the generated text to the user.

[1351] Example 2

[1352] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1353] The purpose of this invention is to provide a richer user experience by utilizing the user's past data stored in cloud storage. However, conventional technologies simply display data, making it difficult to provide services that take the user's emotional state into account. Furthermore, there is a problem in that the quality of reminiscences and personality reproduction is low because emotions are not recognized.

[1354] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means for providing an emotion engine for analyzing emotions from the user's voice and facial expressions, means for generating reminiscences based on the user's emotional state, and means for generating sentences based on the user's writing and speaking style. This makes it possible to provide high-quality reminiscences according to the user's emotional state and improve the accuracy of personality reproduction.

[1355] "Cloud storage" is a storage service on a remote server that stores and manages data over the Internet.

[1356] "User's past data" refers to various data that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[1357] "Means of collection" refers to the functionality for obtaining the required data from authorized cloud storage.

[1358] "Means of analysis" refers to the ability to extract useful information from collected data using machine learning models and algorithms.

[1359] "Means for providing services" refers to the function of generating content such as reminiscences and personality reenactments based on analyzed data and providing it to users.

[1360] "Emotion engine" refers to technology that analyzes a user's voice and facial expressions to identify their current emotional state.

[1361] "Means for generating reminiscences" refers to a function for automatically generating reminiscences for a user based on analyzed past data and the user's current emotional state.

[1362] "Means of personality reproduction" refers to a function that analyzes the user's past data and generates sentences based on the user's writing and speaking style.

[1363] This invention is a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. By incorporating an emotion engine into this system, the quality and user experience of the reminiscence generation mode and personality reproduction mode can be improved. Specific embodiments are described below.

[1364] System configuration

[1365] This system consists of a server that performs a series of processes, such as data collection means, data analysis means, memory generation mode, personality reproduction mode, and emotion engine, a terminal that acts as a user interface, and a series of means for interaction with the user.

[1366] Data collection methods

[1367] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from cloud storage (e.g., a general cloud storage service). In this process, it is important to protect the user's privacy and obtain consent for data collection.

[1368] Data Analysis Methods

[1369] The server uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the collected data. This allows it to accurately analyze photo metadata, audio content, and note entries. The analysis results are used to generate reminiscences and recreate personalities.

[1370] Emotion Engine

[1371] The server is equipped with an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) that analyzes emotions from the user's voice and facial expressions, allowing it to recognize the user's current emotional state and generate an appropriate response.

[1372] Memories generation mode

[1373] The server generates a reminiscence based on the analyzed data. It also takes into account the emotion engine data and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[1374] Examples:

[1375] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[1376] Personality Reproduction Mode

[1377] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[1378] Examples:

[1379] The user starts the app, logs in, and selects "Personality Mode." The emotion engine recognizes that the user's tone of voice sounds tense. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, it generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for your support over the years. I'm especially grateful to everyone who helped me on this project. I'm truly grateful, but I'm nervous about taking on this new challenge." The speech is then sent to the device and displayed to the user.

[1380] User Interface and Operation

[1381] User Interface:

[1382] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. In addition, the app may display real-time emotion analysis results from an emotion engine.

[1383] Operation steps:

[1384] 1. The user launches the app and logs in.

[1385] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1386] 3. The user enters and submits a question or request.

[1387] 4. The server analyzes the data and generates results using the emotion engine.

[1388] 5. The results are displayed on the device and feedback is given to the user.

[1389] This process allows users to share their past memories while feeling the emotions involved, and generate text that reflects their emotional state. This system allows users to effectively utilize past data and enjoy richer life experiences.

[1390] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1391] Step 1:

[1392] The user launches the app and logs in. The user enters their username and password on the device's login screen and clicks the login button. The server checks the user's authentication information against the database, and if authentication is successful, it generates a JWT (JSON Web Token) and sends it to the device. This starts the user's session. The output is the home screen with the user logged in.

[1393] Step 2:

[1394] The user selects a mode. The user selects "Memories Mode" or "Personality Reproduction Mode" on the home screen. The device sends the user's selection to the server. The server records the mode selected by the user and prepares the system to proceed to the next step. The input is an instruction to select a mode, and the output is confirmation of the selected mode.

[1395] Step 3:

[1396] The server obtains consent to collect data. The server displays a dialog to the user requesting permission to access cloud storage. If the user agrees, the server obtains the necessary data access rights from the cloud storage using the OAuth 2.0 protocol. The input is consent information, and the output is the acquisition of access rights. The server collects photos, audio, notes, emails, etc. from the cloud storage service and stores them in a temporary storage area.

[1397] Step 4:

[1398] The server analyzes the collected data. The server analyzes the data using machine learning models (e.g., TensorFlow, PyTorch). Specific operations include extracting metadata from photos, transcribing audio data, and analyzing the content of notes. The input is the collected data, and the output is the analysis results. The analysis results are used in the next step.

[1399] Step 5:

[1400] The server performs emotion analysis using an emotion engine. The server inputs the user's voice and facial expression data into an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) to analyze the user's emotional state. The input is the user's voice and facial expression data, and the output is data on the user's emotional state (e.g., joy, sadness, tension). This allows the user's current emotional state to be quantified.

[1401] Step 6:

[1402] The server generates content based on emotions. Based on the analysis results and the emotion engine results, the server generates sentences for reminiscences or personality reenactment. To do this, a prompt sentence is input to a generative AI model (e.g., GPT-3, BERT), which then generates an appropriate sentence. The input is the prompt sentence and the analysis results, and the output is the generated sentence.

[1403] Specifically, in the reminiscence mode, the system uses the prompt "Tell me your memories of this photo" and adjusts the emotional tone based on the generated text. In the personality reproduction mode, the system uses the prompt "Write a retirement speech" and adjusts the tone of the generated speech, such as making it more relaxed.

[1404] Step 7:

[1405] The server sends the generated content to the terminal. The server sends the generated text to the terminal in JSON format. The terminal parses the received data and displays it on the user interface. The user can check the displayed results and send feedback accordingly. The input is the data of the generated content, and the output is the user's display screen.

[1406] Through the above steps, users can enjoy high-quality emotional reminiscences and personality reproductions using past data.

[1407] (Application example 2)

[1408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1409] Conventional robot assistant systems in factories have limited capabilities for providing work guidance and employee support, making it difficult to provide appropriate support that takes into account the emotional state of employees. Therefore, there has been a demand for a method that can simultaneously promote efficient work and reduce employee stress.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of users from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means having an emotion engine that analyzes facial expressions and voice in real time, and means for providing appropriate instructions and support according to the user's emotions. This enables efficient work guidance and support for employees while taking into account their emotional state.

[1411] "Cloud storage" is a technology that provides remote servers for storing data over the Internet.

[1412] "User's past data" refers collectively to information that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[1413] "Means of collection" refers to the processes and tools used to obtain the required data from cloud storage.

[1414] "Means of analysis" are methods and techniques for analyzing collected data and extracting useful information.

[1415] "Multiple modes" refer to different types of services that the system can provide, including, for example, a reminiscence generation mode and a personality reproduction mode.

[1416] The "emotion engine" is a technology that analyzes the user's emotional state from voice and facial expression data.

[1417] "Real-time facial expression and voice analysis" refers to the ability to instantly assess a user's current emotional state.

[1418] "Means for providing instructions and support" refers to mechanisms for providing specific actions or advice to users based on the analysis results.

[1419] The "mode for generating memories" is a function that generates stories that evoke memories based on the user's past data.

[1420] "Personality Reproduction Mode" is a function that imitates the user's personality based on their past actions and documents, and generates responses in real time.

[1421] The "work instruction mode" is a function in which the robot instructs employees on how to do their work in a factory or other environment.

[1422] "Production history analysis mode" is a function that analyzes past data to evaluate employee work performance.

[1423] "Voicing" refers to the act of the robot providing verbal advice or instructions depending on the employee's emotional state.

[1424] This paper describes an embodiment of a system equipped with an emotion engine for factory robots that incorporates a "production history analysis mode" and a "work guidance mode." This system uses a robot assistant in a factory to analyze the emotional state of employees in real time and provide appropriate guidance and support. The main hardware and software configurations and their processing flow are described.

[1425] Hardware Configuration

[1426] 1. Factory robots

[1427] Supports all operations required for operation.

[1428] 2. Camera and microphone

[1429] Capture employees' facial expressions and voices.

[1430] 3. Cloud Storage

[1431] A remote server for storing employee historical data.

[1432] Software Configuration

[1433] 1. Machine Learning Model

[1434] TensorFlow and Google Cloud AI are used for data analysis.

[1435] 2. Emotion Engine

[1436] Affectiva and Microsoft Azure Emotion API are used as technologies to analyze emotions from voice and facial expressions.

[1437] 3. Robot Control Software

[1438] Control the robot's movements using ROS (Robot Operating System).

[1439] 4. User Interface

[1440] Users interact with the system via factory tablets and displays.

[1441] Program processing flow

[1442] 1. Data Collection

[1443] The server collects users' past data from cloud storage, including photos, audio, notes, and emails.

[1444] Cameras and microphones are used to collect employees' facial expressions and voices in real time.

[1445] 2. Data Analysis

[1446] The collected data is analyzed using machine learning models on a cloud server.

[1447] The emotion engine analyzes the employee's emotional state in real time, determining whether they are tired or tense.

[1448] 3. Memories generation mode and personality reproduction mode

[1449] The server utilizes historical data collected and real-time emotional data to generate appropriate topics and tones depending on the user's emotional state.

[1450] For example, if the user is recognized as depressed, when asked "Tell me about this photo," the memory-generating mode will respond in a gentle tone with, "This photo was taken in the summer of 2018 of the lavender fields in Biei, Hokkaido."

[1451] 4. Production history analysis mode

[1452] The server analyzes past data and real-time emotional data to evaluate the employee's work performance. For example, if the server determines that the employee is tired, the robot will advise them to "pause your work and take a short break."

[1453] 5. Work Instruction Mode

[1454] The server will guide employees through the appropriate work process based on their emotional state, explaining to a nervous employee in a relaxing voice, "Let's simplify the next step."

[1455] Specific examples

[1456] Prompt Sentence Examples

[1457] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[1458] This will realize a system that provides efficient work guidance and support to employees while taking into account their emotional state. Specifically, if an employee is tired, the system will pause work to encourage them to rest, and it will also simplify work processes to reduce stress.

[1459] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1460] Step 1:

[1461] The server collects users' past data from cloud storage. The input requires the cloud storage API key and user authentication information, and the output includes photos, audio, notes, emails, etc. This data is stored digitally and sent to the next analysis step.

[1462] Step 2:

[1463] The server collects facial expression and voice data of employees in real time using a camera and microphone. Real-time data from the camera and microphone is required as input, and facial expression video data and voice data are obtained as output. This data is sent to the emotion engine to analyze the employee's current emotional state.

[1464] Step 3:

[1465] The server analyzes the collected data. It uses as input the previously collected data (output of step 1) and real-time emotion data (output of step 2). This allows for metadata analysis of photos, keyword detection in speech, and text analysis of notes and emails. The output provides key information and insights that are integrated with the emotion engine's analysis.

[1466] Step 4:

[1467] The server uses an emotion engine to analyze the employee's emotional state. It requires real-time collected facial expression and voice data as input, and outputs an emotional status, such as "tired," "tense," or "relaxed." This status is used to generate appropriate instructions for the user in the next step.

[1468] Step 5:

[1469] The server executes the production history analysis mode based on the analysis results. As input, it requires the past data analysis results (output of step 3) and the real-time emotional status (output of step 4). The server evaluates the employee's work performance and generates appropriate instructions and support according to the employee's emotional state. As output, it obtains a specific action plan.

[1470] One specific action is "suggesting rest." For example, if the server determines that an employee is tired, it will generate instructions such as "We recommend that you temporarily stop working and take a short break," and communicate this to the employee via the robot's voice output system.

[1471] Step 6:

[1472] The server executes the work instruction mode. It uses the analysis results (output of Step 5) and emotional status data as input. The server instructs the work procedure according to the employee's emotional state. For example, if the employee is nervous, the server generates instructions encouraging relaxation, such as "Let's perform the next step using a simplified procedure," and the robot conveys this aloud.

[1473] Step 7:

[1474] The server re-stores the collected and analyzed data in cloud storage. The latest analysis results and generated instruction data are required as input, and the updated employee data history is stored in cloud storage as output. This will further increase the accuracy of future analysis and instruction generation.

[1475] Example prompt

[1476] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[1477] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1478] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1479] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1480] [Fourth embodiment]

[1481] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1482] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1483] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1484] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1485] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1486] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1487] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1488] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1489] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1490] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1491] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1492] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1493] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1494] This invention is a system that collects data such as past photos, audio, memos, and emails stored in a user's cloud storage, analyzes the data, and provides a service that recreates memories and the user's personality in various modes.

[1495] System configuration

[1496] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1497] 1. Data Collection Methods

[1498] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[1499] 2. Data analysis methods

[1500] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[1501] 3. Memories Generation Mode

[1502] The server generates a story based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server uses a natural language generation model to generate an answer about the photo's location, date, and time, as well as related episodes. The device then displays the generated answer to the user.

[1503] 4. Personality Reproduction Mode

[1504] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on these. For example, if a user requests "Create a retirement speech," the server learns the user's style from past emails and notes and generates a speech based on that. The generated speech is sent to the user's device and displayed.

[1505] Specific examples

[1506] 1. Example of the reminiscence generation mode

[1507] The user starts the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the metadata of the corresponding photo from cloud storage, generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were very beautiful," sends the answer to the device, and displays it to the user.

[1508] 2. Specific examples of personality reproduction mode

[1509] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[1510] User Interface and Operation

[1511] User Interface

[1512] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question entry screen, and a results display screen.

[1513] Operating Procedure

[1514] 1. The user launches the app and logs in.

[1515] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1516] 3. The user enters and submits a question or request.

[1517] 4. The server parses the data and generates the results.

[1518] 5. The results are displayed on the device and feedback is given to the user.

[1519] In this way, users can enjoy reminiscing about the past and generate sentences that reflect their own personalities. This system allows users to effectively utilize past data and enjoy richer life experiences.

[1520] The processing flow will be explained below.

[1521] Processing steps of data collection instruments

[1522] Step 1:

[1523] The user taps on the app icon to launch it, and the app is launched.

[1524] Step 2:

[1525] The user enters their ID and password on the login screen and presses the "Login" button.

[1526] Step 3:

[1527] The server checks the entered authentication information against a database and performs authentication.

[1528] Step 4:

[1529] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[1530] Step 5:

[1531] The device displays a permission dialog to the user requesting permission to collect data.

[1532] Step 6:

[1533] The user selects "Allow" in the permission dialog.

[1534] Step 7:

[1535] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[1536] Step 8:

[1537] The server stores the acquired data in storage and prepares it for analysis.

[1538] Processing steps of data analysis method

[1539] Step 1:

[1540] The server retrieves the collected data for analysis.

[1541] Step 2:

[1542] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[1543] Step 3:

[1544] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[1545] Step 4:

[1546] The server classifies and organizes the extracted information and stores it in a database so that it can be used to generate memories and recreate personalities.

[1547] Processing steps for reminiscence generation mode

[1548] Step 1:

[1549] The user selects "reminiscence mode" on the mode selection screen.

[1550] Step 2:

[1551] The user enters a question or request into the question input screen and presses the "Submit" button.

[1552] Step 3:

[1553] The terminal sends the user's question to the server.

[1554] Step 4:

[1555] The server analyzes the user's question using a natural language processing model.

[1556] Step 5:

[1557] Based on the analysis results, the server searches the database for relevant memories.

[1558] Step 6:

[1559] The server uses the reminiscence generation model to generate reminiscences in response to the user's questions.

[1560] Step 7:

[1561] The terminal displays the reminiscences received from the server to the user.

[1562] Personality Reproduction Mode Processing Steps

[1563] Step 1:

[1564] The user selects "personality reproduction mode" on the mode selection screen.

[1565] Step 2:

[1566] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[1567] Step 3:

[1568] The terminal transmits the user's request to the server.

[1569] Step 4:

[1570] The server retrieves relevant data to analyze the user's thinking patterns, writing style, and speaking style.

[1571] Step 5:

[1572] The server uses machine learning models to generate sentences based on the user's style.

[1573] Step 6:

[1574] The server sends the generated text to the terminal.

[1575] Step 7:

[1576] The terminal displays the generated text to the user.

[1577] Example 1

[1578] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1579] In today's digitalized society, users store vast amounts of data (photos, audio, notes, emails, etc.) in cloud storage. However, this data simply functions as a storage location and is not used for specific services. It is also difficult to effectively utilize past data to reminisce or recreate a user's writing or speaking style. This prevents users from fully utilizing the potential value of their data.

[1580] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1581] In this invention, the server includes means for collecting data from cloud storage based on the user's access permission, means for analyzing the collected data and extracting metadata and content, means for generating and providing information from the analyzed data in response to the user's request, means for using a natural language generation model to generate reminiscences, and means for learning the user's writing and speaking style and generating text, thereby enabling the user to effectively utilize past data and enjoy a richer digital experience.

[1582] "User" means any person or entity that uses the System.

[1583] "Access permission" refers to the act of authentication by a user to allow the server to access their cloud storage.

[1584] "Cloud storage" refers to an online storage service for storing user data via the Internet.

[1585] "Data" refers to information such as photos, audio, notes, and emails stored in the user's cloud storage.

[1586] "Server" refers to a computer and related hardware and software that executes various processes such as collection, analysis, and generation.

[1587] "Means" refers to the processes, functions, or devices necessary to achieve a certain purpose.

[1588] "Collection" refers to the act of obtaining specific data from cloud storage.

[1589] "Analysis" refers to the process of understanding the content of collected data and extracting metadata and important information.

[1590] "Metadata" refers to additional information about the data (e.g., the date and location of a photo).

[1591] "Information generation" refers to the act of creating new information based on analyzed data.

[1592] "Memories generation mode" refers to a mode that provides the function of generating memories based on past data.

[1593] A "natural language generation model" refers to a machine learning model that generates natural language expressions based on input text information.

[1594] "Learning how to write and speak" refers to the process of analyzing the user's past writing and voice data and learning their characteristics.

[1595] "Text generation" refers to the act of creating new text based on learned features.

[1596] This invention is a system that effectively utilizes past data stored in a user's cloud storage to provide a service that recreates reminiscences and the user's writing and speaking styles. The system is composed of a data collection means, a data analysis means, a server that includes a reminiscence generation mode and a personality reproduction mode, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1597] Data collection methods

[1598] The server obtains the user's permission to collect data from cloud storage. To collect data, it uses common cloud storage APIs such as Google Drive API and Dropbox API. Specifically, after the user launches the app and logs in, the server securely downloads the data by granting permission to access the cloud storage.

[1599] Data Analysis Methods

[1600] The server uses machine learning models (e.g. TensorFlow or PyTorch) to analyze the collected data. The following steps are performed:

[1601] Photo data analysis: The server analyzes the photo data and extracts metadata and content using image recognition algorithms (e.g., OpenCV or TensorFlow models), specifically face detection and scene recognition to determine whether a particular face was captured in a particular location.

[1602] Analyzing the audio data: When the server analyzes the audio data, it uses speech recognition technologies such as Google Cloud Speech-to-Text API or Amazon Transcribe to transcribe the content and extract key keywords and phrases.

[1603] Memo / Email Analysis: The server analyzes text data (memos, emails) and extracts important information using natural language processing techniques (e.g., SpaCy or NLTK).

[1604] Memories generation mode

[1605] When a user selects the reminiscence mode, the server generates a reminiscence based on the analyzed data. For example, if a user asks, "Tell me about this photo," the server generates an answer using a natural language generation model (e.g., GPT-3) based on the metadata of the corresponding photo and related anecdotes. The generated answer is sent to the device and displayed to the user.

[1606] Specific examples

[1607] Example of memory generation mode

[1608] The user launches the app, logs in, and selects "Memories mode." Next, the user asks, "Where was this photo taken?" The server analyzes the photo's metadata and generates an answer such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken as part of a family trip," which is sent to the device and displayed to the user.

[1609] Example prompt for a generative AI model:

[1610] "Generate an episode about a photo taken by a user on a family trip in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Tell them that the lavender fields were very beautiful."

[1611] Personality Reproduction Mode

[1612] When a user selects the personality mode, the server learns the user's writing and speaking style based on the analyzed memo and email data. For example, if a user requests to "write a birthday message for a friend," the server analyzes past emails and memos and generates a message based on the user's style.

[1613] Specific examples

[1614] Specific examples of personality reproduction mode

[1615] The user launches the app, logs in, and selects "Personality Reproduction Mode." Next, the user requests, "Create a birthday message for a friend." The server analyzes the user's past emails and notes and learns the user's writing and speaking characteristics. Based on this, the server generates a message such as, "Happy birthday, dear friend. Your smile always cheers me up. I hope this year will be a wonderful one," sends it to the device, and displays it to the user.

[1616] Example prompt for a generative AI model:

[1617] "We've learned about the user's writing and speaking style from their past emails and notes. Generate the following birthday message for a friend. This user prefers to write in a friendly, warm style."

[1618] User Interface and Operation

[1619] The user interface includes a login screen, a mode selection screen, a question input screen, and a result display screen. The user interacts with the system through these screens. The operation procedure is as follows:

[1620] 1. The user launches the app and logs in.

[1621] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1622] 3. The user enters and submits a question or request.

[1623] 4. The server parses the data and generates the results.

[1624] 5. The results are displayed on the device and feedback is given to the user.

[1625] In this way, users can utilize past data, reminisce, and generate texts that suit their own style. This system allows users to effectively utilize data stored in cloud storage and enjoy a richer digital experience.

[1626] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1627] Step 1: User launches the app and logs in.

[1628] Input: User login information (username, password)

[1629] Data processing / calculation: The server verifies and authenticates the login information, allowing the user to access cloud storage.

[1630] Output: Authentication success message and app home screen

[1631] Step 2: The user grants permission to access the cloud storage.

[1632] Input: A dialog where the user asks for permission to access a cloud storage service

[1633] Data processing / calculation: The server authenticates using OAuth2.0 and establishes a secure connection to the user's cloud storage.

[1634] Output: Obtain cloud storage access permission and display the app mode selection screen.

[1635] Step 3: The user selects "reminiscence mode" or "personality reproduction mode."

[1636] Input: User mode selection (reminiscence mode or personality mode)

[1637] Data processing / calculation: The server sends instructions to the device to display the UI according to the selected mode.

[1638] Output: Screen display according to selected mode

[1639] Step 4: The user enters and submits a question or request.

[1640] Input: A user-typed question or request (e.g., "Where was this photo taken?")

[1641] Data processing / calculation: The server analyzes the user's input and performs appropriate data collection and analysis processing.

[1642] Output: Analysis results of the question or request

[1643] Step 5: The server collects the data from the cloud storage.

[1644] Input: User cloud storage access permissions, data information to be collected

[1645] Data processing / calculation: The server uses the Google Drive API or Dropbox API to download data such as photos, audio, notes, and emails.

[1646] Output: Collected data (photos, audio, notes, emails)

[1647] Step 6: The server analyzes the collected data.

[1648] Input: Collected data

[1649] Data processing / calculation: The server uses machine learning models (e.g., TensorFlow, PyTorch) to perform the following analysis:

[1650] Photo data analysis: Extract metadata and content using image recognition algorithms

[1651] Analysis of audio data: Transcription and content extraction using speech recognition technology

[1652] Memo and email analysis: Extracting important information using natural language processing techniques

[1653] Output: Analyzed data (metadata, transcription results, extracted key information)

[1654] Step 7: The server generates the reminiscence and text.

[1655] Input: Analyzed data, user questions and requests

[1656] Data processing / computation: Generate reminiscences and text using a natural language generation model (e.g., GPT-3). This involves feeding prompts into the generative AI model.

[1657] Specific Action: Example: "Generate an episode about a photo of a family trip taken by a user in the summer of 2018. According to the metadata, this photo was taken in Biei, Hokkaido. Please tell us that the lavender fields were very beautiful."

[1658] Output: Generated reminiscences and text

[1659] Step 8: Provide the generated results to the user.

[1660] Input: Generated reminiscences and text

[1661] Data processing / calculation: The server sends the generated data to the terminal and displays it on the screen.

[1662] Output: Reminiscences and text displayed on the user's terminal

[1663] In this way, by clarifying the specific operations and the associated data input and output at each processing step, users can make full use of past data through the system, enjoy reminiscing, and generate individual sentences.

[1664] (Application example 1)

[1665] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1666] In recent years, the amount of data stored in cloud storage has increased, but methods for effectively utilizing this data are limited. There is also a need for methods to provide personalized services using users' past data and systems that generate accurate responses to user requests. In particular, for food delivery services, functions such as recommendations based on past order history and message generation for special occasions are effective. The objective of this invention is to realize a system that analyzes and utilizes data in cloud storage to provide users with reminiscences, personalized recommendations, and message generation based on personality reproduction.

[1667] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1668] In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, and means for making individualized recommendations based on the data analysis, thereby enabling personalized recommendations based on the user's past data, the generation of stories of memories, and the generation of messages for special occasions.

[1669] "Cloud storage" is an online storage service for storing and managing data via the Internet.

[1670] "Data collection means" refers to a means for obtaining a user's past data from cloud storage.

[1671] "Data analysis means" refers to means for analyzing collected data and extracting necessary information.

[1672] The "means for providing a service" is a means for providing a service in a plurality of modes in response to a user's selection.

[1673] "Personalized recommendations" means making personalized recommendations to users based on data analysis.

[1674] The "mode for generating reminiscences" is a mode for generating reminiscences based on the user's past photos, voice, memos and emails.

[1675] "Personality reproduction mode" is a mode that reproduces the user's thought patterns, writing style, and speaking style based on their past data.

[1676] This invention is a system that collects and analyzes data stored in a user's cloud storage, and provides reminiscences, personalized recommendations, and message generation that recreates the user's personality. Specific embodiments are described below.

[1677] System configuration

[1678] This system is composed of a server including a data collection means, a data analysis means, a service provision means, and a personalized recommendation means, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1679] Data collection methods

[1680] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. In this process, protecting the user's privacy and consent to data collection are important.

[1681] Data Analysis Methods

[1682] The server analyzes the collected data and uses machine learning models to extract the necessary information. This allows it to accurately analyze photo metadata, audio content, and note content. It uses the ImageAI library (ResNet model) for image classification and textgenrnn for text analysis.

[1683] Service delivery and personalized recommendation methods

[1684] The server provides services in three modes depending on the user's selection: memory generation mode, personality reproduction mode, and personalized recommendation mode, allowing the user to analyze past order data and messages to obtain a personalized customer experience.

[1685] Memories generation mode

[1686] When a user asks about a specific photo, "Where was this photo taken?", the server analyzes the photo's metadata from cloud storage and generates a story about the photo, such as "This photo was taken in Biei, Hokkaido in the summer of 2018. It was taken during a family trip, and the lavender fields were beautiful," which is then displayed on the device.

[1687] Personality Reproduction Mode

[1688] When a user requests a retirement speech, the server analyzes past emails and memos to learn the user's writing and speaking style. Based on this, it generates a speech such as, "Thank you all for joining us today. I am truly grateful to have had the opportunity to work with you all," and displays it on the device.

[1689] Personalized recommendation mode

[1690] Based on data on the food a user has ordered in the past, the system makes personalized recommendations such as, "Did you like the pizza you ordered for your birthday last year?" or "How about this new menu item?" The server analyzes the user's past order history and preferences, generates appropriate recommendations, and displays them on the device.

[1691] User Interface and Operation

[1692] Users interact with the system through terminal applications, which include a login screen, a mode selection screen, a question entry screen, and a result display screen.

[1693] Operating Procedure

[1694] 1. The user launches the app and logs in.

[1695] 2. The user selects "Memories mode," "Personality reproduction mode," or "Individualized recommendation mode" on the mode selection screen.

[1696] 3. The user enters and submits a question or request.

[1697] 4. The server analyzes the data and generates results.

[1698] 5. The generated results are displayed on the device and feedback is provided to the user.

[1699] Prompt Sentence Examples

[1700] When a user uploads a photo and asks, "Tell me what memories you have of this photo," the server analyzes the photo and tells them, "This photo is of the pizza I ordered for my birthday last year, and it was delicious."

[1701] In this way, users can enjoy past memories and receive personalized recommendations and thoughtful messages. This system allows users to effectively utilize past data and enjoy a richer experience.

[1702] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1703] Step 1:

[1704] The user starts the application on the terminal and logs in.

[1705] Input: User login information (ID, password).

[1706] Output: Notifies the success or failure of user authentication.

[1707] Specific operation: The user enters their ID and password into the login screen, and the server receives this and performs authentication.

[1708] Step 2:

[1709] The user selects "reminiscence mode," "personality reproduction mode," or "individualized recommendation mode" on the mode selection screen.

[1710] Input: User mode selection.

[1711] Output: The screen of the selected mode.

[1712] Specific operation: The user selects a mode according to the purpose, and the server prepares for data collection and analysis accordingly.

[1713] Step 3:

[1714] The user enters and submits a question or request.

[1715] Input: The user's question or request (e.g., "Tell me what you remember about this photo").

[1716] Output: Success or failure of sending data to the server.

[1717] Specific operation: The application sends the question or request entered by the user to the server.

[1718] Step 4:

[1719] The server collects the user's data from the cloud storage.

[1720] Input: User permissions and cloud storage access information.

[1721] Output: Collected data such as photos, audio, notes, emails, etc.

[1722] Specific operation: The server retrieves the required data using the cloud storage API.

[1723] Step 5:

[1724] The server analyzes the collected data.

[1725] Input: Data collected in step 4.

[1726] Output: Information based on the parsed metadata and content.

[1727] Specific operation: The ResNet model from the ImageAI library is used for image classification, and textgenrnn is used for text analysis, analyzing photo metadata, audio content, note contents, etc.

[1728] Step 6:

[1729] The server generates a service based on the analysis results.

[1730] Input: Analyzed data and user questions or requests.

[1731] Output: Generated memories, personality reenactments, and personalized recommendations.

[1732] Specific behavior: For example, in the reminiscence mode, it generates a narrative such as, "This photo was taken in Biei, Hokkaido in the summer of 2018."

[1733] Step 7:

[1734] The server generates a response that is sent to the terminal and displayed to the user.

[1735] Input: Generated reminiscences, personality reenactments, and personalized recommendations.

[1736] Output: The results that are displayed to the user on the application screen.

[1737] Specific operation: The server sends the generated content to the terminal, and the application displays it.

[1738] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1739] This invention combines an emotion engine with a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. In particular, the aim is to improve the quality and user experience of the memory generation mode and personality reproduction mode by recognizing the user's emotions.

[1740] System configuration

[1741] This system consists of a server that performs a series of processes including data collection means, data analysis means, a memory generation mode, a personality reproduction mode, and an emotion engine, a terminal that acts as a user interface, and a series of means for interacting with the user.

[1742] 1. Data Collection Methods

[1743] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from the cloud storage. It is important to protect the user's privacy and obtain consent for data collection in this process.

[1744] 2. Data analysis methods

[1745] The server analyzes the collected data and uses machine learning models to extract the necessary information, such as photo metadata, audio content, and note entries.

[1746] 3. Emotion Engine

[1747] The server is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their current emotional state and generate an appropriate response.

[1748] 4. Memories Generation Mode

[1749] The server generates a reminiscence based on the analyzed data. It also takes into account the data from the emotion engine and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[1750] 5. Personality Reproduction Mode

[1751] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[1752] Specific examples

[1753] 1. Example of the reminiscence generation mode

[1754] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[1755] 2. Specific examples of personality reproduction mode

[1756] The user launches the app, logs in, and selects "Personality Reproduction Mode." The emotion engine recognizes that the user's tone of voice indicates nervousness. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, the server generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for everything you've done. I'd especially like to thank Tanaka-san, who helped me with Project X. I will continue to walk a new path, keeping the time we spent together in my heart. I'm a little nervous, but I truly appreciate it." The speech is then sent to the device and displayed to the user.

[1757] User Interface and Operation

[1758] User Interface

[1759] Users interact with the system through a device app, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. It may also display real-time emotion analysis results from an emotion engine.

[1760] Operating Procedure

[1761] 1. The user launches the app and logs in.

[1762] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1763] 3. The user enters and submits a question or request.

[1764] 4. The server analyzes the data and generates results using the emotion engine.

[1765] 5. The results are displayed on the device and feedback is given to the user.

[1766] In this way, users can share their past memories while feeling emotionally attached to them, and generate sentences that correspond to their emotional state.This system allows users to effectively utilize past data and enjoy richer life experiences.

[1767] The processing flow will be explained below.

[1768] Processing steps of data collection instruments

[1769] Step 1:

[1770] The user taps on the app icon to launch it, and the app is launched.

[1771] Step 2:

[1772] The user enters their ID and password on the login screen and presses the "Login" button.

[1773] Step 3:

[1774] The server checks the entered authentication information against a database and performs authentication.

[1775] Step 4:

[1776] If the server is successful in the authentication, it obtains access rights to the user's cloud storage.

[1777] Step 5:

[1778] The device displays a permission dialog to the user requesting permission to collect data.

[1779] Step 6:

[1780] The user selects "Allow" in the permission dialog.

[1781] Step 7:

[1782] The server retrieves data such as photos, audio, notes, and emails from the user's cloud storage.

[1783] Step 8:

[1784] The server stores the acquired data in storage in preparation for analysis.

[1785] Processing steps of data analysis method

[1786] Step 1:

[1787] The server retrieves the collected data and begins analyzing it.

[1788] Step 2:

[1789] The server uses machine learning models to preprocess the data and extract metadata and relevant information.

[1790] Step 3:

[1791] The server analyzes the date and time the photo was taken, the location, people, converts voice into text, and the contents of notes and emails.

[1792] Step 4:

[1793] The server classifies the extracted information and stores it in a database.

[1794] Emotion Engine Processing Steps

[1795] Step 1:

[1796] The device captures the user's voice and facial expressions in real time.

[1797] Step 2:

[1798] The device sends the captured data to the server.

[1799] Step 3:

[1800] The server analyzes the received voice and facial expression data.

[1801] Step 4:

[1802] The server uses an emotion engine to recognize the user's emotional state (e.g., joy, sadness, tension, etc.).

[1803] Step 5:

[1804] The emotional data analyzed by the emotion engine is reflected in related processing (memory story generation and personality reproduction).

[1805] Processing steps for reminiscence generation mode

[1806] Step 1:

[1807] The user selects "Memories mode" on the mode selection screen within the app.

[1808] Step 2:

[1809] The user enters a question into the question input screen and presses the "Submit" button.

[1810] Step 3:

[1811] The terminal sends the user's question to the server.

[1812] Step 4:

[1813] The server analyzes the user's question using a natural language processing model.

[1814] Step 5:

[1815] The server searches the database for relevant memories based on the analysis results and emotion engine data.

[1816] Step 6:

[1817] The server uses a reminiscence generation model to generate a reminiscence that is in line with the user's emotions.

[1818] Step 7:

[1819] The terminal displays the reminiscences received from the server to the user.

[1820] Personality Reproduction Mode Processing Steps

[1821] Step 1:

[1822] The user selects "Personality Reproduction Mode" on the mode selection screen within the app.

[1823] Step 2:

[1824] The user inputs a specific request (for example, "Create a speech manuscript") and presses the "Submit" button.

[1825] Step 3:

[1826] The terminal transmits the user's request to the server.

[1827] Step 4:

[1828] The server retrieves relevant past data and analyzes the user's thought patterns, writing style, and speaking style.

[1829] Step 5:

[1830] The server takes into account the data from the emotion engine and generates appropriate sentences according to the user's emotional state.

[1831] Step 6:

[1832] The server sends the generated text to the terminal.

[1833] Step 7:

[1834] The terminal displays the generated text to the user.

[1835] Example 2

[1836] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1837] The purpose of this invention is to provide a richer user experience by utilizing the user's past data stored in cloud storage. However, conventional technologies simply display data, making it difficult to provide services that take the user's emotional state into account. Furthermore, there is a problem in that the quality of reminiscences and personality reproduction is low because emotions are not recognized.

[1838] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting the user's past data from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means for providing an emotion engine for analyzing emotions from the user's voice and facial expressions, means for generating reminiscences based on the user's emotional state, and means for generating sentences based on the user's writing and speaking style. This makes it possible to provide high-quality reminiscences according to the user's emotional state and improve the accuracy of personality reproduction.

[1839] "Cloud storage" is a storage service on a remote server that stores and manages data over the Internet.

[1840] "User's past data" refers to various data that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[1841] "Means of collection" refers to the functionality for obtaining the required data from authorized cloud storage.

[1842] "Means of analysis" refers to the ability to extract useful information from collected data using machine learning models and algorithms.

[1843] "Means for providing services" refers to the function of generating content such as reminiscences and personality reenactments based on analyzed data and providing it to users.

[1844] "Emotion engine" refers to technology that analyzes a user's voice and facial expressions to identify their current emotional state.

[1845] "Means for generating reminiscences" refers to a function for automatically generating reminiscences for a user based on analyzed past data and the user's current emotional state.

[1846] "Means of personality reproduction" refers to a function that analyzes the user's past data and generates sentences based on the user's writing and speaking style.

[1847] This invention is a system that collects and analyzes users' past data stored in cloud storage to provide services in multiple modes. By incorporating an emotion engine into this system, the quality and user experience of the reminiscence generation mode and personality reproduction mode can be improved. Specific embodiments are described below.

[1848] System configuration

[1849] This system consists of a server that performs a series of processes, such as data collection means, data analysis means, memory generation mode, personality reproduction mode, and emotion engine, a terminal that acts as a user interface, and a series of means for interaction with the user.

[1850] Data collection methods

[1851] The server obtains the user's permission to collect data such as photos, audio, notes, and emails from cloud storage (e.g., a general cloud storage service). In this process, it is important to protect the user's privacy and obtain consent for data collection.

[1852] Data Analysis Methods

[1853] The server uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the collected data. This allows it to accurately analyze photo metadata, audio content, and note entries. The analysis results are used to generate reminiscences and recreate personalities.

[1854] Emotion Engine

[1855] The server is equipped with an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) that analyzes emotions from the user's voice and facial expressions, allowing it to recognize the user's current emotional state and generate an appropriate response.

[1856] Memories generation mode

[1857] The server generates a reminiscence based on the analyzed data. It also takes into account the emotion engine data and provides a tone and topic that matches the user's emotions. For example, if the user is feeling sad, the server generates a reminiscence response in a gentle tone that reflects the user's emotions.

[1858] Examples:

[1859] The user launches the app, logs in, and selects "Memories mode." The emotion engine analyzes the user's facial expressions and voice and recognizes that the user is feeling depressed. The user then asks, "Tell me about this photo." The server analyzes the metadata of the relevant photo from cloud storage and generates a gentle response such as, "This photo was taken in the summer of 2018 in Biei, Hokkaido. It was taken during a family trip, and the lavender fields were so beautiful. I'm sure it will bring back many happy memories filled with smiles." The response is then sent to the device and displayed to the user.

[1860] Personality Reproduction Mode

[1861] The server analyzes the user's thought patterns, writing style, and speaking style, and generates sentences based on this. It uses an emotion engine to generate appropriate sentences according to the user's emotional state. For example, if the user is nervous, it uses expressions that will relax them.

[1862] Examples:

[1863] The user starts the app, logs in, and selects "Personality Mode." The emotion engine recognizes that the user's tone of voice sounds tense. The user then requests, "Create a retirement speech." The server analyzes past emails and notes to learn the user's style. Based on this, it generates a speech with a relaxing tone, such as, "Dear colleagues, thank you for your support over the years. I'm especially grateful to everyone who helped me on this project. I'm truly grateful, but I'm nervous about taking on this new challenge." The speech is then sent to the device and displayed to the user.

[1864] User Interface and Operation

[1865] User Interface:

[1866] Users interact with the system through an app on their device, which includes a login screen, a mode selection screen, a question input screen, and a result display screen. In addition, the app may display real-time emotion analysis results from an emotion engine.

[1867] Operation steps:

[1868] 1. The user launches the app and logs in.

[1869] 2. The user selects "Memories mode" or "Personality reproduction mode" on the mode selection screen.

[1870] 3. The user enters and submits a question or request.

[1871] 4. The server analyzes the data and generates results using the emotion engine.

[1872] 5. The results are displayed on the device and feedback is given to the user.

[1873] This process allows users to share their past memories while feeling the emotions involved, and generate text that reflects their emotional state. This system allows users to effectively utilize past data and enjoy richer life experiences.

[1874] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1875] Step 1:

[1876] The user launches the app and logs in. The user enters their username and password on the device's login screen and clicks the login button. The server checks the user's authentication information against the database, and if authentication is successful, it generates a JWT (JSON Web Token) and sends it to the device. This starts the user's session. The output is the home screen with the user logged in.

[1877] Step 2:

[1878] The user selects a mode. The user selects "Memories Mode" or "Personality Reproduction Mode" on the home screen. The device sends the user's selection to the server. The server records the mode selected by the user and prepares the system to proceed to the next step. The input is an instruction to select a mode, and the output is confirmation of the selected mode.

[1879] Step 3:

[1880] The server obtains consent to collect data. The server displays a dialog to the user requesting permission to access cloud storage. If the user agrees, the server obtains the necessary data access rights from the cloud storage using the OAuth 2.0 protocol. The input is consent information, and the output is the acquisition of access rights. The server collects photos, audio, notes, emails, etc. from the cloud storage service and stores them in a temporary storage area.

[1881] Step 4:

[1882] The server analyzes the collected data. The server analyzes the data using machine learning models (e.g., TensorFlow, PyTorch). Specific operations include extracting metadata from photos, transcribing audio data, and analyzing the content of notes. The input is the collected data, and the output is the analysis results. The analysis results are used in the next step.

[1883] Step 5:

[1884] The server performs emotion analysis using an emotion engine. The server inputs the user's voice and facial expression data into an emotion engine (e.g., Microsoft Azure's Emotion API, IBM Watson's Tone Analyzer) to analyze the user's emotional state. The input is the user's voice and facial expression data, and the output is data on the user's emotional state (e.g., joy, sadness, tension). This allows the user's current emotional state to be quantified.

[1885] Step 6:

[1886] The server generates content based on emotions. Based on the analysis results and the emotion engine results, the server generates sentences for reminiscences or personality reenactment. To do this, a prompt sentence is input to a generative AI model (e.g., GPT-3, BERT), which then generates an appropriate sentence. The input is the prompt sentence and the analysis results, and the output is the generated sentence.

[1887] Specifically, in the reminiscence mode, the system uses the prompt "Tell me your memories of this photo" and adjusts the emotional tone based on the generated text. In the personality reproduction mode, the system uses the prompt "Write a retirement speech" and adjusts the tone of the generated speech, such as making it more relaxed.

[1888] Step 7:

[1889] The server sends the generated content to the terminal. The server sends the generated text to the terminal in JSON format. The terminal parses the received data and displays it on the user interface. The user can check the displayed results and send feedback accordingly. The input is the data of the generated content, and the output is the user's display screen.

[1890] Through the above steps, users can enjoy high-quality emotional reminiscences and personality reproductions using past data.

[1891] (Application example 2)

[1892] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1893] Conventional robot assistant systems in factories have limited capabilities for providing work guidance and employee support, making it difficult to provide appropriate support that takes into account the emotional state of employees. Therefore, there has been a demand for a method that can simultaneously promote efficient work and reduce employee stress.

[1894] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of users from cloud storage, means for analyzing the collected data, means for providing services in multiple modes according to the user's selection, means having an emotion engine that analyzes facial expressions and voice in real time, and means for providing appropriate instructions and support according to the user's emotions. This enables efficient work guidance and support for employees while taking into account their emotional state.

[1895] "Cloud storage" is a technology that provides remote servers for storing data over the Internet.

[1896] "User's past data" refers collectively to information that the user has generated or saved in the past, such as photos, audio, notes, and emails.

[1897] "Means of collection" refers to the processes and tools used to obtain the required data from cloud storage.

[1898] "Means of analysis" are methods and techniques for analyzing collected data and extracting useful information.

[1899] "Multiple modes" refer to different types of services that the system can provide, including, for example, a reminiscence generation mode and a personality reproduction mode.

[1900] The "emotion engine" is a technology that analyzes the user's emotional state from voice and facial expression data.

[1901] "Real-time facial expression and voice analysis" refers to the ability to instantly assess a user's current emotional state.

[1902] "Means for providing instructions and support" refers to mechanisms for providing specific actions or advice to users based on the analysis results.

[1903] The "mode for generating memories" is a function that generates stories that evoke memories based on the user's past data.

[1904] "Personality Reproduction Mode" is a function that imitates the user's personality based on their past actions and documents, and generates responses in real time.

[1905] The "work instruction mode" is a function in which the robot instructs employees on how to do their work in a factory or other environment.

[1906] "Production history analysis mode" is a function that analyzes past data to evaluate employee work performance.

[1907] "Voicing" refers to the act of the robot providing verbal advice or instructions depending on the employee's emotional state.

[1908] This paper describes an embodiment of a system equipped with an emotion engine for factory robots that incorporates a "production history analysis mode" and a "work guidance mode." This system uses a robot assistant in a factory to analyze the emotional state of employees in real time and provide appropriate guidance and support. The main hardware and software configurations and their processing flow are described.

[1909] Hardware Configuration

[1910] 1. Factory robots

[1911] Supports all operations required for operation.

[1912] 2. Camera and microphone

[1913] Capture employees' facial expressions and voices.

[1914] 3. Cloud Storage

[1915] A remote server for storing employee historical data.

[1916] Software Configuration

[1917] 1. Machine Learning Model

[1918] TensorFlow and Google Cloud AI are used for data analysis.

[1919] 2. Emotion Engine

[1920] Affectiva and Microsoft Azure Emotion API are used as technologies to analyze emotions from voice and facial expressions.

[1921] 3. Robot Control Software

[1922] Control the robot's movements using ROS (Robot Operating System).

[1923] 4. User Interface

[1924] Users interact with the system via factory tablets and displays.

[1925] Program processing flow

[1926] 1. Data Collection

[1927] The server collects users' past data from cloud storage, including photos, audio, notes, and emails.

[1928] Cameras and microphones are used to collect employees' facial expressions and voices in real time.

[1929] 2. Data Analysis

[1930] The collected data is analyzed using machine learning models on a cloud server.

[1931] The emotion engine analyzes the employee's emotional state in real time, determining whether they are tired or tense.

[1932] 3. Memories generation mode and personality reproduction mode

[1933] The server utilizes historical data collected and real-time emotional data to generate appropriate topics and tones depending on the user's emotional state.

[1934] For example, if the system detects that a user is depressed, and asks, "Tell me about this photo," the memory-generating mode will respond in a gentle tone with, "This photo was taken in the summer of 2018 of the lavender fields in Biei, Hokkaido."

[1935] 4. Production history analysis mode

[1936] The server analyzes past data and real-time emotional data to evaluate the employee's work performance. For example, if the server determines that the employee is tired, the robot will advise them to "pause your work and take a short break."

[1937] 5. Work Instruction Mode

[1938] The server will guide employees through the appropriate work process based on their emotional state, explaining to a nervous employee in a relaxing voice, "Let's simplify the next step."

[1939] Specific examples

[1940] Prompt Sentence Examples

[1941] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[1942] This will realize a system that provides efficient work guidance and support to employees while taking into account their emotional state. Specifically, if an employee is tired, the system will pause work to encourage them to rest, and it will also simplify work processes to reduce stress.

[1943] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1944] Step 1:

[1945] The server collects users' past data from cloud storage. The input requires the cloud storage API key and user authentication information, and the output includes photos, audio, notes, emails, etc. This data is stored digitally and sent to the next analysis step.

[1946] Step 2:

[1947] The server collects facial expression and voice data of employees in real time using a camera and microphone. Real-time data from the camera and microphone is required as input, and facial expression video data and voice data are obtained as output. This data is sent to the emotion engine to analyze the employee's current emotional state.

[1948] Step 3:

[1949] The server analyzes the collected data. It uses as input the previously collected data (output of step 1) and real-time emotion data (output of step 2). This allows for metadata analysis of photos, keyword detection in speech, and text analysis of notes and emails. The output provides key information and insights that are integrated with the emotion engine's analysis.

[1950] Step 4:

[1951] The server uses an emotion engine to analyze the employee's emotional state. It requires real-time collected facial expression and voice data as input, and outputs an emotional status, such as "tired," "tense," or "relaxed." This status is used to generate appropriate instructions for the user in the next step.

[1952] Step 5:

[1953] The server executes the production history analysis mode based on the analysis results. As input, it requires the past data analysis results (output of step 3) and the real-time emotional status (output of step 4). The server evaluates the employee's work performance and generates appropriate instructions and support according to the employee's emotional state. As output, it obtains a specific action plan.

[1954] One specific action is "suggesting rest." For example, if the server determines that an employee is tired, it will generate instructions such as "We recommend that you temporarily stop working and take a short break," and communicate this to the employee via the robot's voice output system.

[1955] Step 6:

[1956] The server executes the work instruction mode. It uses the analysis results (output of Step 5) and emotional status data as input. The server instructs the work procedure according to the employee's emotional state. For example, if the employee is nervous, the server generates instructions encouraging relaxation, such as "Let's perform the next step using a simplified procedure," and the robot conveys this aloud.

[1957] Step 7:

[1958] The server re-stores the collected and analyzed data in cloud storage. The latest analysis results and generated instruction data are required as input, and the updated employee data history is stored in cloud storage as output. This will further increase the accuracy of future analysis and instruction generation.

[1959] Example prompt

[1960] "If you notice that an employee is tired, consider providing instructions to simplify their work and offer suggestions to encourage them to take a break."

[1961] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1962] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1963] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1964] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1965] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1966] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1967] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1968] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1969] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1970] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1971] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1972] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1973] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1974] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1975] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1976] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1977] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1978] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1979] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1980] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1981] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1982] The following is further disclosed regarding the above embodiment.

[1983] (Claim 1)

[1984] A means for collecting user's past data from cloud storage;

[1985] a means for analyzing the collected data;

[1986] means for providing services in multiple modes according to user selection;

[1987] A system including:

[1988] (Claim 2)

[1989] 2. The system of claim 1, wherein the plurality of modes include a mode for generating reminiscences and a mode for recreating the user's personality.

[1990] (Claim 3)

[1991] 3. The system according to claim 2, wherein the mode for generating a reminiscence includes means for generating a reminiscence based on the user's past photographs, voice, memos and emails.

[1992] (Claim 4)

[1993] 3. The system according to claim 2, wherein the personality-reproducing mode includes means for analyzing a user's thought patterns, writing style, and speaking style and generating sentences based thereon.

[1994] (Claim 5)

[1995] 10. The system of claim 1, wherein the data collection means includes means for obtaining user authorization.

[1996] (Claim 6)

[1997] 2. The system according to claim 1, wherein the service providing means includes means for generating answers in real time in response to questions from users.

[1998] (Claim 7)

[1999] 10. The system of claim 1, wherein the data analysis means comprises means for analyzing the data using a machine learning model.

[2000] "Example 1"

[2001] (Claim 1)

[2002] A means for collecting data from cloud storage based on user access permissions;

[2003] A means of analyzing the collected data and extracting metadata and content;

[2004] means for generating and providing information from the analyzed data in response to a user's request;

[2005] a means for using a natural language generation model to generate a reminiscence;

[2006] A means of learning the user's writing and speaking style and generating text;

[2007] A system including:

[2008] (Claim 2)

[2009] 2. The system of claim 1, wherein the plurality of modes include a mode for generating reminiscences and a mode for recreating a user's writing or speaking style.

[2010] (Claim 3)

[2011] 3. The system according to claim 2, wherein the mode for generating a reminiscence includes means for generating a reminiscence based on image data, voice data, text data and communication data of the user's past.

[2012] "Application Example 1"

[2013] (Claim 1)

[2014] A means for collecting user's past data from cloud storage;

[2015] a means for analyzing the collected data;

[2016] means for providing services in multiple modes according to user selection;

[2017] a means for making personalized recommendations based on data analysis;

[2018] A system including:

[2019] (Claim 2)

[2020] 10. The system of claim 1, wherein the plurality of modes include a mode for generating reminiscences, a mode for recreating a user's personality, and a mode for providing personalized recommendations.

[2021] (Claim 3)

[2022] 3. The system according to claim 1, wherein the mode for generating a reminiscence includes means for generating a reminiscence based on the user's past photographs, voice, memos and emails.

[2023] "Example 2: Combining Emotion Engines"

[2024] (Claim 1)

[2025] A means for collecting user's past data from cloud storage;

[2026] a means for analyzing the collected data;

[2027] means for providing services in multiple modes according to user selection;

[2028] A means for providing an emotion engine for analyzing emotions from the user's voice and facial expressions;

[2029] means for generating a reminiscence based on an emotional state;

[2030] A means for generating sentences based on the user's writing and speaking style;

[2031] A system including:

[2032] (Claim 2)

[2033] 2. The system of claim 1, wherein the plurality of modes include a mode for generating reminiscences and a mode for recreating the user's personality.

[2034] (Claim 3)

[2035] 2. The system according to claim 1, wherein the mode for generating a reminiscence includes means for generating a reminiscence based on the user's past photographs, voice, memos and emails.

[2036] "Application example 2 when combining emotion engines"

[2037] (Claim 1)

[2038] A means for collecting user's past data from cloud storage;

[2039] a means for analyzing the collected data;

[2040] means for providing services in multiple modes according to user selection;

[2041] A means with an emotion engine that analyzes facial expressions and voice in real time;

[2042] A means of providing appropriate instructions and support according to the user's emotions, and

[2043] A system including:

[2044] (Claim 2)

[2045] 10. The system of claim 1, wherein the plurality of modes include a mode for generating reminiscences, a mode for recreating the user's personality, and a mode for providing task guidance.

[2046] (Claim 3)

[2047] 10. The system of claim 1, wherein the production history analysis mode includes means for evaluating work performance based on historical data and real-time sentiment analysis and generating appropriate instructions. [Explanation of symbols]

[2048] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting user's past data from cloud storage; a means for analyzing the collected data; means for providing services in multiple modes according to user selection; A system including:

2. 2. The system of claim 1, wherein the plurality of modes include a mode for generating reminiscences and a mode for recreating the user's personality.

3. 3. The system according to claim 2, wherein the mode for generating a reminiscence includes means for generating a reminiscence based on the user's past photographs, voice, memos and emails.

4. 3. The system according to claim 2, wherein said personality-reproducing mode includes means for analyzing a user's thought patterns, writing style, and speaking style, and generating sentences based thereon.

5. The system of claim 1 , wherein the data collection means includes means for obtaining user authorization.

6. 2. The system according to claim 1, wherein said service providing means includes means for generating answers in real time in response to questions from users.

7. The system of claim 1 , wherein the data analysis means comprises means for analyzing the data using a machine learning model.

Images