System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting behavior data of a user; means for analyzing the collected behavior data; means for providing optimal information to the user based on an analysis result; means for collecting an action of the user based on the provided information; and means for analyzing the collected action data and improving accuracy of next information provision.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's aging society, the elderly over 65, who tend to be particularly vulnerable to information, face the problem of digital divides. They have difficulty obtaining the information they need from the internet or social media, and in many cases, they are unable to find the information they need or are presented with a lot of unnecessary information. To solve this problem, a system is needed that automatically provides the most appropriate information based on daily behavioral data, without requiring users to search for the information themselves. [Means for solving the problem]

[0005] The present invention provides a system including: means for collecting user behavioral data; means for analyzing the collected behavioral data; means for providing optimal information to the user based on the analysis results; means for collecting user actions based on the provided information; and means for analyzing the collected action data to improve the accuracy of the next information provision. The system also includes means for collecting user location information, analyzing the user's daily behavioral patterns based on the location information, and generating and providing reminders and notifications based on the behavioral patterns. The system also includes means for collecting voice data and converting it into text; and means for analyzing the converted voice data to infer the user's intentions and necessary information, and providing the user with recommended information based on the inferred necessary information. This system enables elderly people to obtain necessary information in a timely and accurate manner without having to search for the information themselves.

[0006] "User" refers to an individual who uses the system to provide daily behavioral data and receive analysis results thereof.

[0007] "Behavioral data" refers to digital data that includes information such as location information, audio data, and photo data of a user's daily life.

[0008] "Collection means" refers to devices and sensors, such as smartphones or tablets, used to acquire user behavioral data.

[0009] "Analysis Means" refers to the algorithms and software used to process and analyze collected behavioral data and infer user behavior patterns and needs.

[0010] "Information provision means" refers to a system component that delivers optimal information to users based on the analysis results, and includes notification functions and alarms.

[0011] "Action collection means" refers to a system component for recollecting actions that a user may take in response to provided information.

[0012] "Action Data" refers to data regarding actions taken by a user based on information provided by the system.

[0013] "Location information" refers to digital data about a user's geographic location obtained from their smartphone or tablet.

[0014] "Notification" refers to a message or alarm that the system uses to provide timely and important information to the user.

[0015] "Voice data" refers to data recorded in digital format of what a user says to the system.

[0016] "Text-to-speech" refers to the process of analyzing collected voice data and converting it into text form.

[0017] "Inference methods" refer to algorithms or software that predict a user's intentions and required information based on textual voice data and other behavioral data.

[0018] "Reminder" refers to a system-provided advance notice or alarm that prompts a user to take a specific action.

[0019] "Recommended information" refers to useful information that the system provides to the user based on the user's behavioral patterns and intentions. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention relates to a system for collecting user behavior data, analyzing the data, and providing optimal information to the user. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[0042] Overall system configuration

[0043] The main components of the system are:

[0044] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[0045] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[0046] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[0047] 4. Data analysis means: Analysis algorithms and software installed on the server.

[0048] 5. Information provision means: Notification functions and alarms installed on the device.

[0049] 6. Action collection means: Software that allows the terminal to recollect user actions.

[0050] Program processing

[0051] Below, each element of the system and its specific operation will be explained.

[0052] 1. Data Collection:

[0053] Device: The user's device collects real-time location information using GPS, records audio data using a microphone, and captures photos using a camera. These data are temporarily stored on the device. For example, when a user goes for a walk every day, the route and time of the walk are recorded.

[0054] 2. Data transmission:

[0055] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[0056] 3. Data Analysis:

[0057] Server: The server analyzes the received data. It plots GPS data on a map, converts voice data into text, and tags photos with AI. For example, it analyzes a daily activity log to determine when, where, and what the user is doing.

[0058] 4. Information generation:

[0059] Server: Based on the analysis results, messages are generated to provide the user with the necessary information in a timely manner. For example, if a user misses a regular dose of medication, a notification is generated to remind them to take it.

[0060] 5. Information provision:

[0061] Terminal: The information sent from the server is displayed on the terminal. Notifications are displayed and alarms sound if necessary. The user can act based on this information. For example, taking medicine after seeing the notification.

[0062] 6. Action Collection:

[0063] Device: The device records the user's actions (if they take their medicine based on the notification, that action) again. When the user presses the "take medicine" button, the data is sent to the server again.

[0064] 7. Feedback Analysis:

[0065] Server: Analyzes the received feedback data and improves the accuracy of the next notification. For example, it learns medication habits and optimizes the next notification time.

[0066] Specific examples

[0067] Example 1: Mr. A (70 years old) sets his schedule to take a walk at 7:00 a.m. every day and take his medicine at 8:00 a.m. In this case, the following series of processes will be performed.

[0068] Data collection: Mr. A's device collects GPS data and takes voice notes at the start of his walk at 7:00 AM.

[0069] Data transmission: After the walk ends, the data is transmitted to the server just before 8:00 AM.

[0070] Data analysis: The server analyzes the GPS data to determine the end time of Mr. A's walk and analyzes the time of taking his medicine from the voice memo.

[0071] Information generation: The server generates a notification message saying "It's time to take your medicine" just before 8:00 AM.

[0072] Information provided: The notification appears on the device and Person A checks it.

[0073] Action collection: Person A takes medicine and presses the "take" button on the device.

[0074] Feedback analysis: The server receives dose timing data and improves the accuracy of the next notification.

[0075] In this way, the system of the present invention can provide timely information to elderly people without them having to search for the information themselves, thereby improving their quality of life.

[0076] The processing flow will be explained below.

[0077] Step 1:

[0078] Data collection

[0079] Device: Your device periodically collects location information using GPS, records audio data with its built-in microphone, and temporarily stores photos captured by its camera.

[0080] User: Data is collected from users as they go about their daily lives without them having to take any special action.

[0081] Step 2:

[0082] Data transmission

[0083] Device: When a certain amount of collected data is reached or a certain time has passed, the device sends this data in batches to the server.

[0084] Step 3:

[0085] Data reception and storage

[0086] Server: Receives data sent from the device and stores it in a database, including location information, audio data, and photo data.

[0087] Step 4:

[0088] Converting audio data to text

[0089] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the speech "I ate breakfast" is converted into text "I ate breakfast."

[0090] Step 5:

[0091] Data analysis

[0092] Server: Plots the received location information on a map and analyzes the user's daily movements. It also analyzes the converted voice data to identify activities and priorities.

[0093] User: The system automatically analyzes the data without any special awareness.

[0094] Step 6:

[0095] information generation

[0096] Server: Generates necessary information for the user based on the analysis results. For example, if the pattern of "taking a walk at 7:00 AM and then taking medicine at 8:00 AM" is analyzed, a notification urging the user to take their medicine is generated.

[0097] Step 7:

[0098] Information transmission

[0099] Server: Sends the generated information to the user's device. This message contains notifications and reminders.

[0100] Step 8:

[0101] Receiving information and displaying notifications

[0102] Device: The received information is displayed on the device screen and, if necessary, the user is notified using push notifications or alarm functions.

[0103] User: Check the notification on the device and take the necessary action (e.g., take medicine).

[0104] Step 9:

[0105] Action Collection

[0106] On the device: If the user takes action based on the notification, provide a feedback form or button to record that action. When the user presses the "take a dose" button, that information is collected.

[0107] Step 10:

[0108] Sending feedback data

[0109] Device: Send the collected feedback data back to the server.

[0110] User: Follow the instructions on the device to enter feedback information.

[0111] Step 11:

[0112] Analysis of feedback data

[0113] Server: Analyzes the received feedback data and optimizes the timing and content of the next information provision. For example, data is added to fine-tune the time at which the user takes their medicine.

[0114] By repeating these steps, the system can continuously learn the user's behavioral patterns and provide the most appropriate information in a timely manner.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] The problem to be solved by this invention is to effectively collect and analyze user behavior data and provide optimal information to the user based on the analysis results. Furthermore, there is a need to collect user actions based on the provided information and analyze this as feedback to improve the accuracy of the next information provided. There is also a need to provide a system that can provide more accurate user support by integrating various data sources, such as location information and voice data.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes means for collecting user behavioral data, means for saving the collected behavioral data and transmitting it under specific conditions, means for analyzing the transmitted data, means for generating information optimized for the user based on the analysis results, means for providing the generated information to the user, means for collecting user actions based on the provided information, and means for analyzing the collected action data and improving the accuracy of the next information provision. This makes it possible to effectively collect and analyze user behavioral data and provide individually optimized information. Furthermore, it is possible to provide feedback on the user's reactions and improve the accuracy of the next information provision based on that data.

[0120] "Behavioral data" is information related to the user's actions and behavior in daily life, and includes location information, voice data, photograph data, and the like.

[0121] "Location Information" means a user's geographic coordinate data obtained using GPS or other location-determining technology.

[0122] "Collection means" refers to the technologies and devices used to collect user behavioral data, and primarily refers to sensors such as GPS, microphones, and cameras.

[0123] "Analysis means" refers to the techniques and algorithms used to analyze collected data, and are used to extract data features and recognize patterns.

[0124] "Information generation means" refers to techniques and scripts for generating information to be provided to users based on the analysis results.

[0125] "Providing means" refers to the technology or device used to transmit the generated information to the user, such as a notification function or alarm.

[0126] "Action Data" is a record of when a user acts based on the information provided.

[0127] "Feedback analysis means" refers to technologies and algorithms that analyze collected action data and improve the accuracy of the next information provided.

[0128] MODE FOR CARRYING OUT THE INVENTION

[0129] The present invention relates to a system that collects user behavior data, analyzes the data, and provides optimal information to the user. The system is composed of the following elements:

[0130] Overall system configuration

[0131] The main components of the system are:

[0132] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[0133] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[0134] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[0135] 4. Data analysis means: Analysis algorithms and software installed on the server.

[0136] 5. Information provision means: Notification functions and alarms installed on the device.

[0137] 6. Action collection means: Software that allows the terminal to recollect user actions.

[0138] Data collection

[0139] Device: The smartphone or tablet that a user carries around collects behavioral data using built-in sensors such as GPS, microphone, and camera. Location information is acquired in real time by the GPS sensor, audio data is recorded through the microphone, and photo data is taken with the camera. This data is temporarily stored on the device.

[0140] Data transmission

[0141] Device: The collected data is sent to the server after a certain period of time has passed or when a certain amount of data has been collected. Wi-Fi or mobile data is used as the communication method.

[0142] Data analysis

[0143] Server: The server analyzes the data it receives. GPS data is plotted on a map, voice data is converted to text, and photo data is tagged with AI to identify user behavioral patterns and interests.

[0144] information generation

[0145] Server: Generates necessary information for the user based on the analysis results. For example, if a user misses a medication time, it generates a reminder message.

[0146] Information provision

[0147] Device: Provides the user with the information sent from the server. The notification is displayed on the device screen and, if necessary, an audio alarm is set. The user can then take action.

[0148] Action Collection

[0149] Device: After providing the information, the device collects the user's actual actions. For example, after a notification to take medicine is displayed, pressing the "take medicine" button records the action.

[0150] Feedback Analysis

[0151] Server: Analyzes user action data and optimizes the timing and content of the next information provided, thereby providing the most appropriate information for the user.

[0152] Specific examples

[0153] Example 1:

[0154] Mr. A (70 years old) schedules his daily walk at 7:00 AM and his medication at 8:00 AM. The following process takes place:

[0155] Data collection: When Mr. A starts his walk at 7:00 AM, his device acquires GPS data and also records audio data.

[0156] Data transmission: After the walk is completed, the collected data is sent to the server.

[0157] Data analysis: The server analyzes the walking route and time from GPS data, and confirms the time to take medicine from voice data.

[0158] Information generation: The server generates a notification just before 8:00 AM saying "It's time to take your medicine."

[0159] Information provided: A notification appears on the device and Person A checks it.

[0160] Action collection: After taking the medicine, Person A presses the "take button."

[0161] Feedback analysis: The server optimizes the next notification time based on the action data.

[0162] In this way, the system of the invention can provide users with timely and relevant information to improve their quality of life.

[0163] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0164] System program processing flow

[0165] Step 1:

[0166] Data collection

[0167] Device: The user's device uses the built-in GPS to obtain real-time location information, uses a microphone to record ambient sounds and voice memos, and also uses a camera to take photos at regular intervals.

[0168] What it does: The device's GPS sensor periodically logs the current location, the microphone sensor periodically records audio, and the camera also periodically takes pictures and saves them.

[0169] Input: User's real-time location, ambient sounds, and photos.

[0170] Output: Location data, audio data, and photo data temporarily stored on the device.

[0171] Step 2:

[0172] Data transmission

[0173] Device: After a certain amount of time has passed or a certain amount of collected data has been collected, the device will send the data to the server via Wi-Fi or mobile data.

[0174] Specific operation: The device uses an internal timer and data volume monitoring function to connect to the server and upload data when conditions are met.

[0175] Input: Location data, audio data, and photo data stored on the device.

[0176] Output: Location data, audio data, and photo data sent to the server.

[0177] Step 3:

[0178] Data analysis

[0179] Server: The server analyzes the data it receives: GPS data is plotted on a map, voice data is converted to text, and photo data is tagged using AI.

[0180] What it does: The server uses a map API to display location data on a map, uses voice recognition software to convert voice data to text, and uses image analysis algorithms to tag photos.

[0181] Input: Location data, audio data, and photo data sent to the server.

[0182] Output: Parsed map data, text data, tagged image data.

[0183] Step 4:

[0184] information generation

[0185] Server: Based on the analysis results, the server generates information to provide to the user, for example, a reminder if the user is about to miss a dose of medication.

[0186] What happens: The server evaluates certain conditions based on the parsed data and executes scripts that automatically generate notification messages if those conditions are met.

[0187] Input: Parsed map data, text data, tagged image data.

[0188] Output: A notification message to the user.

[0189] Step 5:

[0190] Information provision

[0191] Device: To provide the user with the information sent from the server, the device will display a notification on the screen and sound an alarm if necessary.

[0192] Specific behavior: The device's notification system receives the notification message and displays it to the user as a popup. It also invokes the function to sound an audio alarm at the specified time.

[0193] Input: The notification message sent by the server.

[0194] Output: Notifications displayed on the device screen, alarms sounding.

[0195] Step 6:

[0196] Action Collection

[0197] Device: If the user acts on the notification, the action is collected again. For example, if the user presses the "take a pill" button, the action data is recorded.

[0198] What it does: The device interface detects user actions, logs the information, and uploads it to a server.

[0199] Input: User action (e.g., pressing the dose button).

[0200] Output: Action data sent from the device to the server.

[0201] Step 7:

[0202] Feedback Analysis

[0203] Server: The server analyzes the action data it receives and optimizes the timing and content of the next information provision.

[0204] What it does: The server's analysis algorithm evaluates your action data and automatically adjusts the timing and content of your next notification based on the results.

[0205] Input: Action data sent from the device.

[0206] Output: Optimized next notification settings.

[0207] (Application example 1)

[0208] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] Conventional information provision systems were able to collect and analyze user behavior data, but were insufficient for providing customized information in real time. Furthermore, they lacked the technology to simultaneously analyze user behavior patterns and interests and make highly accurate recommendations. Furthermore, they were unable to flexibly generate information using generative AI models, making it difficult to provide information tailored to user needs.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0211] In this invention, the server includes means for collecting user behavior data, means for analyzing the collected behavior data, means for providing optimal information to the user based on the analysis results, means for collecting user actions based on the provided information, means for analyzing the collected action data and improving the accuracy of the next information provided, means for simultaneously collecting and analyzing audio data, video data, and location information, and means for providing customized product information to the user in real time based on the generated analysis results. This makes it possible to analyze user behavior with high accuracy and provide optimal information in real time.

[0212] "Behavioral data" refers to various types of information related to the user's daily activities, such as location information, audio data, and video data.

[0213] "Analysis" refers to the processing of collected data to identify user behavior patterns and interests.

[0214] "Optimal information" refers to useful information that is individually customized based on the user's behavior and interests.

[0215] "Action data" refers to data regarding actions and operations performed by a user based on the information provided.

[0216] A "generative AI model" is an artificial intelligence model used to generate optimal information for each user in real time based on collected data.

[0217] A "prompt" is text containing instructions or suggestions that a generative AI model generates to provide information.

[0218] "Customized product information" is information about products selected according to the user's behavioral patterns and interests.

[0219] "Providing in real time" refers to providing information instantly in conjunction with the user's current actions.

[0220] This invention is a system that collects user behavior data, analyzes the data, and provides optimal information to the user in real time. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[0221] Overall system configuration

[0222] The main components of the system are as follows:

[0223] 1. Device: Smart glasses or smartphones that users use on a daily basis, which collect behavioral data (location information, video data, audio data).

[0224] 2. Server: This is the central system that analyzes behavioral data and provides optimal information to users based on the analysis results.

[0225] 3. Data collection means: Sensors such as GPS, camera, and microphone equipped on the device.

[0226] 4. Data analysis means: Analysis algorithms and generative AI models installed on the server.

[0227] 5. Information provision means: Notification functions and alarms installed on the device.

[0228] 6. Action collection means: Software that allows the terminal to collect user actions.

[0229] 7. Generative AI model: An artificial intelligence model that generates prompts in real time based on user behavior data.

[0230] Program processing

[0231] 1. Data Collection:

[0232] Device: The user's smart glasses or smartphone collects real-time location information using GPS, captures video data with a camera, and records audio data with a microphone. These data are temporarily stored on the device.

[0233] 2. Data transmission:

[0234] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[0235] 3. Data Analysis:

[0236] Server: The server analyzes the received data. It plots GPS data on a map, analyzes video data using an image processing library (e.g., OpenCV), and converts audio data to text using a machine learning library (e.g., TensorFlow).

[0237] 4. Information generation:

[0238] Server: Based on the analysis results, the server uses a generative AI model to generate information to be provided to the user in the form of prompt sentences. For example, it generates prompt sentences that display information about products that the user is interested in in real time.

[0239] 5. Information provision:

[0240] Terminal: The prompt sent from the server is displayed on the terminal, notifications are displayed and alarms are sounded if necessary, and the user can act accordingly.

[0241] 6. Action Collection:

[0242] Device: The device again records the user's actions (e.g., purchasing the presented product, reacting to the displayed information).

[0243] 7. Feedback Analysis:

[0244] Server: Analyzes the received feedback data and improves the accuracy of the next information provided.

[0245] Specific examples

[0246] Example 1:

[0247] As users walk around the virtual store, their behavioral data (location, video, and audio data) is collected in real time. The server analyzes this data to identify the user's interests. For example, if a user spends a long time looking at a particular group of products, the generative AI model generates and provides detailed information about those products and suggestions for related products. In this case, the generative AI model generates prompts such as "You might be interested in this product" and displays them on the smart glasses.

[0248] Example 2:

[0249] When a user is walking through a food section, the server predicts the ingredients the user will need based on their behavioral data and past purchase history, and displays a prompt such as "We recommend this ingredient."

[0250] Example prompt sentence:

[0251] "This product might interest you"

[0252] "I recommend this ingredient."

[0253] "This section has the product you're looking for"

[0254] In this way, the system of the present invention individually optimizes the shopping experience in the virtual store, making it more convenient for the user.

[0255] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0256] Step 1:

[0257] Data collection

[0258] The device starts collecting the user's location information in real time using GPS, where the input is the user's real-time location information, which is obtained from the GPS sensor.

[0259] At the same time, the camera is used to capture video data and the microphone is used to record audio data. These data are temporarily stored in the device. The input is the video data from the camera and the audio data from the microphone, and these are collected.

[0260] Step 2:

[0261] Data transmission

[0262] The terminal transmits the data to the server after a certain time has passed or when the amount of collected data reaches a certain amount. The input is the data collected in step 1, and the output is the data sent to the server.

[0263] Step 3:

[0264] Data analysis

[0265] The server analyzes the received data. First, it plots the GPS data on a map to visualize the user's location fluctuations. The input is location data, and the output is a visualization of the location.

[0266] Next, the video data is analyzed using an image processing library (e.g., OpenCV) to identify the scene and the user's interests. The input is the video data, and the output is the analyzed scene information.

[0267] The audio data is converted to text using a machine learning library (e.g., TensorFlow). The input is the audio data, and the output is the converted audio data.

[0268] Step 4:

[0269] information generation

[0270] The server generates prompt sentences using a generative AI model based on the analysis results from step 3. For example, if the user is interested in a specific product category, it generates prompt sentences for related products. The input is the analysis results, and the output is the generated prompt sentences.

[0271] Step 5:

[0272] Information provision

[0273] The terminal displays the prompt sentence received from the server in real time, for example, by displaying a notification on the display of smart glasses. The input is the generated prompt sentence, and the output is the notification displayed to the user.

[0274] Step 6:

[0275] Action Collection

[0276] The terminal again records the user's actions (e.g., purchasing a suggested product or reacting to a notification). The input is the user's action, and the output is the recorded action data.

[0277] Step 7:

[0278] Feedback Analysis

[0279] The server analyzes the action data and improves the accuracy of the next information provided. For example, if a particular prompt phrase is likely to lead to a user action, it learns that pattern. The input is the recorded action data, and the output is a new pattern that will be reflected in improving the accuracy of the next information provided.

[0280] In this way, a system can be realized that analyzes user behavior in detail and provides optimal information in real time.

[0281] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0282] This invention combines an emotion engine with a system that collects and analyzes user behavioral data to provide optimal information to users. The system aims to improve the quality of users' lives by collecting and analyzing user behavioral and emotional data and providing appropriate information based on this data.

[0283] Overall system configuration

[0284] The main components of the system are:

[0285] 1. Device: A smartphone or tablet used by a user in their daily life, used to collect behavioral and emotional data.

[0286] 2. Server: A central system that analyzes the collected data and provides information based on the results.

[0287] 3. Data collection methods: GPS, microphone, camera and emotion recognition sensors on the device.

[0288] 4. Data analysis means: Analysis algorithms and software in the server.

[0289] 5. Information provision means: Notification functions and alarms installed on the device.

[0290] 6. Action collection means: terminal applications and interfaces.

[0291] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[0292] Program processing

[0293] The components of the system and their specific operations are described below.

[0294] 1. Data Collection:

[0295] Device: GPS is used to collect location information, a microphone is used to record audio data, and a camera is used to capture facial expressions and photographs. In addition, an emotion engine recognizes the user's emotions from audio and camera data. For example, when a user is taking a walk, location information is collected along with the emotions (relaxed, stressed, etc.) at the time.

[0296] Users: Just use a smartphone or tablet as you normally would, with no special operations required.

[0297] 2. Data transmission:

[0298] Terminal: Periodically transmits collected behavioral and emotional data to the server.

[0299] User: Continues with daily life without performing any special operations.

[0300] 3. Data Receipt and Storage:

[0301] Server: Receives data sent from the device and stores it in a database. Collected data includes location information, voice data, photo data, and emotion data.

[0302] 4. Speech and emotion data transcription and recognition:

[0303] Server: Converts the voice data into text using a speech recognition algorithm and recognizes the user's emotions using an emotion engine. For example, emotion tags are assigned such as "happy," "sad," and "stressed."

[0304] 5. Data Analysis:

[0305] Server: Plots location data on a map and analyzes user behavior patterns. At the same time, converts voice data into text and analyzes emotion data to identify specific activity and emotional trends. For example, it analyzes whether a user feels stressed when in a specific location.

[0306] 6. Information generation:

[0307] Server: Generates appropriate information for the user based on the analysis results, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[0308] 7. Information Transmission:

[0309] Server: Sends server-generated information to the user's device, including notifications and reminders.

[0310] 8. Information Receipt and Notification Display:

[0311] Device: Displays the received information to the user and provides notifications or alarms as needed. For example, the user might see a message saying "Relax now, keep it up."

[0312] 9. Action Collection:

[0313] On the device: If the user takes action based on the notification, collect that action again. For example, record the action taken based on the reminder (e.g., take medicine, take a deep breath).

[0314] 10. Feedback Data Submission:

[0315] Device: Sends collected feedback data to the server, including data based on user behavior.

[0316] 11. Feedback data analysis and model updating:

[0317] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, it identifies the best time to relax from the feedback data and suggests relaxing music for the next time.

[0318] Specific examples

[0319] Example 1: Mr. B (65 years old) uses it daily:

[0320] Data collection and transmission: The emotion engine recognizes that Mr. B takes a walk every day at 1 p.m. and feels relaxed during that time. The device transmits this data to the server.

[0321] Data analysis: The server analyzes the location information and recognizes that you are walking in a specific park at 1 PM. The emotion engine detects your state of relaxation.

[0322] Information generation and provision: The server generates information about relaxation (e.g., a relaxation guide) and sends it to Mr. B's terminal.

[0323] Information presentation and action: Person B checks the information and takes action according to the relaxation guide.

[0324] Feedback collection and transmission: The actions taken by User B following the guide are recorded and the data is sent to the server.

[0325] Feedback analysis: The server analyzes this feedback and uses it for next time.

[0326] In this way, the system of the present invention can combine the user's behavioral data and emotional data to provide optimal information and improve the quality of life.

[0327] The processing flow will be explained below.

[0328] Step 1:

[0329] Data collection

[0330] Device: To record the user's daily activities, the device uses a GPS sensor to collect location information, a microphone to record voice data, and a camera to capture photos and facial expression data. In addition, an emotion engine recognizes the user's emotions from voice and facial expressions in real time.

[0331] User: The person uses a smartphone or tablet on a daily basis and data is collected without any special action.

[0332] Step 2:

[0333] Data transmission

[0334] Terminal: The collected location information, voice data, photo data, and emotion data are sent to the server at regular intervals or when a certain amount of data is reached.

[0335] Step 3:

[0336] Data reception and storage

[0337] Server: Receives data sent from the device and stores it in a database, including location information, audio data, photo data, and emotion data.

[0338] Step 4:

[0339] Converting audio data to text

[0340] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the voice data "My walk this morning was pleasant" is converted into text.

[0341] Step 5:

[0342] emotion recognition

[0343] Server: Uses an emotion engine to recognize the user's emotions from voice data and photo data (mainly facial expressions). For example, it assigns emotion tags such as "relaxed," "stressed," and "happy" based on voice tone and facial expressions.

[0344] Step 6:

[0345] Data analysis

[0346] Server: Plots the collected location information on a map and analyzes the user's behavioral patterns. At the same time, it analyzes the converted voice data and recognized emotion data to understand trends in specific activities and emotions. For example, it analyzes whether the user always relaxes in a specific place (a park).

[0347] Step 7:

[0348] information generation

[0349] Server: Generates appropriate information for the user based on all the analysis results. For example, if the user is in the park at 1pm every day and the data shows that they are relaxing, it generates a message that provides them with tips on how to relax.

[0350] Step 8:

[0351] Information transmission

[0352] Server: Generates information messages and notifications and sends them to the user's device. The notification contents include reminders and emotional advice.

[0353] Step 9:

[0354] Receiving information and displaying notifications

[0355] Device: The device displays received information and notifications on the device screen and alerts the user with sound or vibration as needed. For example, a notification saying "Take deep breaths for relaxation" may appear on the screen.

[0356] Step 10:

[0357] Action Collection

[0358] Terminal: Collects feedback data based on user behavior. When a user follows a relaxation guide, their behavior is recorded. For example, data is collected by pressing a button that says "I took a deep breath."

[0359] Step 11:

[0360] Sending feedback data

[0361] Device: Sends collected feedback data to the server.

[0362] User: Follows instructions to take action and provides feedback information as a result.

[0363] Step 12:

[0364] Analysis of feedback data

[0365] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, if it turns out that a user finds it easier to relax at a certain time of day, it will suggest relaxation methods that are appropriate for that time of day from the next time.

[0366] In this way, this system can improve the quality of life of users by comprehensively analyzing their behavioral and emotional data and providing optimal information tailored to their individual needs.

[0367] Example 2

[0368] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0369] Conventional information provision systems provide information based solely on user behavior data, making it impossible to perform detailed analysis of the user's emotional state or behavioral patterns, resulting in the problem that the information provided does not adequately meet the user's needs.In addition, there is a lack of a mechanism for integrating and analyzing emotional data in addition to voice data and location information, making it difficult to effectively provide the most appropriate information to the user.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user behavioral data and emotional data, means for analyzing the collected data, and means for providing optimal information to the user based on the analysis results. This makes it possible to combine and analyze the user's behavioral data and emotional data, and to provide more personalized information. Please take your time to complete the information you want and add some extra features!

[0371] "Behavioral data" refers to information about various actions that a user takes in their daily life, and includes, for example, location information, movement history, information about applications used, and types of activities.

[0372] "Emotion data" refers to information about the user's emotional state, and includes, for example, data representing emotions such as "relaxed," "stressed," "joyed," and "sad" obtained by analyzing voice tone and facial expressions.

[0373] "Data collection means" refers to the devices and processes that collect behavioral and emotional data using sensors and applications built into the device.

[0374] "Data analysis means" refers to algorithms or software that process collected behavioral and emotional data and analyze the user's behavioral patterns and emotional state.

[0375] "Information provision means" refers to a process or device that generates optimal information for the user based on analyzed data and provides it to the terminal as a notification or alarm.

[0376] "Action collection means" refers to a device or process for recording and recollecting actions taken by users based on notifications.

[0377] "Feedback data" refers to data regarding the actions taken by users based on the information provided and the results of those actions, and is used to improve the accuracy of the next information provided.

[0378] An "emotion engine" refers to a software component that analyzes voice data, facial expression data, etc. to recognize the user's emotions.

[0379] "Reminder" refers to a notification or alarm sent to remind a user of a specific time or activity.

[0380] "Analysis results" refers to information obtained after analyzing collected data, including insights into the user's behavioral patterns and emotional state.

[0381] "Recommended information" refers to information or suggestions that are generated based on the analysis results and that correspond to the user's situation and needs.

[0382] This invention relates to a system that collects and analyzes user behavioral and emotional data and provides optimal information to users based on the results. The system aims to improve the user's quality of life and is implemented using devices such as smartphones and tablets.

[0383] Overall system configuration

[0384] This system is broadly composed of the following elements:

[0385] 1. Device: A smartphone or tablet that users use on a daily basis, used to collect behavioral and emotional data.

[0386] 2. Server: A central system that analyzes the collected data and provides information based on the analysis results.

[0387] 3. Data collection methods: GPS, microphone, camera, and emotion recognition sensors built into the device.

[0388] 4. Data analysis means: Analysis algorithms and software in the server.

[0389] 5. Information provision means: Notification functions and alarms installed on the device.

[0390] 6. Action collection means: terminal applications and interfaces.

[0391] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[0392] About program processing

[0393] Data collection

[0394] Device:

[0395] Location information is collected using GPS.

[0396] Record audio data using a microphone.

[0397] Facial expressions and photographic data are captured using a camera.

[0398] The emotion engine recognizes the user's emotions from voice data and camera data.

[0399] Example: When a user goes for a walk, their location information is collected along with their emotions (relaxed, stressed, etc.).

[0400] User:

[0401] No special operations are required; just use your smartphone or tablet as you normally would.

[0402] Data transmission

[0403] Device:

[0404] The collected behavioral data and emotional data are periodically sent to a server.

[0405] User:

[0406] No special operations are required while going about your daily life.

[0407] Data reception and storage

[0408] server:

[0409] The data sent from the terminal is received and stored in a database.

[0410] Data analysis and information provision

[0411] Text conversion and recognition of voice and emotion data

[0412] server:

[0413] It uses a speech recognition algorithm to convert voice data into text, and an emotion engine to recognize the user's emotions, tagging them as "happy," "sad," or "stressed," for example.

[0414] Data analysis

[0415] server:

[0416] The system plots location data on a map and analyzes user behavior patterns. At the same time, it analyzes specific activity and emotional trends based on the text content of the voice data and emotional data. For example, it analyzes whether a user feels stressed when in a specific location.

[0417] information generation

[0418] server:

[0419] Based on the analysis results, appropriate information is generated for the user, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[0420] Information transmission

[0421] server:

[0422] Sends server-generated information to the user's device, including notifications and reminders.

[0423] Receiving information and displaying notifications

[0424] Device:

[0425] It displays the received information to the user and provides notifications or alarms as needed, for example, it displays a message to the user saying "You're relaxing now, keep it up."

[0426] Action collection and feedback analysis

[0427] Action Collection

[0428] Device:

[0429] If the user takes action based on the notification, that action is collected again, for example, recording the action taken based on the reminder (e.g., taking medicine, taking a deep breath, etc.).

[0430] Sending feedback data

[0431] Device:

[0432] The collected feedback data is sent to a server.

[0433] Analysis of feedback data and model updating

[0434] server:

[0435] The feedback data is analyzed to optimize the content and timing of the next information provision. For example, the feedback data can be used to understand the time of day when you want to relax, and the next time, relaxing music can be suggested.

[0436] Examples of concrete examples and prompts

[0437] Examples:

[0438] Example 1:

[0439] The emotion engine recognizes that the user takes a walk every day at 1 p.m. and feels relaxed during the walk. The device sends this data to the server.

[0440] The server analyzes the location information and recognizes that you are taking a walk in a particular park at 1 p.m., and the emotion engine detects your relaxed state.

[0441] The server generates information about relaxation (for example, a relaxation guide) and transmits it to the user's terminal.

[0442] The user reviews the information and takes action according to the relaxation guide.

[0443] The user's actions following the guide are recorded and the data is sent to a server.

[0444] The server analyzes this feedback and uses it for next time.

[0445] Example prompt sentence:

[0446] "Based on the fact that Person B relaxes in the park every day at 1 p.m., please suggest what kind of relaxation guide we should provide for Person B."

[0447] "If a user is feeling stressed, what kind of encouraging message would be effective to send them?"

[0448] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0449] Step 1: Data collection

[0450] Device:

[0451] Input: The smartphone or tablet that users use every day.

[0452] Specific operation:

[0453] Obtain location information using GPS.

[0454] Record audio data using a microphone.

[0455] Facial expressions and photographic data are captured using a camera.

[0456] The emotion engine recognizes the user's emotions (e.g., relaxed, stressed, etc.) from voice and camera data.

[0457] Data processing: Organize location information, audio data, camera data, and emotion recognition results and temporarily store them on the device.

[0458] Output: Organized behavioral and emotional data.

[0459] Step 2: Send data

[0460] Device:

[0461] Input: Organized behavioral and emotional data.

[0462] Specific operation:

[0463] The data is packetized and sent to the server at regular intervals.

[0464] Output: The data packets sent.

[0465] Step 3: Receiving and storing data

[0466] server:

[0467] Input: Data packets sent from the terminal.

[0468] Specific operation:

[0469] The data packets are received at the receiving port and decrypted.

[0470] Store the received data in a database.

[0471] Data processing: When storing data in a database, location information, audio data, photo data, and emotion data are stored separately in separate fields.

[0472] Output: Behavioral and emotional data stored in a database.

[0473] Step 4: Converting speech and emotion data into text and recognizing it

[0474] server:

[0475] Input: Audio data stored in a database.

[0476] Specific operation:

[0477] Convert the audio data into text using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API).

[0478] The text data is input into an emotion engine, which analyzes the user's emotions (e.g., "happy," "sad," "stressed").

[0479] Data processing: Convert the audio data into text and assign it an emotion tag.

[0480] Output: Translated speech data and emotion tags.

[0481] Step 5: Data analysis

[0482] server:

[0483] Input: location data, transcribed audio data, emotion tag.

[0484] Specific operation:

[0485] Location data is plotted on a map and behavioral patterns are analyzed.

[0486] Analyze specific activity and emotional trends based on text and emotion data.

[0487] For example, it analyzes whether a user feels stressed when in a particular place.

[0488] Data calculations: Statistical analysis of collected data to extract behavioral patterns and emotional trends.

[0489] Output: Analysis of behavioral patterns and emotional tendencies.

[0490] Step 6: Information Generation

[0491] server:

[0492] Input: Analysis results.

[0493] Specific operation:

[0494] Based on the analysis results, appropriate information is generated for the user.

[0495] For example, providing a relaxation guide when the user is relaxing.

[0496] Data processing: Generate appropriate notification messages and reminders based on the analysis results.

[0497] Output: The generated notification message or reminder.

[0498] Step 7: Send information

[0499] server:

[0500] Input: The generated notification message or reminder.

[0501] Specific operation:

[0502] The generated message is sent to the user's terminal.

[0503] Output: Notification message sent to the terminal.

[0504] Step 8: Receiving information and displaying notifications

[0505] Device:

[0506] Input: Notification messages and reminders sent from the server.

[0507] Specific operation:

[0508] The received information is displayed to the user.

[0509] Sound notifications and alarms as needed.

[0510] Output: Messages and reminders displayed on the user's device.

[0511] Step 9: Action Collection

[0512] Device:

[0513] Input: The action the user takes based on the notification.

[0514] Specific operation:

[0515] Record user actions within the device.

[0516] For example, record actions such as taking medicine or taking deep breaths based on reminders.

[0517] Output: Recorded user action data.

[0518] Step 10: Submitting feedback data

[0519] Device:

[0520] Input: Recorded user action data.

[0521] Specific operation:

[0522] Send the feedback data to the server.

[0523] Output: The feedback data sent.

[0524] Step 11: Analyze feedback data and update the model

[0525] server:

[0526] Input: Feedback data submitted by the user.

[0527] Specific operation:

[0528] Analyze feedback data to optimize the content and timing of the next information provision.

[0529] Generative AI models learn from feedback data to maintain a more accurate information delivery system.

[0530] Output: Updated AI model and optimized information delivery system.

[0531] (Application example 2)

[0532] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0533] Conventional information provision systems were limited to providing optimal information based on user behavior data, but no systems took the user's emotions into consideration. This made it difficult to provide appropriate information to alleviate the psychological challenges and stress that users actually experience. Furthermore, there were insufficient means to utilize user feedback on given information to improve the accuracy of future information provision.

[0534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0535] In this invention, the server includes means for collecting and analyzing user behavioral data and emotional data, means for providing optimal content based on the analysis results, and means for collecting user actions based on the provided content. This makes it possible to provide optimal content that takes into account the user's emotional state, thereby reducing the user's psychological burden and improving their quality of life. Furthermore, analyzing the feedback data can improve the accuracy of future content provision.

[0536] "Behavioral data" refers to data related to the user's daily behavior, such as the user's movement history, application usage history, and interaction history.

[0537] "Emotional data" is data that indicates the emotional state of a user, obtained by analyzing the user's tone of voice, facial expressions, text messages, and the like.

[0538] "Content" includes information and entertainment elements provided to users, such as videos, music, articles, and guides.

[0539] "Location information" is data related to a user's current location or movement history, and is obtained using GPS, Wi-Fi information, etc.

[0540] "Analysis means" refers to algorithms or software that analyze collected data and infer user behavioral patterns and emotional states.

[0541] "Recommended content" is content that is provided in a form that is optimal for the user's emotional state and behavioral patterns based on the analysis results.

[0542] "Feedback data" refers to data that indicates the user's reaction and actions to the provided content.

[0543] This invention relates to a system that collects user behavioral data and emotional data, analyzes the data, and provides optimal content to the user. The main components of this system are a terminal that collects user behavioral data and emotional data, a server that analyzes the collected data, means for providing content based on the analysis results, means for collecting user reactions, data analysis means, and an emotion engine.

[0544] Overall system overview

[0545] 1. Device:

[0546] The main collection method is the smartphone used by users in their daily lives, and behavioral and emotional data is collected using GPS, microphones, cameras, and emotion recognition sensors. This allows users to provide data in a natural way without any special operations.

[0547] 2. Server:

[0548] The collected data is received and stored in a database. The server is equipped with a speech recognition algorithm that converts voice data into text, and an emotion analysis algorithm that uses an emotion engine. These functions enable detailed analysis of the user's behavioral patterns and emotional state.

[0549] 3. Emotion Engine:

[0550] This software component analyzes the user's emotions from voice and camera data. For example, if the user is feeling stressed, it can detect their stress level from their voice tone and facial expression.

[0551] 4. Data analysis methods:

[0552] The system plots location and behavioral data on a map and analyzes the user's behavioral patterns. From the analysis results, it identifies the times and places where the user feels relaxed and the situations in which they feel stressed.

[0553] 5. Information provision method:

[0554] The server generates content (e.g., relaxation music, stress relief guides) and sends it to the user's device, notifying them with notifications and alarms. For example, if the server determines that the user is relaxed, it provides music to help them maintain that state.

[0555] 6. How we collect user actions:

[0556] The device records the user's actions based on the provided content (e.g., following a relaxation guide) and sends them back to the server. This allows the system to receive user feedback and analyze the data to improve the accuracy of the information provided.

[0557] Specific Examples

[0558] example:

[0559] If the emotion engine detects through analysis that the user is feeling stressed at work, the server generates relaxation music and a guide video for stress relief and provides it to the user's smartphone. The results of the user's actions in response are collected and used to make the next recommendation.

[0560] Example prompt sentence:

[0561] "I want to build a system that provides recommended content when a user is feeling stressed. Please tell me how to analyze voice and location data and recommend relaxation guides and music based on the user's emotions."

[0562] This system uses TensorFlow / Keras to build an emotion analysis model and collects data using the smartphone's GPS, microphone, and camera. It also uses Amazon RDS and Google Firebase as databases, generating and providing appropriate content based on the analysis results. This enables highly accurate information provision based on the user's emotional state and behavior.

[0563] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0564] Step 1:

[0565] Data collection

[0566] The device (smartphone) collects the user's behavioral data (location information, application history) and emotional data (voice tone, facial expressions). The device's sensor data (GPS, microphone, camera) is used as input, and the behavioral data and emotional data are obtained as output. This data is temporarily stored within the device.

[0567] Step 2:

[0568] Data transmission

[0569] The terminal transmits the collected behavioral data and emotional data to the server. The input is the collected behavioral data and emotional data, and the output is the data transmitted to the server. The data is transmitted periodically.

[0570] Step 3:

[0571] Data reception and storage

[0572] The server receives behavioral and emotional data sent from the device. It uses the data sent as input and stores it in a database as output. It uses database systems such as Amazon RDS and Google Firebase.

[0573] Step 4:

[0574] Speech to text conversion and sentiment analysis

[0575] The server converts the received voice data into text using a speech recognition algorithm (e.g., Google Speech-to-Text) and analyzes emotions using an emotion analysis engine (e.g., TensorFlow / Keras). The input is voice data, and the output is text data and emotion tags (e.g., happy, sad, stressed).

[0576] Step 5:

[0577] Data analysis

[0578] The server plots the behavioral data on a map and analyzes the behavioral patterns. At the same time, it analyzes the user's specific activities and emotional tendencies from the textualized voice data and emotional data. Location data and textualized voice data are used as input, and the output is the user's behavioral patterns and emotional tendencies.

[0579] Step 6:

[0580] information generation

[0581] The server generates content (e.g., relaxation guides, music to relieve stress, etc.) to recommend to users based on the analysis results. The input is behavioral patterns and emotional tendency information, and the output is the generated content.

[0582] Step 7:

[0583] Information transmission

[0584] The server sends the generated content to the user's terminal. The input is the generated content, and the output is the content sent to the terminal.

[0585] Step 8:

[0586] Receiving information and displaying notifications

[0587] The terminal receives the content sent from the server and displays it to the user using a notification or alarm function. The input is the content sent from the server, and the output is the notification content displayed on the user's screen.

[0588] Step 9:

[0589] Action Collection

[0590] The terminal records the results of the user's actions based on the notification. The input is the user's actions, and the output is the action record data.

[0591] Step 10:

[0592] Sending feedback data

[0593] The terminal sends the collected action data to the server. The input is the collected action data, and the output is the feedback data sent to the server.

[0594] Step 11:

[0595] Analysis of feedback data and model updating

[0596] The server analyzes the feedback data and updates the model to optimize the content and timing of the next information provision. The input is the feedback data, and the output is the updated analytical model and information provision algorithm.

[0597] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0598] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0599] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0600] [Second embodiment]

[0601] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0602] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0603] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0604] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0605] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0606] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0607] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0608] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0609] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0610] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0611] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0612] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0613] The present invention relates to a system for collecting user behavior data, analyzing the data, and providing optimal information to the user. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[0614] Overall system configuration

[0615] The main components of the system are:

[0616] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[0617] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[0618] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[0619] 4. Data analysis means: Analysis algorithms and software installed on the server.

[0620] 5. Information provision means: Notification functions and alarms installed on the device.

[0621] 6. Action collection means: Software that allows the terminal to recollect user actions.

[0622] Program processing

[0623] Below, each element of the system and its specific operation will be explained.

[0624] 1. Data Collection:

[0625] Device: The user's device collects real-time location information using GPS, records audio data using a microphone, and captures photos using a camera. These data are temporarily stored on the device. For example, when a user goes for a walk every day, the route and time of the walk are recorded.

[0626] 2. Data transmission:

[0627] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[0628] 3. Data Analysis:

[0629] Server: The server analyzes the received data. It plots GPS data on a map, converts voice data into text, and tags photos with AI. For example, it analyzes a daily activity log to determine when, where, and what the user is doing.

[0630] 4. Information generation:

[0631] Server: Based on the analysis results, messages are generated to provide the user with the necessary information in a timely manner. For example, if a user misses a regular dose of medication, a notification is generated to remind them to take it.

[0632] 5. Information provision:

[0633] Terminal: The information sent from the server is displayed on the terminal. Notifications are displayed and alarms sound if necessary. The user can act based on this information. For example, taking medicine after seeing the notification.

[0634] 6. Action Collection:

[0635] Device: The device records the user's actions (if they take their medicine based on the notification, that action) again. When the user presses the "take medicine" button, the data is sent to the server again.

[0636] 7. Feedback Analysis:

[0637] Server: Analyzes the received feedback data and improves the accuracy of the next notification. For example, it learns medication habits and optimizes the next notification time.

[0638] Specific examples

[0639] Example 1: Mr. A (70 years old) sets his schedule to take a walk at 7:00 a.m. every day and take his medicine at 8:00 a.m. In this case, the following series of processes will be performed.

[0640] Data collection: Mr. A's device collects GPS data and takes voice notes at the start of his walk at 7:00 AM.

[0641] Data transmission: After the walk ends, the data is transmitted to the server just before 8:00 AM.

[0642] Data analysis: The server analyzes the GPS data to determine the end time of Mr. A's walk and analyzes the time of taking his medicine from the voice memo.

[0643] Information generation: The server generates a notification message saying "It's time to take your medicine" just before 8:00 AM.

[0644] Information provided: The notification appears on the device and Person A checks it.

[0645] Action collection: Person A takes medicine and presses the "take" button on the device.

[0646] Feedback analysis: The server receives dose timing data and improves the accuracy of the next notification.

[0647] In this way, the system of the present invention can provide timely information to elderly people without them having to search for the information themselves, thereby improving their quality of life.

[0648] The processing flow will be explained below.

[0649] Step 1:

[0650] Data collection

[0651] Device: Your device periodically collects location information using GPS, records audio data with its built-in microphone, and temporarily stores photos captured by its camera.

[0652] User: Data is collected from users as they go about their daily lives without them having to take any special action.

[0653] Step 2:

[0654] Data transmission

[0655] Device: When a certain amount of collected data is reached or a certain time has passed, the device sends this data in batches to the server.

[0656] Step 3:

[0657] Data reception and storage

[0658] Server: Receives data sent from the device and stores it in a database, including location information, audio data, and photo data.

[0659] Step 4:

[0660] Converting audio data to text

[0661] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the speech "I ate breakfast" is converted into text "I ate breakfast."

[0662] Step 5:

[0663] Data analysis

[0664] Server: Plots the received location information on a map and analyzes the user's daily movements. It also analyzes the converted voice data to identify activities and priorities.

[0665] User: The system automatically analyzes the data without any special awareness.

[0666] Step 6:

[0667] information generation

[0668] Server: Generates necessary information for the user based on the analysis results. For example, if the pattern of "taking a walk at 7:00 AM and then taking medicine at 8:00 AM" is analyzed, a notification urging the user to take their medicine is generated.

[0669] Step 7:

[0670] Information transmission

[0671] Server: Sends the generated information to the user's device. This message contains notifications and reminders.

[0672] Step 8:

[0673] Receiving information and displaying notifications

[0674] Device: The received information is displayed on the device screen and, if necessary, the user is notified using push notifications or alarm functions.

[0675] User: Check the notification on the device and take the necessary action (e.g., take medicine).

[0676] Step 9:

[0677] Action Collection

[0678] On the device: If the user takes action based on the notification, provide a feedback form or button to record that action. When the user presses the "take a dose" button, that information is collected.

[0679] Step 10:

[0680] Sending feedback data

[0681] Device: Send the collected feedback data back to the server.

[0682] User: Follow the instructions on the device to enter feedback information.

[0683] Step 11:

[0684] Analysis of feedback data

[0685] Server: Analyzes the received feedback data and optimizes the timing and content of the next information provision. For example, data is added to fine-tune the time at which the user takes their medicine.

[0686] By repeating these steps, the system can continuously learn the user's behavioral patterns and provide the most appropriate information in a timely manner.

[0687] Example 1

[0688] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0689] The problem to be solved by this invention is to effectively collect and analyze user behavior data and provide optimal information to the user based on the analysis results. Furthermore, there is a need to collect user actions based on the provided information and analyze this as feedback to improve the accuracy of the next information provided. There is also a need to provide a system that can provide more accurate user support by integrating various data sources, such as location information and voice data.

[0690] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0691] In this invention, the server includes means for collecting user behavioral data, means for saving the collected behavioral data and transmitting it under specific conditions, means for analyzing the transmitted data, means for generating information optimized for the user based on the analysis results, means for providing the generated information to the user, means for collecting user actions based on the provided information, and means for analyzing the collected action data and improving the accuracy of the next information provision. This makes it possible to effectively collect and analyze user behavioral data and provide individually optimized information. Furthermore, it is possible to provide feedback on the user's reactions and improve the accuracy of the next information provision based on that data.

[0692] "Behavioral data" is information related to the user's actions and behavior in daily life, and includes location information, voice data, photograph data, and the like.

[0693] "Location Information" means a user's geographic coordinate data obtained using GPS or other location-determining technology.

[0694] "Collection means" refers to the technologies and devices used to collect user behavioral data, and primarily refers to sensors such as GPS, microphones, and cameras.

[0695] "Analysis means" refers to the techniques and algorithms used to analyze collected data, and are used to extract data features and recognize patterns.

[0696] "Information generation means" refers to techniques and scripts for generating information to be provided to users based on the analysis results.

[0697] "Providing means" refers to the technology or device used to transmit the generated information to the user, such as a notification function or alarm.

[0698] "Action Data" is a record of when a user acts based on the information provided.

[0699] "Feedback analysis means" refers to technologies and algorithms that analyze collected action data and improve the accuracy of the next information provided.

[0700] MODE FOR CARRYING OUT THE INVENTION

[0701] The present invention relates to a system that collects user behavior data, analyzes the data, and provides optimal information to the user. The system is composed of the following elements:

[0702] Overall system configuration

[0703] The main components of the system are:

[0704] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[0705] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[0706] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[0707] 4. Data analysis means: Analysis algorithms and software installed on the server.

[0708] 5. Information provision means: Notification functions and alarms installed on the device.

[0709] 6. Action collection means: Software that allows the terminal to recollect user actions.

[0710] Data collection

[0711] Device: The smartphone or tablet that a user carries around collects behavioral data using built-in sensors such as GPS, microphone, and camera. Location information is acquired in real time by the GPS sensor, audio data is recorded through the microphone, and photo data is taken with the camera. This data is temporarily stored on the device.

[0712] Data transmission

[0713] Device: The collected data is sent to the server after a certain period of time has passed or when a certain amount of data has been collected. Wi-Fi or mobile data is used as the communication method.

[0714] Data analysis

[0715] Server: The server analyzes the data it receives. GPS data is plotted on a map, voice data is converted to text, and photo data is tagged with AI to identify user behavioral patterns and interests.

[0716] information generation

[0717] Server: Generates necessary information for the user based on the analysis results. For example, if a user misses a medication time, it generates a reminder message.

[0718] Information provision

[0719] Device: Provides the user with the information sent from the server. The notification is displayed on the device screen and, if necessary, an audio alarm is set. The user can then take action.

[0720] Action Collection

[0721] Device: After providing the information, the device collects the user's actual actions. For example, after a notification to take medicine is displayed, pressing the "take medicine" button records the action.

[0722] Feedback Analysis

[0723] Server: Analyzes user action data and optimizes the timing and content of the next information provided, thereby providing the most appropriate information for the user.

[0724] Specific examples

[0725] Example 1:

[0726] Mr. A (70 years old) schedules his daily walk at 7:00 AM and his medication at 8:00 AM. The following process takes place:

[0727] Data collection: When Mr. A starts his walk at 7:00 AM, his device acquires GPS data and also records audio data.

[0728] Data transmission: After the walk is completed, the collected data is sent to the server.

[0729] Data analysis: The server analyzes the walking route and time from GPS data, and confirms the time to take medicine from voice data.

[0730] Information generation: The server generates a notification just before 8:00 AM saying "It's time to take your medicine."

[0731] Information provided: A notification appears on the device and Person A checks it.

[0732] Action collection: After taking the medicine, Person A presses the "take button."

[0733] Feedback analysis: The server optimizes the next notification time based on the action data.

[0734] In this way, the system of the invention can provide users with timely and relevant information to improve their quality of life.

[0735] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0736] System program processing flow

[0737] Step 1:

[0738] Data collection

[0739] Device: The user's device uses the built-in GPS to obtain real-time location information, uses a microphone to record ambient sounds and voice memos, and also uses a camera to take photos at regular intervals.

[0740] What it does: The device's GPS sensor periodically logs the current location, the microphone sensor periodically records audio, and the camera also periodically takes pictures and saves them.

[0741] Input: User's real-time location, ambient sounds, and photos.

[0742] Output: Location data, audio data, and photo data temporarily stored on the device.

[0743] Step 2:

[0744] Data transmission

[0745] Device: After a certain amount of time has passed or a certain amount of collected data has been collected, the device will send the data to the server via Wi-Fi or mobile data.

[0746] Specific operation: The device uses an internal timer and data volume monitoring function to connect to the server and upload data when conditions are met.

[0747] Input: Location data, audio data, and photo data stored on the device.

[0748] Output: Location data, audio data, and photo data sent to the server.

[0749] Step 3:

[0750] Data analysis

[0751] Server: The server analyzes the data it receives: GPS data is plotted on a map, voice data is converted to text, and photo data is tagged using AI.

[0752] What it does: The server uses a map API to display location data on a map, uses voice recognition software to convert voice data to text, and uses image analysis algorithms to tag photos.

[0753] Input: Location data, audio data, and photo data sent to the server.

[0754] Output: Parsed map data, text data, tagged image data.

[0755] Step 4:

[0756] information generation

[0757] Server: Based on the analysis results, the server generates information to provide to the user, for example, a reminder if the user is about to miss a dose of medication.

[0758] What happens: The server evaluates certain conditions based on the parsed data and executes scripts that automatically generate notification messages if those conditions are met.

[0759] Input: Parsed map data, text data, tagged image data.

[0760] Output: A notification message to the user.

[0761] Step 5:

[0762] Information provision

[0763] Device: To provide the user with the information sent from the server, the device will display a notification on the screen and sound an alarm if necessary.

[0764] Specific behavior: The device's notification system receives the notification message and displays it to the user as a popup. It also invokes the function to sound an audio alarm at the specified time.

[0765] Input: The notification message sent by the server.

[0766] Output: Notifications displayed on the device screen, alarms sounding.

[0767] Step 6:

[0768] Action Collection

[0769] Device: If the user acts on the notification, the action is collected again. For example, if the user presses the "take a pill" button, the action data is recorded.

[0770] What it does: The device interface detects user actions, logs the information, and uploads it to a server.

[0771] Input: User action (e.g., pressing the dose button).

[0772] Output: Action data sent from the device to the server.

[0773] Step 7:

[0774] Feedback Analysis

[0775] Server: The server analyzes the action data it receives and optimizes the timing and content of the next information provision.

[0776] What it does: The server's analysis algorithm evaluates your action data and automatically adjusts the timing and content of your next notification based on the results.

[0777] Input: Action data sent from the device.

[0778] Output: Optimized next notification settings.

[0779] (Application example 1)

[0780] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0781] Conventional information provision systems were able to collect and analyze user behavior data, but were insufficient for providing customized information in real time. Furthermore, they lacked the technology to simultaneously analyze user behavior patterns and interests and make highly accurate recommendations. Furthermore, they were unable to flexibly generate information using generative AI models, making it difficult to provide information tailored to user needs.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0783] In this invention, the server includes means for collecting user behavior data, means for analyzing the collected behavior data, means for providing optimal information to the user based on the analysis results, means for collecting user actions based on the provided information, means for analyzing the collected action data and improving the accuracy of the next information provided, means for simultaneously collecting and analyzing audio data, video data, and location information, and means for providing customized product information to the user in real time based on the generated analysis results. This makes it possible to analyze user behavior with high accuracy and provide optimal information in real time.

[0784] "Behavioral data" refers to various types of information related to the user's daily activities, such as location information, audio data, and video data.

[0785] "Analysis" refers to the processing of collected data to identify user behavior patterns and interests.

[0786] "Optimal information" refers to useful information that is individually customized based on the user's behavior and interests.

[0787] "Action data" refers to data regarding actions and operations performed by a user based on the information provided.

[0788] A "generative AI model" is an artificial intelligence model used to generate optimal information for each user in real time based on collected data.

[0789] A "prompt" is text containing instructions or suggestions that a generative AI model generates to provide information.

[0790] "Customized product information" is information about products selected according to the user's behavioral patterns and interests.

[0791] "Providing in real time" refers to providing information instantly in conjunction with the user's current actions.

[0792] This invention is a system that collects user behavior data, analyzes the data, and provides optimal information to the user in real time. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[0793] Overall system configuration

[0794] The main components of the system are as follows:

[0795] 1. Device: Smart glasses or smartphones that users use on a daily basis, which collect behavioral data (location information, video data, audio data).

[0796] 2. Server: This is the central system that analyzes behavioral data and provides optimal information to users based on the analysis results.

[0797] 3. Data collection means: Sensors such as GPS, camera, and microphone equipped on the device.

[0798] 4. Data analysis means: Analysis algorithms and generative AI models installed on the server.

[0799] 5. Information provision means: Notification functions and alarms installed on the device.

[0800] 6. Action collection means: Software that allows the terminal to collect user actions.

[0801] 7. Generative AI model: An artificial intelligence model that generates prompts in real time based on user behavior data.

[0802] Program processing

[0803] 1. Data Collection:

[0804] Device: The user's smart glasses or smartphone collects real-time location information using GPS, captures video data with a camera, and records audio data with a microphone. These data are temporarily stored on the device.

[0805] 2. Data transmission:

[0806] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[0807] 3. Data Analysis:

[0808] Server: The server analyzes the received data. It plots GPS data on a map, analyzes video data using an image processing library (e.g., OpenCV), and converts audio data to text using a machine learning library (e.g., TensorFlow).

[0809] 4. Information generation:

[0810] Server: Based on the analysis results, the server uses a generative AI model to generate information to be provided to the user in the form of prompt sentences. For example, it generates prompt sentences that display information about products that the user is interested in in real time.

[0811] 5. Information provision:

[0812] Terminal: The prompt sent from the server is displayed on the terminal, notifications are displayed and alarms are sounded if necessary, and the user can act accordingly.

[0813] 6. Action Collection:

[0814] Device: The device again records the user's actions (e.g., purchasing the presented product, reacting to the displayed information).

[0815] 7. Feedback Analysis:

[0816] Server: Analyzes the received feedback data and improves the accuracy of the next information provided.

[0817] Specific examples

[0818] Example 1:

[0819] As users walk around the virtual store, their behavioral data (location, video, and audio data) is collected in real time. The server analyzes this data to identify the user's interests. For example, if a user spends a long time looking at a particular group of products, the generative AI model generates and provides detailed information about those products and suggestions for related products. In this case, the generative AI model generates prompts such as "You might be interested in this product" and displays them on the smart glasses.

[0820] Example 2:

[0821] When a user is walking through a food section, the server predicts the ingredients the user will need based on their behavioral data and past purchase history, and displays a prompt such as "We recommend this ingredient."

[0822] Example prompt sentence:

[0823] "This product might interest you"

[0824] "I recommend this ingredient."

[0825] "This section has the product you're looking for"

[0826] In this way, the system of the present invention individually optimizes the shopping experience in the virtual store, making it more convenient for the user.

[0827] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0828] Step 1:

[0829] Data collection

[0830] The device starts collecting the user's location information in real time using GPS, where the input is the user's real-time location information, which is obtained from the GPS sensor.

[0831] At the same time, the camera is used to capture video data and the microphone is used to record audio data. These data are temporarily stored in the device. The input is the video data from the camera and the audio data from the microphone, and these are collected.

[0832] Step 2:

[0833] Data transmission

[0834] The terminal transmits the data to the server after a certain time has passed or when the amount of collected data reaches a certain amount. The input is the data collected in step 1, and the output is the data sent to the server.

[0835] Step 3:

[0836] Data analysis

[0837] The server analyzes the received data. First, it plots the GPS data on a map to visualize the user's location fluctuations. The input is location data, and the output is a visualization of the location.

[0838] Next, the video data is analyzed using an image processing library (e.g., OpenCV) to identify the scene and the user's interests. The input is the video data, and the output is the analyzed scene information.

[0839] The audio data is converted to text using a machine learning library (e.g., TensorFlow). The input is the audio data, and the output is the converted audio data.

[0840] Step 4:

[0841] information generation

[0842] The server generates prompt sentences using a generative AI model based on the analysis results from step 3. For example, if the user is interested in a specific product category, it generates prompt sentences for related products. The input is the analysis results, and the output is the generated prompt sentences.

[0843] Step 5:

[0844] Information provision

[0845] The terminal displays the prompt sentence received from the server in real time, for example, by displaying a notification on the display of smart glasses. The input is the generated prompt sentence, and the output is the notification displayed to the user.

[0846] Step 6:

[0847] Action Collection

[0848] The terminal again records the user's actions (e.g., purchasing a suggested product or reacting to a notification). The input is the user's action, and the output is the recorded action data.

[0849] Step 7:

[0850] Feedback Analysis

[0851] The server analyzes the action data and improves the accuracy of the next information provided. For example, if a particular prompt phrase is likely to lead to a user action, it learns that pattern. The input is the recorded action data, and the output is a new pattern that will be reflected in improving the accuracy of the next information provided.

[0852] In this way, a system can be realized that analyzes user behavior in detail and provides optimal information in real time.

[0853] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0854] This invention combines an emotion engine with a system that collects and analyzes user behavioral data to provide optimal information to users. The system aims to improve the quality of users' lives by collecting and analyzing user behavioral and emotional data and providing appropriate information based on this data.

[0855] Overall system configuration

[0856] The main components of the system are:

[0857] 1. Device: A smartphone or tablet used by a user in their daily life, used to collect behavioral and emotional data.

[0858] 2. Server: A central system that analyzes the collected data and provides information based on the results.

[0859] 3. Data collection methods: GPS, microphone, camera and emotion recognition sensors on the device.

[0860] 4. Data analysis means: Analysis algorithms and software in the server.

[0861] 5. Information provision means: Notification functions and alarms installed on the device.

[0862] 6. Action collection means: terminal applications and interfaces.

[0863] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[0864] Program processing

[0865] The components of the system and their specific operations are described below.

[0866] 1. Data Collection:

[0867] Device: GPS is used to collect location information, a microphone is used to record audio data, and a camera is used to capture facial expressions and photographs. In addition, an emotion engine recognizes the user's emotions from audio and camera data. For example, when a user is taking a walk, location information is collected along with the emotions (relaxed, stressed, etc.) at the time.

[0868] Users: Just use a smartphone or tablet as you normally would, with no special operations required.

[0869] 2. Data transmission:

[0870] Terminal: Periodically transmits collected behavioral and emotional data to the server.

[0871] User: Continues with daily life without performing any special operations.

[0872] 3. Data Receipt and Storage:

[0873] Server: Receives data sent from the device and stores it in a database. Collected data includes location information, voice data, photo data, and emotion data.

[0874] 4. Speech and emotion data transcription and recognition:

[0875] Server: Converts the voice data into text using a speech recognition algorithm and recognizes the user's emotions using an emotion engine. For example, emotion tags are assigned such as "happy," "sad," and "stressed."

[0876] 5. Data Analysis:

[0877] Server: Plots location data on a map and analyzes user behavior patterns. At the same time, converts voice data into text and analyzes emotion data to identify specific activity and emotional trends. For example, it analyzes whether a user feels stressed when in a specific location.

[0878] 6. Information generation:

[0879] Server: Generates appropriate information for the user based on the analysis results, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[0880] 7. Information Transmission:

[0881] Server: Sends server-generated information to the user's device, including notifications and reminders.

[0882] 8. Information Receipt and Notification Display:

[0883] Device: Displays the received information to the user and provides notifications or alarms as needed. For example, the user might see a message saying "Relax now, keep it up."

[0884] 9. Action Collection:

[0885] On the device: If the user takes action based on the notification, collect that action again. For example, record the action taken based on the reminder (e.g., take medicine, take a deep breath).

[0886] 10. Feedback Data Submission:

[0887] Device: Sends collected feedback data to the server, including data based on user behavior.

[0888] 11. Feedback data analysis and model updating:

[0889] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, it identifies the best time to relax from the feedback data and suggests relaxing music for the next time.

[0890] Specific examples

[0891] Example 1: Mr. B (65 years old) uses it daily:

[0892] Data collection and transmission: The emotion engine recognizes that Mr. B takes a walk every day at 1 p.m. and feels relaxed during that time. The device transmits this data to the server.

[0893] Data analysis: The server analyzes the location information and recognizes that you are walking in a specific park at 1 PM. The emotion engine detects your state of relaxation.

[0894] Information generation and provision: The server generates information about relaxation (e.g., a relaxation guide) and sends it to Mr. B's terminal.

[0895] Information presentation and action: Person B checks the information and takes action according to the relaxation guide.

[0896] Feedback collection and transmission: The actions taken by User B following the guide are recorded and the data is sent to the server.

[0897] Feedback analysis: The server analyzes this feedback and uses it for next time.

[0898] In this way, the system of the present invention can combine the user's behavioral data and emotional data to provide optimal information and improve the quality of life.

[0899] The processing flow will be explained below.

[0900] Step 1:

[0901] Data collection

[0902] Device: To record the user's daily activities, the device uses a GPS sensor to collect location information, a microphone to record voice data, and a camera to capture photos and facial expression data. In addition, an emotion engine recognizes the user's emotions from voice and facial expressions in real time.

[0903] User: The person uses a smartphone or tablet on a daily basis and data is collected without any special action.

[0904] Step 2:

[0905] Data transmission

[0906] Terminal: The collected location information, voice data, photo data, and emotion data are sent to the server at regular intervals or when a certain amount of data is reached.

[0907] Step 3:

[0908] Data reception and storage

[0909] Server: Receives data sent from the device and stores it in a database, including location information, audio data, photo data, and emotion data.

[0910] Step 4:

[0911] Converting audio data to text

[0912] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the voice data "My walk this morning was pleasant" is converted into text.

[0913] Step 5:

[0914] emotion recognition

[0915] Server: Uses an emotion engine to recognize the user's emotions from voice data and photo data (mainly facial expressions). For example, it assigns emotion tags such as "relaxed," "stressed," and "happy" based on voice tone and facial expressions.

[0916] Step 6:

[0917] Data analysis

[0918] Server: Plots the collected location information on a map and analyzes the user's behavioral patterns. At the same time, it analyzes the converted voice data and recognized emotion data to understand trends in specific activities and emotions. For example, it analyzes whether the user always relaxes in a specific place (a park).

[0919] Step 7:

[0920] information generation

[0921] Server: Generates appropriate information for the user based on all the analysis results. For example, if the user is in the park at 1pm every day and the data shows that they are relaxing, it generates a message that provides them with tips on how to relax.

[0922] Step 8:

[0923] Information transmission

[0924] Server: Generates information messages and notifications and sends them to the user's device. The notification contents include reminders and emotional advice.

[0925] Step 9:

[0926] Receiving information and displaying notifications

[0927] Device: The device displays received information and notifications on the device screen and alerts the user with sound or vibration as needed. For example, a notification saying "Take deep breaths for relaxation" may appear on the screen.

[0928] Step 10:

[0929] Action Collection

[0930] Terminal: Collects feedback data based on user behavior. When a user follows a relaxation guide, their behavior is recorded. For example, data is collected by pressing a button that says "I took a deep breath."

[0931] Step 11:

[0932] Sending feedback data

[0933] Device: Sends collected feedback data to the server.

[0934] User: Follows instructions to take action and provides feedback information as a result.

[0935] Step 12:

[0936] Analysis of feedback data

[0937] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, if it turns out that a user finds it easier to relax at a certain time of day, it will suggest relaxation methods that are appropriate for that time of day from the next time.

[0938] In this way, this system can improve the quality of life of users by comprehensively analyzing their behavioral and emotional data and providing optimal information tailored to their individual needs.

[0939] Example 2

[0940] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0941] Conventional information provision systems provide information based solely on user behavior data, making it impossible to perform detailed analysis of the user's emotional state or behavioral patterns, resulting in the problem that the information provided does not adequately meet the user's needs.In addition, there is a lack of a mechanism for integrating and analyzing emotional data in addition to voice data and location information, making it difficult to effectively provide the most appropriate information to the user.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user behavioral data and emotional data, means for analyzing the collected data, and means for providing optimal information to the user based on the analysis results. This makes it possible to combine and analyze the user's behavioral data and emotional data, and to provide more personalized information. Please take your time to complete the information you want and add some extra features!

[0943] "Behavioral data" refers to information about various actions that a user takes in their daily life, and includes, for example, location information, movement history, information about applications used, and types of activities.

[0944] "Emotion data" refers to information about the user's emotional state, and includes, for example, data representing emotions such as "relaxed," "stressed," "joyed," and "sad" obtained by analyzing voice tone and facial expressions.

[0945] "Data collection means" refers to the devices and processes that collect behavioral and emotional data using sensors and applications built into the device.

[0946] "Data analysis means" refers to algorithms or software that process collected behavioral and emotional data and analyze the user's behavioral patterns and emotional state.

[0947] "Information provision means" refers to a process or device that generates optimal information for the user based on analyzed data and provides it to the terminal as a notification or alarm.

[0948] "Action collection means" refers to a device or process for recording and recollecting actions taken by users based on notifications.

[0949] "Feedback data" refers to data regarding the actions taken by users based on the information provided and the results of those actions, and is used to improve the accuracy of the next information provided.

[0950] An "emotion engine" refers to a software component that analyzes voice data, facial expression data, etc. to recognize the user's emotions.

[0951] "Reminder" refers to a notification or alarm sent to remind a user of a specific time or activity.

[0952] "Analysis results" refers to information obtained after analyzing collected data, including insights into the user's behavioral patterns and emotional state.

[0953] "Recommended information" refers to information or suggestions that are generated based on the analysis results and that correspond to the user's situation and needs.

[0954] This invention relates to a system that collects and analyzes user behavioral and emotional data and provides optimal information to users based on the results. The system aims to improve the user's quality of life and is implemented using devices such as smartphones and tablets.

[0955] Overall system configuration

[0956] This system is broadly composed of the following elements:

[0957] 1. Device: A smartphone or tablet that users use on a daily basis, used to collect behavioral and emotional data.

[0958] 2. Server: A central system that analyzes the collected data and provides information based on the analysis results.

[0959] 3. Data collection methods: GPS, microphone, camera, and emotion recognition sensors built into the device.

[0960] 4. Data analysis means: Analysis algorithms and software in the server.

[0961] 5. Information provision means: Notification functions and alarms installed on the device.

[0962] 6. Action collection means: terminal applications and interfaces.

[0963] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[0964] About program processing

[0965] Data collection

[0966] Device:

[0967] Location information is collected using GPS.

[0968] Record audio data using a microphone.

[0969] Facial expressions and photographic data are captured using a camera.

[0970] The emotion engine recognizes the user's emotions from voice data and camera data.

[0971] Example: When a user goes for a walk, their location information is collected along with their emotions (relaxed, stressed, etc.).

[0972] User:

[0973] No special operations are required; just use your smartphone or tablet as you normally would.

[0974] Data transmission

[0975] Device:

[0976] The collected behavioral data and emotional data are periodically sent to a server.

[0977] User:

[0978] No special operations are required while going about your daily life.

[0979] Data reception and storage

[0980] server:

[0981] The data sent from the terminal is received and stored in a database.

[0982] Data analysis and information provision

[0983] Text conversion and recognition of voice and emotion data

[0984] server:

[0985] It uses a speech recognition algorithm to convert voice data into text, and an emotion engine to recognize the user's emotions, tagging them as "happy," "sad," or "stressed," for example.

[0986] Data analysis

[0987] server:

[0988] The system plots location data on a map and analyzes user behavior patterns. At the same time, it analyzes specific activity and emotional trends based on the text content of the voice data and emotional data. For example, it analyzes whether a user feels stressed when in a specific location.

[0989] information generation

[0990] server:

[0991] Based on the analysis results, appropriate information is generated for the user, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[0992] Information transmission

[0993] server:

[0994] Sends server-generated information to the user's device, including notifications and reminders.

[0995] Receiving information and displaying notifications

[0996] Device:

[0997] It displays the received information to the user and provides notifications or alarms as needed, for example, it displays a message to the user saying "You're relaxing now, keep it up."

[0998] Action collection and feedback analysis

[0999] Action Collection

[1000] Device:

[1001] If the user takes action based on the notification, that action is collected again, for example, recording the action taken based on the reminder (e.g., taking medicine, taking a deep breath, etc.).

[1002] Sending feedback data

[1003] Device:

[1004] The collected feedback data is sent to a server.

[1005] Analysis of feedback data and model updating

[1006] server:

[1007] The feedback data is analyzed to optimize the content and timing of the next information provision. For example, the feedback data can be used to understand the time of day when you want to relax, and the next time, relaxing music can be suggested.

[1008] Examples of concrete examples and prompts

[1009] Examples:

[1010] Example 1:

[1011] The emotion engine recognizes that the user takes a walk every day at 1 p.m. and feels relaxed during the walk. The device sends this data to the server.

[1012] The server analyzes the location information and recognizes that you are taking a walk in a particular park at 1 p.m., and the emotion engine detects your relaxed state.

[1013] The server generates information about relaxation (for example, a relaxation guide) and transmits it to the user's terminal.

[1014] The user reviews the information and takes action according to the relaxation guide.

[1015] The user's actions following the guide are recorded and the data is sent to a server.

[1016] The server analyzes this feedback and uses it for next time.

[1017] Example prompt sentence:

[1018] "Based on the fact that Person B relaxes in the park every day at 1 p.m., please suggest what kind of relaxation guide we should provide for Person B."

[1019] "If a user is feeling stressed, what kind of encouraging message would be effective to send them?"

[1020] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1021] Step 1: Data collection

[1022] Device:

[1023] Input: The smartphone or tablet that users use every day.

[1024] Specific operation:

[1025] Obtain location information using GPS.

[1026] Record audio data using a microphone.

[1027] Facial expressions and photographic data are captured using a camera.

[1028] The emotion engine recognizes the user's emotions (e.g., relaxed, stressed, etc.) from voice and camera data.

[1029] Data processing: Organize location information, audio data, camera data, and emotion recognition results and temporarily store them on the device.

[1030] Output: Organized behavioral and emotional data.

[1031] Step 2: Send data

[1032] Device:

[1033] Input: Organized behavioral and emotional data.

[1034] Specific operation:

[1035] The data is packetized and sent to the server at regular intervals.

[1036] Output: The data packets sent.

[1037] Step 3: Receiving and storing data

[1038] server:

[1039] Input: Data packets sent from the terminal.

[1040] Specific operation:

[1041] The data packets are received at the receiving port and decrypted.

[1042] Store the received data in a database.

[1043] Data processing: When storing data in a database, location information, audio data, photo data, and emotion data are stored separately in separate fields.

[1044] Output: Behavioral and emotional data stored in a database.

[1045] Step 4: Converting speech and emotion data into text and recognizing it

[1046] server:

[1047] Input: Audio data stored in a database.

[1048] Specific operation:

[1049] Convert the audio data into text using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API).

[1050] The text data is input into an emotion engine, which analyzes the user's emotions (e.g., "happy," "sad," "stressed").

[1051] Data processing: Convert the audio data into text and assign it an emotion tag.

[1052] Output: Translated speech data and emotion tags.

[1053] Step 5: Data analysis

[1054] server:

[1055] Input: location data, transcribed audio data, emotion tag.

[1056] Specific operation:

[1057] Location data is plotted on a map and behavioral patterns are analyzed.

[1058] Analyze specific activity and emotional trends based on text and emotion data.

[1059] For example, it analyzes whether a user feels stressed when in a particular place.

[1060] Data calculations: Statistical analysis of collected data to extract behavioral patterns and emotional trends.

[1061] Output: Analysis of behavioral patterns and emotional tendencies.

[1062] Step 6: Information Generation

[1063] server:

[1064] Input: Analysis results.

[1065] Specific operation:

[1066] Based on the analysis results, appropriate information is generated for the user.

[1067] For example, providing a relaxation guide when the user is relaxing.

[1068] Data processing: Generate appropriate notification messages and reminders based on the analysis results.

[1069] Output: The generated notification message or reminder.

[1070] Step 7: Send information

[1071] server:

[1072] Input: The generated notification message or reminder.

[1073] Specific operation:

[1074] The generated message is sent to the user's terminal.

[1075] Output: Notification message sent to the terminal.

[1076] Step 8: Receiving information and displaying notifications

[1077] Device:

[1078] Input: Notification messages and reminders sent from the server.

[1079] Specific operation:

[1080] The received information is displayed to the user.

[1081] Sound notifications and alarms as needed.

[1082] Output: Messages and reminders displayed on the user's device.

[1083] Step 9: Action Collection

[1084] Device:

[1085] Input: The action the user takes based on the notification.

[1086] Specific operation:

[1087] Record user actions within the device.

[1088] For example, record actions such as taking medicine or taking deep breaths based on reminders.

[1089] Output: Recorded user action data.

[1090] Step 10: Submitting feedback data

[1091] Device:

[1092] Input: Recorded user action data.

[1093] Specific operation:

[1094] Send the feedback data to the server.

[1095] Output: The feedback data sent.

[1096] Step 11: Analyze feedback data and update the model

[1097] server:

[1098] Input: Feedback data submitted by the user.

[1099] Specific operation:

[1100] Analyze feedback data to optimize the content and timing of the next information provision.

[1101] Generative AI models learn from feedback data to maintain a more accurate information delivery system.

[1102] Output: Updated AI model and optimized information delivery system.

[1103] (Application example 2)

[1104] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1105] Conventional information provision systems were limited to providing optimal information based on user behavior data, but no systems took the user's emotions into consideration. This made it difficult to provide appropriate information to alleviate the psychological challenges and stress that users actually experience. Furthermore, there were insufficient means to utilize user feedback on given information to improve the accuracy of future information provision.

[1106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1107] In this invention, the server includes means for collecting and analyzing user behavioral data and emotional data, means for providing optimal content based on the analysis results, and means for collecting user actions based on the provided content. This makes it possible to provide optimal content that takes into account the user's emotional state, thereby reducing the user's psychological burden and improving their quality of life. Furthermore, analyzing the feedback data can improve the accuracy of future content provision.

[1108] "Behavioral data" refers to data related to the user's daily behavior, such as the user's movement history, application usage history, and interaction history.

[1109] "Emotional data" is data that indicates the emotional state of a user, obtained by analyzing the user's tone of voice, facial expressions, text messages, and the like.

[1110] "Content" includes information and entertainment elements provided to users, such as videos, music, articles, and guides.

[1111] "Location information" is data related to a user's current location or movement history, and is obtained using GPS, Wi-Fi information, etc.

[1112] "Analysis means" refers to algorithms or software that analyze collected data and infer user behavioral patterns and emotional states.

[1113] "Recommended content" is content that is provided in a form that is optimal for the user's emotional state and behavioral patterns based on the analysis results.

[1114] "Feedback data" refers to data that indicates the user's reaction and actions to the provided content.

[1115] This invention relates to a system that collects user behavioral data and emotional data, analyzes the data, and provides optimal content to the user. The main components of this system are a terminal that collects user behavioral data and emotional data, a server that analyzes the collected data, means for providing content based on the analysis results, means for collecting user reactions, data analysis means, and an emotion engine.

[1116] Overall system overview

[1117] 1. Device:

[1118] The main collection method is the smartphone used by users in their daily lives, and behavioral and emotional data is collected using GPS, microphones, cameras, and emotion recognition sensors. This allows users to provide data in a natural way without any special operations.

[1119] 2. Server:

[1120] The collected data is received and stored in a database. The server is equipped with a speech recognition algorithm that converts voice data into text, and an emotion analysis algorithm that uses an emotion engine. These functions enable detailed analysis of the user's behavioral patterns and emotional state.

[1121] 3. Emotion Engine:

[1122] This software component analyzes the user's emotions from voice and camera data. For example, if the user is feeling stressed, it can detect their stress level from their voice tone and facial expression.

[1123] 4. Data analysis methods:

[1124] The system plots location and behavioral data on a map and analyzes the user's behavioral patterns. From the analysis results, it identifies the times and places where the user feels relaxed and the situations in which they feel stressed.

[1125] 5. Information provision method:

[1126] The server generates content (e.g., relaxation music, stress relief guides) and sends it to the user's device, notifying them with notifications and alarms. For example, if the server determines that the user is relaxed, it provides music to help them maintain that state.

[1127] 6. How we collect user actions:

[1128] The device records the user's actions based on the provided content (e.g., following a relaxation guide) and sends them back to the server. This allows the system to receive user feedback and analyze the data to improve the accuracy of the information provided.

[1129] Specific Examples

[1130] example:

[1131] If the emotion engine detects through analysis that the user is feeling stressed at work, the server generates relaxation music and a guide video for stress relief and provides it to the user's smartphone. The results of the user's actions in response are collected and used to make the next recommendation.

[1132] Example prompt sentence:

[1133] "I want to build a system that provides recommended content when a user is feeling stressed. Please tell me how to analyze voice and location data and recommend relaxation guides and music based on the user's emotions."

[1134] This system uses TensorFlow / Keras to build an emotion analysis model and collects data using the smartphone's GPS, microphone, and camera. It also uses Amazon RDS and Google Firebase as databases, generating and providing appropriate content based on the analysis results. This enables highly accurate information provision based on the user's emotional state and behavior.

[1135] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1136] Step 1:

[1137] Data collection

[1138] The device (smartphone) collects the user's behavioral data (location information, application history) and emotional data (voice tone, facial expressions). The device's sensor data (GPS, microphone, camera) is used as input, and the behavioral data and emotional data are obtained as output. This data is temporarily stored within the device.

[1139] Step 2:

[1140] Data transmission

[1141] The terminal transmits the collected behavioral data and emotional data to the server. The input is the collected behavioral data and emotional data, and the output is the data transmitted to the server. The data is transmitted periodically.

[1142] Step 3:

[1143] Data reception and storage

[1144] The server receives behavioral and emotional data sent from the device. It uses the data sent as input and stores it in a database as output. It uses database systems such as Amazon RDS and Google Firebase.

[1145] Step 4:

[1146] Speech to text conversion and sentiment analysis

[1147] The server converts the received voice data into text using a speech recognition algorithm (e.g., Google Speech-to-Text) and analyzes emotions using an emotion analysis engine (e.g., TensorFlow / Keras). The input is voice data, and the output is text data and emotion tags (e.g., happy, sad, stressed).

[1148] Step 5:

[1149] Data analysis

[1150] The server plots the behavioral data on a map and analyzes the behavioral patterns. At the same time, it analyzes the user's specific activities and emotional tendencies from the textualized voice data and emotional data. Location data and textualized voice data are used as input, and the output is the user's behavioral patterns and emotional tendencies.

[1151] Step 6:

[1152] information generation

[1153] The server generates content (e.g., relaxation guides, music to relieve stress, etc.) to recommend to users based on the analysis results. The input is behavioral patterns and emotional tendency information, and the output is the generated content.

[1154] Step 7:

[1155] Information transmission

[1156] The server sends the generated content to the user's terminal. The input is the generated content, and the output is the content sent to the terminal.

[1157] Step 8:

[1158] Receiving information and displaying notifications

[1159] The terminal receives the content sent from the server and displays it to the user using a notification or alarm function. The input is the content sent from the server, and the output is the notification content displayed on the user's screen.

[1160] Step 9:

[1161] Action Collection

[1162] The terminal records the results of the user's actions based on the notification. The input is the user's actions, and the output is the action record data.

[1163] Step 10:

[1164] Sending feedback data

[1165] The terminal sends the collected action data to the server. The input is the collected action data, and the output is the feedback data sent to the server.

[1166] Step 11:

[1167] Analysis of feedback data and model updating

[1168] The server analyzes the feedback data and updates the model to optimize the content and timing of the next information provision. The input is the feedback data, and the output is the updated analytical model and information provision algorithm.

[1169] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1170] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1171] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1172] [Third embodiment]

[1173] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1174] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1175] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1176] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1177] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1178] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1179] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1180] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1181] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1182] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1183] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1184] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1185] The present invention relates to a system for collecting user behavior data, analyzing the data, and providing optimal information to the user. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[1186] Overall system configuration

[1187] The main components of the system are:

[1188] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[1189] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[1190] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[1191] 4. Data analysis means: Analysis algorithms and software installed on the server.

[1192] 5. Information provision means: Notification functions and alarms installed on the device.

[1193] 6. Action collection means: Software that allows the terminal to recollect user actions.

[1194] Program processing

[1195] Below, each element of the system and its specific operation will be explained.

[1196] 1. Data Collection:

[1197] Device: The user's device collects real-time location information using GPS, records audio data using a microphone, and captures photos using a camera. These data are temporarily stored on the device. For example, when a user goes for a walk every day, the route and time of the walk are recorded.

[1198] 2. Data transmission:

[1199] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[1200] 3. Data Analysis:

[1201] Server: The server analyzes the received data. It plots GPS data on a map, converts voice data into text, and tags photos with AI. For example, it analyzes a daily activity log to determine when, where, and what the user is doing.

[1202] 4. Information generation:

[1203] Server: Based on the analysis results, messages are generated to provide the user with the necessary information in a timely manner. For example, if a user misses a regular dose of medication, a notification is generated to remind them to take it.

[1204] 5. Information provision:

[1205] Terminal: The information sent from the server is displayed on the terminal. Notifications are displayed and alarms sound if necessary. The user can act based on this information. For example, taking medicine after seeing the notification.

[1206] 6. Action Collection:

[1207] Device: The device records the user's actions (if they take their medicine based on the notification, that action) again. When the user presses the "take medicine" button, the data is sent to the server again.

[1208] 7. Feedback Analysis:

[1209] Server: Analyzes the received feedback data and improves the accuracy of the next notification. For example, it learns medication habits and optimizes the next notification time.

[1210] Specific examples

[1211] Example 1: Mr. A (70 years old) sets his schedule to take a walk at 7:00 a.m. every day and take his medicine at 8:00 a.m. In this case, the following series of processes will be performed.

[1212] Data collection: Mr. A's device collects GPS data and takes voice notes at the start of his walk at 7:00 AM.

[1213] Data transmission: After the walk ends, the data is transmitted to the server just before 8:00 AM.

[1214] Data analysis: The server analyzes the GPS data to determine the end time of Mr. A's walk and analyzes the time of taking his medicine from the voice memo.

[1215] Information generation: The server generates a notification message saying "It's time to take your medicine" just before 8:00 AM.

[1216] Information provided: The notification appears on the device and Person A checks it.

[1217] Action collection: Person A takes medicine and presses the "take" button on the device.

[1218] Feedback analysis: The server receives dose timing data and improves the accuracy of the next notification.

[1219] In this way, the system of the present invention can provide timely information to elderly people without them having to search for the information themselves, thereby improving their quality of life.

[1220] The processing flow will be explained below.

[1221] Step 1:

[1222] Data collection

[1223] Device: Your device periodically collects location information using GPS, records audio data with its built-in microphone, and temporarily stores photos captured by its camera.

[1224] User: Data is collected from users as they go about their daily lives without them having to take any special action.

[1225] Step 2:

[1226] Data transmission

[1227] Device: When a certain amount of collected data is reached or a certain time has passed, the device sends this data in batches to the server.

[1228] Step 3:

[1229] Data reception and storage

[1230] Server: Receives data sent from the device and stores it in a database, including location information, audio data, and photo data.

[1231] Step 4:

[1232] Converting audio data to text

[1233] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the speech "I ate breakfast" is converted into text "I ate breakfast."

[1234] Step 5:

[1235] Data analysis

[1236] Server: Plots the received location information on a map and analyzes the user's daily movements. It also analyzes the converted voice data to identify activities and priorities.

[1237] User: The system automatically analyzes the data without any special awareness.

[1238] Step 6:

[1239] information generation

[1240] Server: Generates necessary information for the user based on the analysis results. For example, if the pattern of "taking a walk at 7:00 AM and then taking medicine at 8:00 AM" is analyzed, a notification urging the user to take their medicine is generated.

[1241] Step 7:

[1242] Information transmission

[1243] Server: Sends the generated information to the user's device. This message contains notifications and reminders.

[1244] Step 8:

[1245] Receiving information and displaying notifications

[1246] Device: The received information is displayed on the device screen and, if necessary, the user is notified using push notifications or alarm functions.

[1247] User: Check the notification on the device and take the necessary action (e.g., take medicine).

[1248] Step 9:

[1249] Action Collection

[1250] On the device: If the user takes action based on the notification, provide a feedback form or button to record that action. When the user presses the "take a dose" button, that information is collected.

[1251] Step 10:

[1252] Sending feedback data

[1253] Device: Send the collected feedback data back to the server.

[1254] User: Follow the instructions on the device to enter feedback information.

[1255] Step 11:

[1256] Analysis of feedback data

[1257] Server: Analyzes the received feedback data and optimizes the timing and content of the next information provision. For example, data is added to fine-tune the time at which the user takes their medicine.

[1258] By repeating these steps, the system can continuously learn the user's behavioral patterns and provide the most appropriate information in a timely manner.

[1259] Example 1

[1260] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1261] The problem to be solved by this invention is to effectively collect and analyze user behavior data and provide optimal information to the user based on the analysis results. Furthermore, there is a need to collect user actions based on the provided information and analyze this as feedback to improve the accuracy of the next information provided. There is also a need to provide a system that can provide more accurate user support by integrating various data sources, such as location information and voice data.

[1262] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1263] In this invention, the server includes means for collecting user behavioral data, means for saving the collected behavioral data and transmitting it under specific conditions, means for analyzing the transmitted data, means for generating information optimized for the user based on the analysis results, means for providing the generated information to the user, means for collecting user actions based on the provided information, and means for analyzing the collected action data and improving the accuracy of the next information provision. This makes it possible to effectively collect and analyze user behavioral data and provide individually optimized information. Furthermore, it is possible to provide feedback on the user's reactions and improve the accuracy of the next information provision based on that data.

[1264] "Behavioral data" is information related to the user's actions and behavior in daily life, and includes location information, voice data, photograph data, and the like.

[1265] "Location Information" means a user's geographic coordinate data obtained using GPS or other location-determining technology.

[1266] "Collection means" refers to the technologies and devices used to collect user behavioral data, and primarily refers to sensors such as GPS, microphones, and cameras.

[1267] "Analysis means" refers to the techniques and algorithms used to analyze collected data, and are used to extract data features and recognize patterns.

[1268] "Information generation means" refers to techniques and scripts for generating information to be provided to users based on the analysis results.

[1269] "Providing means" refers to the technology or device used to transmit the generated information to the user, such as a notification function or alarm.

[1270] "Action Data" is a record of when a user acts based on the information provided.

[1271] "Feedback analysis means" refers to technologies and algorithms that analyze collected action data and improve the accuracy of the next information provided.

[1272] MODE FOR CARRYING OUT THE INVENTION

[1273] The present invention relates to a system that collects user behavior data, analyzes the data, and provides optimal information to the user. The system is composed of the following elements:

[1274] Overall system configuration

[1275] The main components of the system are:

[1276] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[1277] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[1278] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[1279] 4. Data analysis means: Analysis algorithms and software installed on the server.

[1280] 5. Information provision means: Notification functions and alarms installed on the device.

[1281] 6. Action collection means: Software that allows the terminal to recollect user actions.

[1282] Data collection

[1283] Device: The smartphone or tablet that a user carries around collects behavioral data using built-in sensors such as GPS, microphone, and camera. Location information is acquired in real time by the GPS sensor, audio data is recorded through the microphone, and photo data is taken with the camera. This data is temporarily stored on the device.

[1284] Data transmission

[1285] Device: The collected data is sent to the server after a certain period of time has passed or when a certain amount of data has been collected. Wi-Fi or mobile data is used as the communication method.

[1286] Data analysis

[1287] Server: The server analyzes the data it receives. GPS data is plotted on a map, voice data is converted to text, and photo data is tagged with AI to identify user behavioral patterns and interests.

[1288] information generation

[1289] Server: Generates necessary information for the user based on the analysis results. For example, if a user misses a medication time, it generates a reminder message.

[1290] Information provision

[1291] Device: Provides the user with the information sent from the server. The notification is displayed on the device screen and, if necessary, an audio alarm is set. The user can then take action.

[1292] Action Collection

[1293] Device: After providing the information, the device collects the user's actual actions. For example, after a notification to take medicine is displayed, pressing the "take medicine" button records the action.

[1294] Feedback Analysis

[1295] Server: Analyzes user action data and optimizes the timing and content of the next information provided, thereby providing the most appropriate information for the user.

[1296] Specific examples

[1297] Example 1:

[1298] Mr. A (70 years old) schedules his daily walk at 7:00 AM and his medication at 8:00 AM. The following process takes place:

[1299] Data collection: When Mr. A starts his walk at 7:00 AM, his device acquires GPS data and also records audio data.

[1300] Data transmission: After the walk is completed, the collected data is sent to the server.

[1301] Data analysis: The server analyzes the walking route and time from GPS data, and confirms the time to take medicine from voice data.

[1302] Information generation: The server generates a notification just before 8:00 AM saying "It's time to take your medicine."

[1303] Information provided: A notification appears on the device and Person A checks it.

[1304] Action collection: After taking the medicine, Person A presses the "take button."

[1305] Feedback analysis: The server optimizes the next notification time based on the action data.

[1306] In this way, the system of the invention can provide users with timely and relevant information to improve their quality of life.

[1307] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1308] System program processing flow

[1309] Step 1:

[1310] Data collection

[1311] Device: The user's device uses the built-in GPS to obtain real-time location information, uses a microphone to record ambient sounds and voice memos, and also uses a camera to take photos at regular intervals.

[1312] What it does: The device's GPS sensor periodically logs the current location, the microphone sensor periodically records audio, and the camera also periodically takes pictures and saves them.

[1313] Input: User's real-time location, ambient sounds, and photos.

[1314] Output: Location data, audio data, and photo data temporarily stored on the device.

[1315] Step 2:

[1316] Data transmission

[1317] Device: After a certain amount of time has passed or a certain amount of collected data has been collected, the device will send the data to the server via Wi-Fi or mobile data.

[1318] Specific operation: The device uses an internal timer and data volume monitoring function to connect to the server and upload data when conditions are met.

[1319] Input: Location data, audio data, and photo data stored on the device.

[1320] Output: Location data, audio data, and photo data sent to the server.

[1321] Step 3:

[1322] Data analysis

[1323] Server: The server analyzes the data it receives: GPS data is plotted on a map, voice data is converted to text, and photo data is tagged using AI.

[1324] What it does: The server uses a map API to display location data on a map, uses voice recognition software to convert voice data to text, and uses image analysis algorithms to tag photos.

[1325] Input: Location data, audio data, and photo data sent to the server.

[1326] Output: Parsed map data, text data, tagged image data.

[1327] Step 4:

[1328] information generation

[1329] Server: Based on the analysis results, the server generates information to provide to the user, for example, a reminder if the user is about to miss a dose of medication.

[1330] What happens: The server evaluates certain conditions based on the parsed data and executes scripts that automatically generate notification messages if those conditions are met.

[1331] Input: Parsed map data, text data, tagged image data.

[1332] Output: A notification message to the user.

[1333] Step 5:

[1334] Information provision

[1335] Device: To provide the user with the information sent from the server, the device will display a notification on the screen and sound an alarm if necessary.

[1336] Specific behavior: The device's notification system receives the notification message and displays it to the user as a popup. It also invokes the function to sound an audio alarm at the specified time.

[1337] Input: The notification message sent by the server.

[1338] Output: Notifications displayed on the device screen, alarms sounding.

[1339] Step 6:

[1340] Action Collection

[1341] Device: If the user acts on the notification, the action is collected again. For example, if the user presses the "take a pill" button, the action data is recorded.

[1342] What it does: The device interface detects user actions, logs the information, and uploads it to a server.

[1343] Input: User action (e.g., pressing the dose button).

[1344] Output: Action data sent from the device to the server.

[1345] Step 7:

[1346] Feedback Analysis

[1347] Server: The server analyzes the action data it receives and optimizes the timing and content of the next information provision.

[1348] What it does: The server's analysis algorithm evaluates your action data and automatically adjusts the timing and content of your next notification based on the results.

[1349] Input: Action data sent from the device.

[1350] Output: Optimized next notification settings.

[1351] (Application example 1)

[1352] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1353] Conventional information provision systems were able to collect and analyze user behavior data, but were insufficient for providing customized information in real time. Furthermore, they lacked the technology to simultaneously analyze user behavior patterns and interests and make highly accurate recommendations. Furthermore, they were unable to flexibly generate information using generative AI models, making it difficult to provide information tailored to user needs.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1355] In this invention, the server includes means for collecting user behavior data, means for analyzing the collected behavior data, means for providing optimal information to the user based on the analysis results, means for collecting user actions based on the provided information, means for analyzing the collected action data and improving the accuracy of the next information provided, means for simultaneously collecting and analyzing audio data, video data, and location information, and means for providing customized product information to the user in real time based on the generated analysis results. This makes it possible to analyze user behavior with high accuracy and provide optimal information in real time.

[1356] "Behavioral data" refers to various types of information related to the user's daily activities, such as location information, audio data, and video data.

[1357] "Analysis" refers to the processing of collected data to identify user behavior patterns and interests.

[1358] "Optimal information" refers to useful information that is individually customized based on the user's behavior and interests.

[1359] "Action data" refers to data regarding actions and operations performed by a user based on the information provided.

[1360] A "generative AI model" is an artificial intelligence model used to generate optimal information for each user in real time based on collected data.

[1361] A "prompt" is text containing instructions or suggestions that a generative AI model generates to provide information.

[1362] "Customized product information" is information about products selected according to the user's behavioral patterns and interests.

[1363] "Providing in real time" refers to providing information instantly in conjunction with the user's current actions.

[1364] This invention is a system that collects user behavior data, analyzes the data, and provides optimal information to the user in real time. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[1365] Overall system configuration

[1366] The main components of the system are as follows:

[1367] 1. Device: Smart glasses or smartphones that users use on a daily basis, which collect behavioral data (location information, video data, audio data).

[1368] 2. Server: This is the central system that analyzes behavioral data and provides optimal information to users based on the analysis results.

[1369] 3. Data collection means: Sensors such as GPS, camera, and microphone equipped on the device.

[1370] 4. Data analysis means: Analysis algorithms and generative AI models installed on the server.

[1371] 5. Information provision means: Notification functions and alarms installed on the device.

[1372] 6. Action collection means: Software that allows the terminal to collect user actions.

[1373] 7. Generative AI model: An artificial intelligence model that generates prompts in real time based on user behavior data.

[1374] Program processing

[1375] 1. Data Collection:

[1376] Device: The user's smart glasses or smartphone collects real-time location information using GPS, captures video data with a camera, and records audio data with a microphone. These data are temporarily stored on the device.

[1377] 2. Data transmission:

[1378] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[1379] 3. Data Analysis:

[1380] Server: The server analyzes the received data. It plots GPS data on a map, analyzes video data using an image processing library (e.g., OpenCV), and converts audio data to text using a machine learning library (e.g., TensorFlow).

[1381] 4. Information generation:

[1382] Server: Based on the analysis results, the server uses a generative AI model to generate information to be provided to the user in the form of prompt sentences. For example, it generates prompt sentences that display information about products that the user is interested in in real time.

[1383] 5. Information provision:

[1384] Terminal: The prompt sent from the server is displayed on the terminal, notifications are displayed and alarms are sounded if necessary, and the user can act accordingly.

[1385] 6. Action Collection:

[1386] Device: The device again records the user's actions (e.g., purchasing the presented product, reacting to the displayed information).

[1387] 7. Feedback Analysis:

[1388] Server: Analyzes the received feedback data and improves the accuracy of the next information provided.

[1389] Specific examples

[1390] Example 1:

[1391] As users walk around the virtual store, their behavioral data (location, video, and audio data) is collected in real time. The server analyzes this data to identify the user's interests. For example, if a user spends a long time looking at a particular group of products, the generative AI model generates and provides detailed information about those products and suggestions for related products. In this case, the generative AI model generates prompts such as "You might be interested in this product" and displays them on the smart glasses.

[1392] Example 2:

[1393] When a user is walking through a food section, the server predicts the ingredients the user will need based on their behavioral data and past purchase history, and displays a prompt such as "We recommend this ingredient."

[1394] Example prompt sentence:

[1395] "This product might interest you"

[1396] "I recommend this ingredient."

[1397] "This section has the product you're looking for"

[1398] In this way, the system of the present invention individually optimizes the shopping experience in the virtual store, making it more convenient for the user.

[1399] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1400] Step 1:

[1401] Data collection

[1402] The device starts collecting the user's location information in real time using GPS, where the input is the user's real-time location information, which is obtained from the GPS sensor.

[1403] At the same time, the camera is used to capture video data and the microphone is used to record audio data. These data are temporarily stored in the device. The input is the video data from the camera and the audio data from the microphone, and these are collected.

[1404] Step 2:

[1405] Data transmission

[1406] The terminal transmits the data to the server after a certain time has passed or when the amount of collected data reaches a certain amount. The input is the data collected in step 1, and the output is the data sent to the server.

[1407] Step 3:

[1408] Data analysis

[1409] The server analyzes the received data. First, it plots the GPS data on a map to visualize the user's location fluctuations. The input is location data, and the output is a visualization of the location.

[1410] Next, the video data is analyzed using an image processing library (e.g., OpenCV) to identify the scene and the user's interests. The input is the video data, and the output is the analyzed scene information.

[1411] The audio data is converted to text using a machine learning library (e.g., TensorFlow). The input is the audio data, and the output is the converted audio data.

[1412] Step 4:

[1413] information generation

[1414] The server generates prompt sentences using a generative AI model based on the analysis results from step 3. For example, if the user is interested in a specific product category, it generates prompt sentences for related products. The input is the analysis results, and the output is the generated prompt sentences.

[1415] Step 5:

[1416] Information provision

[1417] The terminal displays the prompt sentence received from the server in real time, for example, by displaying a notification on the display of smart glasses. The input is the generated prompt sentence, and the output is the notification displayed to the user.

[1418] Step 6:

[1419] Action Collection

[1420] The terminal again records the user's actions (e.g., purchasing a suggested product or reacting to a notification). The input is the user's action, and the output is the recorded action data.

[1421] Step 7:

[1422] Feedback Analysis

[1423] The server analyzes the action data and improves the accuracy of the next information provided. For example, if a particular prompt phrase is likely to lead to a user action, it learns that pattern. The input is the recorded action data, and the output is a new pattern that will be reflected in improving the accuracy of the next information provided.

[1424] In this way, a system can be realized that analyzes user behavior in detail and provides optimal information in real time.

[1425] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1426] This invention combines an emotion engine with a system that collects and analyzes user behavioral data to provide optimal information to users. The system aims to improve the quality of users' lives by collecting and analyzing user behavioral and emotional data and providing appropriate information based on this data.

[1427] Overall system configuration

[1428] The main components of the system are:

[1429] 1. Device: A smartphone or tablet used by a user in their daily life, used to collect behavioral and emotional data.

[1430] 2. Server: A central system that analyzes the collected data and provides information based on the results.

[1431] 3. Data collection methods: GPS, microphone, camera and emotion recognition sensors on the device.

[1432] 4. Data analysis means: Analysis algorithms and software in the server.

[1433] 5. Information provision means: Notification functions and alarms installed on the device.

[1434] 6. Action collection means: terminal applications and interfaces.

[1435] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[1436] Program processing

[1437] The components of the system and their specific operations are described below.

[1438] 1. Data Collection:

[1439] Device: GPS is used to collect location information, a microphone is used to record audio data, and a camera is used to capture facial expressions and photographs. In addition, an emotion engine recognizes the user's emotions from audio and camera data. For example, when a user is taking a walk, location information is collected along with the emotions (relaxed, stressed, etc.) at the time.

[1440] Users: Just use a smartphone or tablet as you normally would, with no special operations required.

[1441] 2. Data transmission:

[1442] Terminal: Periodically transmits collected behavioral and emotional data to the server.

[1443] User: Continues with daily life without performing any special operations.

[1444] 3. Data Receipt and Storage:

[1445] Server: Receives data sent from the device and stores it in a database. Collected data includes location information, voice data, photo data, and emotion data.

[1446] 4. Speech and emotion data transcription and recognition:

[1447] Server: Converts the voice data into text using a speech recognition algorithm and recognizes the user's emotions using an emotion engine. For example, emotion tags are assigned such as "happy," "sad," and "stressed."

[1448] 5. Data Analysis:

[1449] Server: Plots location data on a map and analyzes user behavior patterns. At the same time, converts voice data into text and analyzes emotion data to identify specific activity and emotional trends. For example, it analyzes whether a user feels stressed when in a specific location.

[1450] 6. Information generation:

[1451] Server: Generates appropriate information for the user based on the analysis results, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[1452] 7. Information Transmission:

[1453] Server: Sends server-generated information to the user's device, including notifications and reminders.

[1454] 8. Information Receipt and Notification Display:

[1455] Device: Displays the received information to the user and provides notifications or alarms as needed. For example, the user might see a message saying "Relax now, keep it up."

[1456] 9. Action Collection:

[1457] On the device: If the user takes action based on the notification, collect that action again. For example, record the action taken based on the reminder (e.g., take medicine, take a deep breath).

[1458] 10. Feedback Data Submission:

[1459] Device: Sends collected feedback data to the server, including data based on user behavior.

[1460] 11. Feedback data analysis and model updating:

[1461] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, it identifies the best time to relax from the feedback data and suggests relaxing music for the next time.

[1462] Specific examples

[1463] Example 1: Mr. B (65 years old) uses it daily:

[1464] Data collection and transmission: The emotion engine recognizes that Mr. B takes a walk every day at 1 p.m. and feels relaxed during that time. The device transmits this data to the server.

[1465] Data analysis: The server analyzes the location information and recognizes that you are walking in a specific park at 1 PM. The emotion engine detects your state of relaxation.

[1466] Information generation and provision: The server generates information about relaxation (e.g., a relaxation guide) and sends it to Mr. B's terminal.

[1467] Information presentation and action: Person B checks the information and takes action according to the relaxation guide.

[1468] Feedback collection and transmission: The actions taken by User B following the guide are recorded and the data is sent to the server.

[1469] Feedback analysis: The server analyzes this feedback and uses it for next time.

[1470] In this way, the system of the present invention can combine the user's behavioral data and emotional data to provide optimal information and improve the quality of life.

[1471] The processing flow will be explained below.

[1472] Step 1:

[1473] Data collection

[1474] Device: To record the user's daily activities, the device uses a GPS sensor to collect location information, a microphone to record voice data, and a camera to capture photos and facial expression data. In addition, an emotion engine recognizes the user's emotions from voice and facial expressions in real time.

[1475] User: The person uses a smartphone or tablet on a daily basis and data is collected without any special action.

[1476] Step 2:

[1477] Data transmission

[1478] Terminal: The collected location information, voice data, photo data, and emotion data are sent to the server at regular intervals or when a certain amount of data is reached.

[1479] Step 3:

[1480] Data reception and storage

[1481] Server: Receives data sent from the device and stores it in a database, including location information, audio data, photo data, and emotion data.

[1482] Step 4:

[1483] Converting audio data to text

[1484] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the voice data "My walk this morning was pleasant" is converted into text.

[1485] Step 5:

[1486] emotion recognition

[1487] Server: Uses an emotion engine to recognize the user's emotions from voice data and photo data (mainly facial expressions). For example, it assigns emotion tags such as "relaxed," "stressed," and "happy" based on voice tone and facial expressions.

[1488] Step 6:

[1489] Data analysis

[1490] Server: Plots the collected location information on a map and analyzes the user's behavioral patterns. At the same time, it analyzes the converted voice data and recognized emotion data to understand trends in specific activities and emotions. For example, it analyzes whether the user always relaxes in a specific place (a park).

[1491] Step 7:

[1492] information generation

[1493] Server: Generates appropriate information for the user based on all the analysis results. For example, if the user is in the park at 1pm every day and the data shows that they are relaxing, it generates a message that provides them with tips on how to relax.

[1494] Step 8:

[1495] Information transmission

[1496] Server: Generates information messages and notifications and sends them to the user's device. The notification contents include reminders and emotional advice.

[1497] Step 9:

[1498] Receiving information and displaying notifications

[1499] Device: The device displays received information and notifications on the device screen and alerts the user with sound or vibration as needed. For example, a notification saying "Take deep breaths for relaxation" may appear on the screen.

[1500] Step 10:

[1501] Action Collection

[1502] Terminal: Collects feedback data based on user behavior. When a user follows a relaxation guide, their behavior is recorded. For example, data is collected by pressing a button that says "I took a deep breath."

[1503] Step 11:

[1504] Sending feedback data

[1505] Device: Sends collected feedback data to the server.

[1506] User: Follows instructions to take action and provides feedback information as a result.

[1507] Step 12:

[1508] Analysis of feedback data

[1509] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, if it turns out that a user finds it easier to relax at a certain time of day, it will suggest relaxation methods that are appropriate for that time of day from the next time.

[1510] In this way, this system can improve the quality of life of users by comprehensively analyzing their behavioral and emotional data and providing optimal information tailored to their individual needs.

[1511] Example 2

[1512] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1513] Conventional information provision systems provide information based solely on user behavior data, making it impossible to perform detailed analysis of the user's emotional state or behavioral patterns, resulting in the problem that the information provided does not adequately meet the user's needs.In addition, there is a lack of a mechanism for integrating and analyzing emotional data in addition to voice data and location information, making it difficult to effectively provide the most appropriate information to the user.

[1514] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user behavioral data and emotional data, means for analyzing the collected data, and means for providing optimal information to the user based on the analysis results. This makes it possible to combine and analyze the user's behavioral data and emotional data, and to provide more personalized information. Please take your time to complete the information you want and add some extra features!

[1515] "Behavioral data" refers to information about various actions that a user takes in their daily life, and includes, for example, location information, movement history, information about applications used, and types of activities.

[1516] "Emotion data" refers to information about the user's emotional state, and includes, for example, data representing emotions such as "relaxed," "stressed," "joyed," and "sad" obtained by analyzing voice tone and facial expressions.

[1517] "Data collection means" refers to the devices and processes that collect behavioral and emotional data using sensors and applications built into the device.

[1518] "Data analysis means" refers to algorithms or software that process collected behavioral and emotional data and analyze the user's behavioral patterns and emotional state.

[1519] "Information provision means" refers to a process or device that generates optimal information for the user based on analyzed data and provides it to the terminal as a notification or alarm.

[1520] "Action collection means" refers to a device or process for recording and recollecting actions taken by users based on notifications.

[1521] "Feedback data" refers to data regarding the actions taken by users based on the information provided and the results of those actions, and is used to improve the accuracy of the next information provided.

[1522] An "emotion engine" refers to a software component that analyzes voice data, facial expression data, etc. to recognize the user's emotions.

[1523] "Reminder" refers to a notification or alarm sent to remind a user of a specific time or activity.

[1524] "Analysis results" refers to information obtained after analyzing collected data, including insights into the user's behavioral patterns and emotional state.

[1525] "Recommended information" refers to information or suggestions that are generated based on the analysis results and that correspond to the user's situation and needs.

[1526] This invention relates to a system that collects and analyzes user behavioral and emotional data and provides optimal information to users based on the results. The system aims to improve the user's quality of life and is implemented using devices such as smartphones and tablets.

[1527] Overall system configuration

[1528] This system is broadly composed of the following elements:

[1529] 1. Device: A smartphone or tablet that users use on a daily basis, used to collect behavioral and emotional data.

[1530] 2. Server: A central system that analyzes the collected data and provides information based on the analysis results.

[1531] 3. Data collection methods: GPS, microphone, camera, and emotion recognition sensors built into the device.

[1532] 4. Data analysis means: Analysis algorithms and software in the server.

[1533] 5. Information provision means: Notification functions and alarms installed on the device.

[1534] 6. Action collection means: terminal applications and interfaces.

[1535] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[1536] About program processing

[1537] Data collection

[1538] Device:

[1539] Location information is collected using GPS.

[1540] Record audio data using a microphone.

[1541] Facial expressions and photographic data are captured using a camera.

[1542] The emotion engine recognizes the user's emotions from voice data and camera data.

[1543] Example: When a user goes for a walk, their location information is collected along with their emotions (relaxed, stressed, etc.).

[1544] User:

[1545] No special operations are required; just use your smartphone or tablet as you normally would.

[1546] Data transmission

[1547] Device:

[1548] The collected behavioral data and emotional data are periodically sent to a server.

[1549] User:

[1550] No special operations are required while going about your daily life.

[1551] Data reception and storage

[1552] server:

[1553] The data sent from the terminal is received and stored in a database.

[1554] Data analysis and information provision

[1555] Text conversion and recognition of voice and emotion data

[1556] server:

[1557] It uses a speech recognition algorithm to convert voice data into text, and an emotion engine to recognize the user's emotions, tagging them as "happy," "sad," or "stressed," for example.

[1558] Data analysis

[1559] server:

[1560] The system plots location data on a map and analyzes user behavior patterns. At the same time, it analyzes specific activity and emotional trends based on the text content of the voice data and emotional data. For example, it analyzes whether a user feels stressed when in a specific location.

[1561] information generation

[1562] server:

[1563] Based on the analysis results, appropriate information is generated for the user, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[1564] Information transmission

[1565] server:

[1566] Sends server-generated information to the user's device, including notifications and reminders.

[1567] Receiving information and displaying notifications

[1568] Device:

[1569] It displays the received information to the user and provides notifications or alarms as needed, for example, it displays a message to the user saying "You're relaxing now, keep it up."

[1570] Action collection and feedback analysis

[1571] Action Collection

[1572] Device:

[1573] If the user takes action based on the notification, that action is collected again, for example, recording the action taken based on the reminder (e.g., taking medicine, taking a deep breath, etc.).

[1574] Sending feedback data

[1575] Device:

[1576] The collected feedback data is sent to a server.

[1577] Analysis of feedback data and model updating

[1578] server:

[1579] The feedback data is analyzed to optimize the content and timing of the next information provision. For example, the feedback data can be used to understand the time of day when you want to relax, and the next time, relaxing music can be suggested.

[1580] Examples of concrete examples and prompts

[1581] Examples:

[1582] Example 1:

[1583] The emotion engine recognizes that the user takes a walk every day at 1 p.m. and feels relaxed during the walk. The device sends this data to the server.

[1584] The server analyzes the location information and recognizes that you are taking a walk in a particular park at 1 p.m., and the emotion engine detects your relaxed state.

[1585] The server generates information about relaxation (for example, a relaxation guide) and transmits it to the user's terminal.

[1586] The user reviews the information and takes action according to the relaxation guide.

[1587] The user's actions following the guide are recorded and the data is sent to a server.

[1588] The server analyzes this feedback and uses it for next time.

[1589] Example prompt sentence:

[1590] "Based on the fact that Person B relaxes in the park every day at 1 p.m., please suggest what kind of relaxation guide we should provide for Person B."

[1591] "If a user is feeling stressed, what kind of encouraging message would be effective to send them?"

[1592] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1593] Step 1: Data collection

[1594] Device:

[1595] Input: The smartphone or tablet that users use every day.

[1596] Specific operation:

[1597] Obtain location information using GPS.

[1598] Record audio data using a microphone.

[1599] Facial expressions and photographic data are captured using a camera.

[1600] The emotion engine recognizes the user's emotions (e.g., relaxed, stressed, etc.) from voice and camera data.

[1601] Data processing: Organize location information, audio data, camera data, and emotion recognition results and temporarily store them on the device.

[1602] Output: Organized behavioral and emotional data.

[1603] Step 2: Send data

[1604] Device:

[1605] Input: Organized behavioral and emotional data.

[1606] Specific operation:

[1607] The data is packetized and sent to the server at regular intervals.

[1608] Output: The data packets sent.

[1609] Step 3: Receiving and storing data

[1610] server:

[1611] Input: Data packets sent from the terminal.

[1612] Specific operation:

[1613] The data packets are received at the receiving port and decrypted.

[1614] Store the received data in a database.

[1615] Data processing: When storing data in a database, location information, audio data, photo data, and emotion data are stored separately in separate fields.

[1616] Output: Behavioral and emotional data stored in a database.

[1617] Step 4: Converting speech and emotion data into text and recognizing it

[1618] server:

[1619] Input: Audio data stored in a database.

[1620] Specific operation:

[1621] Convert the audio data into text using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API).

[1622] The text data is input into an emotion engine, which analyzes the user's emotions (e.g., "happy," "sad," "stressed").

[1623] Data processing: Convert the audio data into text and assign it an emotion tag.

[1624] Output: Translated speech data and emotion tags.

[1625] Step 5: Data analysis

[1626] server:

[1627] Input: location data, transcribed audio data, emotion tag.

[1628] Specific operation:

[1629] Location data is plotted on a map and behavioral patterns are analyzed.

[1630] Analyze specific activity and emotional trends based on text and emotion data.

[1631] For example, it analyzes whether a user feels stressed when in a particular place.

[1632] Data calculations: Statistical analysis of collected data to extract behavioral patterns and emotional trends.

[1633] Output: Analysis of behavioral patterns and emotional tendencies.

[1634] Step 6: Information Generation

[1635] server:

[1636] Input: Analysis results.

[1637] Specific operation:

[1638] Based on the analysis results, appropriate information is generated for the user.

[1639] For example, providing a relaxation guide when the user is relaxing.

[1640] Data processing: Generate appropriate notification messages and reminders based on the analysis results.

[1641] Output: The generated notification message or reminder.

[1642] Step 7: Send information

[1643] server:

[1644] Input: The generated notification message or reminder.

[1645] Specific operation:

[1646] The generated message is sent to the user's terminal.

[1647] Output: Notification message sent to the terminal.

[1648] Step 8: Receiving information and displaying notifications

[1649] Device:

[1650] Input: Notification messages and reminders sent from the server.

[1651] Specific operation:

[1652] The received information is displayed to the user.

[1653] Sound notifications and alarms as needed.

[1654] Output: Messages and reminders displayed on the user's device.

[1655] Step 9: Action Collection

[1656] Device:

[1657] Input: The action the user takes based on the notification.

[1658] Specific operation:

[1659] Record user actions within the device.

[1660] For example, record actions such as taking medicine or taking deep breaths based on reminders.

[1661] Output: Recorded user action data.

[1662] Step 10: Submitting feedback data

[1663] Device:

[1664] Input: Recorded user action data.

[1665] Specific operation:

[1666] Send the feedback data to the server.

[1667] Output: The feedback data sent.

[1668] Step 11: Analyze feedback data and update the model

[1669] server:

[1670] Input: Feedback data submitted by the user.

[1671] Specific operation:

[1672] Analyze feedback data to optimize the content and timing of the next information provision.

[1673] Generative AI models learn from feedback data to maintain a more accurate information delivery system.

[1674] Output: Updated AI model and optimized information delivery system.

[1675] (Application example 2)

[1676] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1677] Conventional information provision systems were limited to providing optimal information based on user behavior data, but no systems took the user's emotions into consideration. This made it difficult to provide appropriate information to alleviate the psychological challenges and stress that users actually experience. Furthermore, there were insufficient means to utilize user feedback on given information to improve the accuracy of future information provision.

[1678] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1679] In this invention, the server includes means for collecting and analyzing user behavioral data and emotional data, means for providing optimal content based on the analysis results, and means for collecting user actions based on the provided content. This makes it possible to provide optimal content that takes into account the user's emotional state, thereby reducing the user's psychological burden and improving their quality of life. Furthermore, analyzing the feedback data can improve the accuracy of future content provision.

[1680] "Behavioral data" refers to data related to the user's daily behavior, such as the user's movement history, application usage history, and interaction history.

[1681] "Emotional data" is data that indicates the emotional state of a user, obtained by analyzing the user's tone of voice, facial expressions, text messages, and the like.

[1682] "Content" includes information and entertainment elements provided to users, such as videos, music, articles, and guides.

[1683] "Location information" is data related to a user's current location or movement history, and is obtained using GPS, Wi-Fi information, etc.

[1684] "Analysis means" refers to algorithms or software that analyze collected data and infer user behavioral patterns and emotional states.

[1685] "Recommended content" is content that is provided in a form that is optimal for the user's emotional state and behavioral patterns based on the analysis results.

[1686] "Feedback data" refers to data that indicates the user's reaction and actions to the provided content.

[1687] This invention relates to a system that collects user behavioral data and emotional data, analyzes the data, and provides optimal content to the user. The main components of this system are a terminal that collects user behavioral data and emotional data, a server that analyzes the collected data, means for providing content based on the analysis results, means for collecting user reactions, data analysis means, and an emotion engine.

[1688] Overall system overview

[1689] 1. Device:

[1690] The main collection method is the smartphone used by users in their daily lives, and behavioral and emotional data is collected using GPS, microphones, cameras, and emotion recognition sensors. This allows users to provide data in a natural way without any special operations.

[1691] 2. Server:

[1692] The collected data is received and stored in a database. The server is equipped with a speech recognition algorithm that converts voice data into text, and an emotion analysis algorithm that uses an emotion engine. These functions enable detailed analysis of the user's behavioral patterns and emotional state.

[1693] 3. Emotion Engine:

[1694] This software component analyzes the user's emotions from voice and camera data. For example, if the user is feeling stressed, it can detect their stress level from their voice tone and facial expression.

[1695] 4. Data analysis methods:

[1696] The system plots location and behavioral data on a map and analyzes the user's behavioral patterns. From the analysis results, it identifies the times and places where the user feels relaxed and the situations in which they feel stressed.

[1697] 5. Information provision method:

[1698] The server generates content (e.g., relaxation music, stress relief guides) and sends it to the user's device, notifying them with notifications and alarms. For example, if the server determines that the user is relaxed, it provides music to help them maintain that state.

[1699] 6. How we collect user actions:

[1700] The device records the user's actions based on the provided content (e.g., following a relaxation guide) and sends them back to the server. This allows the system to receive user feedback and analyze the data to improve the accuracy of the information provided.

[1701] Specific Examples

[1702] example:

[1703] If the emotion engine detects through analysis that the user is feeling stressed at work, the server generates relaxation music and a guide video for stress relief and provides it to the user's smartphone. The results of the user's actions in response are collected and used to make the next recommendation.

[1704] Example prompt sentence:

[1705] "I want to build a system that provides recommended content when a user is feeling stressed. Please tell me how to analyze voice and location data and recommend relaxation guides and music based on the user's emotions."

[1706] This system uses TensorFlow / Keras to build an emotion analysis model and collects data using the smartphone's GPS, microphone, and camera. It also uses Amazon RDS and Google Firebase as databases, generating and providing appropriate content based on the analysis results. This enables highly accurate information provision based on the user's emotional state and behavior.

[1707] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1708] Step 1:

[1709] Data collection

[1710] The device (smartphone) collects the user's behavioral data (location information, application history) and emotional data (voice tone, facial expressions). The device's sensor data (GPS, microphone, camera) is used as input, and the behavioral data and emotional data are obtained as output. This data is temporarily stored within the device.

[1711] Step 2:

[1712] Data transmission

[1713] The terminal transmits the collected behavioral data and emotional data to the server. The input is the collected behavioral data and emotional data, and the output is the data transmitted to the server. The data is transmitted periodically.

[1714] Step 3:

[1715] Data reception and storage

[1716] The server receives behavioral and emotional data sent from the device. It uses the data sent as input and stores it in a database as output. It uses database systems such as Amazon RDS and Google Firebase.

[1717] Step 4:

[1718] Speech to text conversion and sentiment analysis

[1719] The server converts the received voice data into text using a speech recognition algorithm (e.g., Google Speech-to-Text) and analyzes emotions using an emotion analysis engine (e.g., TensorFlow / Keras). The input is voice data, and the output is text data and emotion tags (e.g., happy, sad, stressed).

[1720] Step 5:

[1721] Data analysis

[1722] The server plots the behavioral data on a map and analyzes the behavioral patterns. At the same time, it analyzes the user's specific activities and emotional tendencies from the textualized voice data and emotional data. Location data and textualized voice data are used as input, and the output is the user's behavioral patterns and emotional tendencies.

[1723] Step 6:

[1724] information generation

[1725] The server generates content (e.g., relaxation guides, music to relieve stress, etc.) to recommend to users based on the analysis results. The input is behavioral patterns and emotional tendency information, and the output is the generated content.

[1726] Step 7:

[1727] Information transmission

[1728] The server sends the generated content to the user's terminal. The input is the generated content, and the output is the content sent to the terminal.

[1729] Step 8:

[1730] Receiving information and displaying notifications

[1731] The terminal receives the content sent from the server and displays it to the user using a notification or alarm function. The input is the content sent from the server, and the output is the notification content displayed on the user's screen.

[1732] Step 9:

[1733] Action Collection

[1734] The terminal records the results of the user's actions based on the notification. The input is the user's actions, and the output is the action record data.

[1735] Step 10:

[1736] Sending feedback data

[1737] The terminal sends the collected action data to the server. The input is the collected action data, and the output is the feedback data sent to the server.

[1738] Step 11:

[1739] Analysis of feedback data and model updating

[1740] The server analyzes the feedback data and updates the model to optimize the content and timing of the next information provision. The input is the feedback data, and the output is the updated analytical model and information provision algorithm.

[1741] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1742] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1743] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1744] [Fourth embodiment]

[1745] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1746] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1747] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1748] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1749] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1750] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1751] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1752] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1753] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1754] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1755] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1756] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1757] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1758] The present invention relates to a system for collecting user behavior data, analyzing the data, and providing optimal information to the user. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[1759] Overall system configuration

[1760] The main components of the system are:

[1761] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[1762] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[1763] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[1764] 4. Data analysis means: Analysis algorithms and software installed on the server.

[1765] 5. Information provision means: Notification functions and alarms installed on the device.

[1766] 6. Action collection means: Software that allows the terminal to recollect user actions.

[1767] Program processing

[1768] Below, each element of the system and its specific operation will be explained.

[1769] 1. Data Collection:

[1770] Device: The user's device collects real-time location information using GPS, records audio data using a microphone, and captures photos using a camera. These data are temporarily stored on the device. For example, when a user goes for a walk every day, the route and time of the walk are recorded.

[1771] 2. Data transmission:

[1772] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[1773] 3. Data Analysis:

[1774] Server: The server analyzes the received data. It plots GPS data on a map, converts voice data into text, and tags photos with AI. For example, it analyzes a daily activity log to determine when, where, and what the user is doing.

[1775] 4. Information generation:

[1776] Server: Based on the analysis results, messages are generated to provide the user with the necessary information in a timely manner. For example, if a user misses a regular dose of medication, a notification is generated to remind them to take it.

[1777] 5. Information provision:

[1778] Terminal: The information sent from the server is displayed on the terminal. Notifications are displayed and alarms sound if necessary. The user can act based on this information. For example, taking medicine after seeing the notification.

[1779] 6. Action Collection:

[1780] Device: The device records the user's actions (if they take their medicine based on the notification, that action) again. When the user presses the "take medicine" button, the data is sent to the server again.

[1781] 7. Feedback Analysis:

[1782] Server: Analyzes the received feedback data and improves the accuracy of the next notification. For example, it learns medication habits and optimizes the next notification time.

[1783] Specific examples

[1784] Example 1: Mr. A (70 years old) sets his schedule to take a walk at 7:00 a.m. every day and take his medicine at 8:00 a.m. In this case, the following series of processes will be performed.

[1785] Data collection: Mr. A's device collects GPS data and takes voice notes at the start of his walk at 7:00 AM.

[1786] Data transmission: After the walk ends, the data is transmitted to the server just before 8:00 AM.

[1787] Data analysis: The server analyzes the GPS data to determine the end time of Mr. A's walk and analyzes the time of taking his medicine from the voice memo.

[1788] Information generation: The server generates a notification message saying "It's time to take your medicine" just before 8:00 AM.

[1789] Information provided: The notification appears on the device and Person A checks it.

[1790] Action collection: Person A takes medicine and presses the "take" button on the device.

[1791] Feedback analysis: The server receives dose timing data and improves the accuracy of the next notification.

[1792] In this way, the system of the present invention can provide timely information to elderly people without them having to search for the information themselves, thereby improving their quality of life.

[1793] The processing flow will be explained below.

[1794] Step 1:

[1795] Data collection

[1796] Device: Your device periodically collects location information using GPS, records audio data with its built-in microphone, and temporarily stores photos captured by its camera.

[1797] User: Data is collected from users as they go about their daily lives without them having to take any special action.

[1798] Step 2:

[1799] Data transmission

[1800] Device: When a certain amount of collected data is reached or a certain time has passed, the device sends this data in batches to the server.

[1801] Step 3:

[1802] Data reception and storage

[1803] Server: Receives data sent from the device and stores it in a database, including location information, audio data, and photo data.

[1804] Step 4:

[1805] Converting audio data to text

[1806] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the speech "I ate breakfast" is converted into text "I ate breakfast."

[1807] Step 5:

[1808] Data analysis

[1809] Server: Plots the received location information on a map and analyzes the user's daily movements. It also analyzes the converted voice data to identify activities and priorities.

[1810] User: The system automatically analyzes the data without any special awareness.

[1811] Step 6:

[1812] information generation

[1813] Server: Generates necessary information for the user based on the analysis results. For example, if the pattern of "taking a walk at 7:00 AM and then taking medicine at 8:00 AM" is analyzed, a notification urging the user to take their medicine is generated.

[1814] Step 7:

[1815] Information transmission

[1816] Server: Sends the generated information to the user's device. This message contains notifications and reminders.

[1817] Step 8:

[1818] Receiving information and displaying notifications

[1819] Device: The received information is displayed on the device screen and, if necessary, the user is notified using push notifications or alarm functions.

[1820] User: Check the notification on the device and take the necessary action (e.g., take medicine).

[1821] Step 9:

[1822] Action Collection

[1823] On the device: If the user takes action based on the notification, provide a feedback form or button to record that action. When the user presses the "take a dose" button, that information is collected.

[1824] Step 10:

[1825] Sending feedback data

[1826] Device: Send the collected feedback data back to the server.

[1827] User: Follow the instructions on the device to enter feedback information.

[1828] Step 11:

[1829] Analysis of feedback data

[1830] Server: Analyzes the received feedback data and optimizes the timing and content of the next information provision. For example, data is added to fine-tune the time at which the user takes their medicine.

[1831] By repeating these steps, the system can continuously learn the user's behavioral patterns and provide the most appropriate information in a timely manner.

[1832] Example 1

[1833] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1834] The problem to be solved by this invention is to effectively collect and analyze user behavior data and provide optimal information to the user based on the analysis results. Furthermore, there is a need to collect user actions based on the provided information and analyze this as feedback to improve the accuracy of the next information provided. There is also a need to provide a system that can provide more accurate user support by integrating various data sources, such as location information and voice data.

[1835] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1836] In this invention, the server includes means for collecting user behavioral data, means for saving the collected behavioral data and transmitting it under specific conditions, means for analyzing the transmitted data, means for generating information optimized for the user based on the analysis results, means for providing the generated information to the user, means for collecting user actions based on the provided information, and means for analyzing the collected action data and improving the accuracy of the next information provision. This makes it possible to effectively collect and analyze user behavioral data and provide individually optimized information. Furthermore, it is possible to provide feedback on the user's reactions and improve the accuracy of the next information provision based on that data.

[1837] "Behavioral data" is information related to the user's actions and behavior in daily life, and includes location information, voice data, photograph data, and the like.

[1838] "Location Information" means a user's geographic coordinate data obtained using GPS or other location-determining technology.

[1839] "Collection means" refers to the technologies and devices used to collect user behavioral data, and primarily refers to sensors such as GPS, microphones, and cameras.

[1840] "Analysis means" refers to the techniques and algorithms used to analyze collected data, and are used to extract data features and recognize patterns.

[1841] "Information generation means" refers to techniques and scripts for generating information to be provided to users based on the analysis results.

[1842] "Providing means" refers to the technology or device used to transmit the generated information to the user, such as a notification function or alarm.

[1843] "Action Data" is a record of when a user acts based on the information provided.

[1844] "Feedback analysis means" refers to technologies and algorithms that analyze collected action data and improve the accuracy of the next information provided.

[1845] MODE FOR CARRYING OUT THE INVENTION

[1846] The present invention relates to a system that collects user behavior data, analyzes the data, and provides optimal information to the user. The system is composed of the following elements:

[1847] Overall system configuration

[1848] The main components of the system are:

[1849] 1. Device: A smartphone or tablet that users carry around with them on a daily basis and is used to collect behavioral data.

[1850] 2. Server: This is the central system that analyzes the behavioral data and provides information based on the results.

[1851] 3. Data collection means: Sensors such as GPS, microphone, and camera equipped on the device.

[1852] 4. Data analysis means: Analysis algorithms and software installed on the server.

[1853] 5. Information provision means: Notification functions and alarms installed on the device.

[1854] 6. Action collection means: Software that allows the terminal to recollect user actions.

[1855] Data collection

[1856] Device: The smartphone or tablet that a user carries around collects behavioral data using built-in sensors such as GPS, microphone, and camera. Location information is acquired in real time by the GPS sensor, audio data is recorded through the microphone, and photo data is taken with the camera. This data is temporarily stored on the device.

[1857] Data transmission

[1858] Device: The collected data is sent to the server after a certain period of time has passed or when a certain amount of data has been collected. Wi-Fi or mobile data is used as the communication method.

[1859] Data analysis

[1860] Server: The server analyzes the data it receives. GPS data is plotted on a map, voice data is converted to text, and photo data is tagged with AI to identify user behavioral patterns and interests.

[1861] information generation

[1862] Server: Generates necessary information for the user based on the analysis results. For example, if a user misses a medication time, it generates a reminder message.

[1863] Information provision

[1864] Device: Provides the user with the information sent from the server. The notification is displayed on the device screen and, if necessary, an audio alarm is set. The user can then take action.

[1865] Action Collection

[1866] Device: After providing the information, the device collects the user's actual actions. For example, after a notification to take medicine is displayed, pressing the "take medicine" button records the action.

[1867] Feedback Analysis

[1868] Server: Analyzes user action data and optimizes the timing and content of the next information provided, thereby providing the most appropriate information for the user.

[1869] Specific examples

[1870] Example 1:

[1871] Mr. A (70 years old) schedules his daily walk at 7:00 AM and his medication at 8:00 AM. The following process takes place:

[1872] Data collection: When Mr. A starts his walk at 7:00 AM, his device acquires GPS data and also records audio data.

[1873] Data transmission: After the walk is completed, the collected data is sent to the server.

[1874] Data analysis: The server analyzes the walking route and time from GPS data, and confirms the time to take medicine from voice data.

[1875] Information generation: The server generates a notification just before 8:00 AM saying "It's time to take your medicine."

[1876] Information provided: A notification appears on the device and Person A checks it.

[1877] Action collection: After taking the medicine, Person A presses the "take button."

[1878] Feedback analysis: The server optimizes the next notification time based on the action data.

[1879] In this way, the system of the invention can provide users with timely and relevant information to improve their quality of life.

[1880] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1881] System program processing flow

[1882] Step 1:

[1883] Data collection

[1884] Device: The user's device uses the built-in GPS to obtain real-time location information, uses a microphone to record ambient sounds and voice memos, and also uses a camera to take photos at regular intervals.

[1885] What it does: The device's GPS sensor periodically logs the current location, the microphone sensor periodically records audio, and the camera also periodically takes pictures and saves them.

[1886] Input: User's real-time location, ambient sounds, and photos.

[1887] Output: Location data, audio data, and photo data temporarily stored on the device.

[1888] Step 2:

[1889] Data transmission

[1890] Device: After a certain amount of time has passed or a certain amount of collected data has been collected, the device will send the data to the server via Wi-Fi or mobile data.

[1891] Specific operation: The device uses an internal timer and data volume monitoring function to connect to the server and upload data when conditions are met.

[1892] Input: Location data, audio data, and photo data stored on the device.

[1893] Output: Location data, audio data, and photo data sent to the server.

[1894] Step 3:

[1895] Data analysis

[1896] Server: The server analyzes the data it receives: GPS data is plotted on a map, voice data is converted to text, and photo data is tagged using AI.

[1897] What it does: The server uses a map API to display location data on a map, uses voice recognition software to convert voice data to text, and uses image analysis algorithms to tag photos.

[1898] Input: Location data, audio data, and photo data sent to the server.

[1899] Output: Parsed map data, text data, tagged image data.

[1900] Step 4:

[1901] information generation

[1902] Server: Based on the analysis results, the server generates information to provide to the user, for example, a reminder if the user is about to miss a dose of medication.

[1903] What happens: The server evaluates certain conditions based on the parsed data and executes scripts that automatically generate notification messages if those conditions are met.

[1904] Input: Parsed map data, text data, tagged image data.

[1905] Output: A notification message to the user.

[1906] Step 5:

[1907] Information provision

[1908] Device: To provide the user with the information sent from the server, the device will display a notification on the screen and sound an alarm if necessary.

[1909] Specific behavior: The device's notification system receives the notification message and displays it to the user as a popup. It also invokes the function to sound an audio alarm at the specified time.

[1910] Input: The notification message sent by the server.

[1911] Output: Notifications displayed on the device screen, alarms sounding.

[1912] Step 6:

[1913] Action Collection

[1914] Device: If the user acts on the notification, the action is collected again. For example, if the user presses the "take a pill" button, the action data is recorded.

[1915] What it does: The device interface detects user actions, logs the information, and uploads it to a server.

[1916] Input: User action (e.g., pressing the dose button).

[1917] Output: Action data sent from the device to the server.

[1918] Step 7:

[1919] Feedback Analysis

[1920] Server: The server analyzes the action data it receives and optimizes the timing and content of the next information provision.

[1921] What it does: The server's analysis algorithm evaluates your action data and automatically adjusts the timing and content of your next notification based on the results.

[1922] Input: Action data sent from the device.

[1923] Output: Optimized next notification settings.

[1924] (Application example 1)

[1925] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1926] Conventional information provision systems were able to collect and analyze user behavior data, but were insufficient for providing customized information in real time. Furthermore, they lacked the technology to simultaneously analyze user behavior patterns and interests and make highly accurate recommendations. Furthermore, they were unable to flexibly generate information using generative AI models, making it difficult to provide information tailored to user needs.

[1927] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1928] In this invention, the server includes means for collecting user behavior data, means for analyzing the collected behavior data, means for providing optimal information to the user based on the analysis results, means for collecting user actions based on the provided information, means for analyzing the collected action data and improving the accuracy of the next information provided, means for simultaneously collecting and analyzing audio data, video data, and location information, and means for providing customized product information to the user in real time based on the generated analysis results. This makes it possible to analyze user behavior with high accuracy and provide optimal information in real time.

[1929] "Behavioral data" refers to various types of information related to the user's daily activities, such as location information, audio data, and video data.

[1930] "Analysis" refers to the processing of collected data to identify user behavior patterns and interests.

[1931] "Optimal information" refers to useful information that is individually customized based on the user's behavior and interests.

[1932] "Action data" refers to data regarding actions and operations performed by a user based on the information provided.

[1933] A "generative AI model" is an artificial intelligence model used to generate optimal information for each user in real time based on collected data.

[1934] A "prompt" is text containing instructions or suggestions that a generative AI model generates to provide information.

[1935] "Customized product information" is information about products selected according to the user's behavioral patterns and interests.

[1936] "Providing in real time" refers to providing information instantly in conjunction with the user's current actions.

[1937] This invention is a system that collects user behavior data, analyzes the data, and provides optimal information to the user in real time. The system includes a terminal that collects user behavior data, a server that analyzes the collected data, and multiple means for providing information to the user based on the analysis results.

[1938] Overall system configuration

[1939] The main components of the system are as follows:

[1940] 1. Device: Smart glasses or smartphones that users use on a daily basis, which collect behavioral data (location information, video data, audio data).

[1941] 2. Server: This is the central system that analyzes behavioral data and provides optimal information to users based on the analysis results.

[1942] 3. Data collection means: Sensors such as GPS, camera, and microphone equipped on the device.

[1943] 4. Data analysis means: Analysis algorithms and generative AI models installed on the server.

[1944] 5. Information provision means: Notification functions and alarms installed on the device.

[1945] 6. Action collection means: Software that allows the terminal to collect user actions.

[1946] 7. Generative AI model: An artificial intelligence model that generates prompts in real time based on user behavior data.

[1947] Program processing

[1948] 1. Data Collection:

[1949] Device: The user's smart glasses or smartphone collects real-time location information using GPS, captures video data with a camera, and records audio data with a microphone. These data are temporarily stored on the device.

[1950] 2. Data transmission:

[1951] Terminal: After a certain time has passed or when a certain amount of data has been received, the terminal sends the data to the server.

[1952] 3. Data Analysis:

[1953] Server: The server analyzes the received data. It plots GPS data on a map, analyzes video data using an image processing library (e.g., OpenCV), and converts audio data to text using a machine learning library (e.g., TensorFlow).

[1954] 4. Information generation:

[1955] Server: Based on the analysis results, the server uses a generative AI model to generate information to be provided to the user in the form of prompt sentences. For example, it generates prompt sentences that display information about products that the user is interested in in real time.

[1956] 5. Information provision:

[1957] Terminal: The prompt sent from the server is displayed on the terminal, notifications are displayed and alarms are sounded if necessary, and the user can act accordingly.

[1958] 6. Action Collection:

[1959] Device: The device again records the user's actions (e.g., purchasing the presented product, reacting to the displayed information).

[1960] 7. Feedback Analysis:

[1961] Server: Analyzes the received feedback data and improves the accuracy of the next information provided.

[1962] Specific examples

[1963] Example 1:

[1964] As users walk around the virtual store, their behavioral data (location, video, and audio data) is collected in real time. The server analyzes this data to identify the user's interests. For example, if a user spends a long time looking at a particular group of products, the generative AI model generates and provides detailed information about those products and suggestions for related products. In this case, the generative AI model generates prompts such as "You might be interested in this product" and displays them on the smart glasses.

[1965] Example 2:

[1966] When a user is walking through a food section, the server predicts the ingredients the user will need based on their behavioral data and past purchase history, and displays a prompt such as "We recommend this ingredient."

[1967] Example prompt sentence:

[1968] "This product might interest you"

[1969] "I recommend this ingredient."

[1970] "This section has the product you're looking for"

[1971] In this way, the system of the present invention individually optimizes the shopping experience in the virtual store, making it more convenient for the user.

[1972] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1973] Step 1:

[1974] Data collection

[1975] The device starts collecting the user's location information in real time using GPS, where the input is the user's real-time location information, which is obtained from the GPS sensor.

[1976] At the same time, the camera is used to capture video data and the microphone is used to record audio data. These data are temporarily stored in the device. The input is the video data from the camera and the audio data from the microphone, and these are collected.

[1977] Step 2:

[1978] Data transmission

[1979] The terminal transmits the data to the server after a certain time has passed or when the amount of collected data reaches a certain amount. The input is the data collected in step 1, and the output is the data sent to the server.

[1980] Step 3:

[1981] Data analysis

[1982] The server analyzes the received data. First, it plots the GPS data on a map to visualize the user's location fluctuations. The input is location data, and the output is a visualization of the location.

[1983] Next, the video data is analyzed using an image processing library (e.g., OpenCV) to identify the scene and the user's interests. The input is the video data, and the output is the analyzed scene information.

[1984] The audio data is converted to text using a machine learning library (e.g., TensorFlow). The input is the audio data, and the output is the converted audio data.

[1985] Step 4:

[1986] information generation

[1987] The server generates prompt sentences using a generative AI model based on the analysis results from step 3. For example, if the user is interested in a specific product category, it generates prompt sentences for related products. The input is the analysis results, and the output is the generated prompt sentences.

[1988] Step 5:

[1989] Information provision

[1990] The terminal displays the prompt sentence received from the server in real time, for example, by displaying a notification on the display of smart glasses. The input is the generated prompt sentence, and the output is the notification displayed to the user.

[1991] Step 6:

[1992] Action Collection

[1993] The terminal again records the user's actions (e.g., purchasing a suggested product or reacting to a notification). The input is the user's action, and the output is the recorded action data.

[1994] Step 7:

[1995] Feedback Analysis

[1996] The server analyzes the action data and improves the accuracy of the next information provided. For example, if a particular prompt phrase is likely to lead to a user action, it learns that pattern. The input is the recorded action data, and the output is a new pattern that will be reflected in improving the accuracy of the next information provided.

[1997] In this way, a system can be realized that analyzes user behavior in detail and provides optimal information in real time.

[1998] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1999] This invention combines an emotion engine with a system that collects and analyzes user behavioral data to provide optimal information to users. The system aims to improve the quality of users' lives by collecting and analyzing user behavioral and emotional data and providing appropriate information based on this data.

[2000] Overall system configuration

[2001] The main components of the system are:

[2002] 1. Device: A smartphone or tablet used by a user in their daily life, used to collect behavioral and emotional data.

[2003] 2. Server: A central system that analyzes the collected data and provides information based on the results.

[2004] 3. Data collection methods: GPS, microphone, camera and emotion recognition sensors on the device.

[2005] 4. Data analysis means: Analysis algorithms and software in the server.

[2006] 5. Information provision means: Notification functions and alarms installed on the device.

[2007] 6. Action collection means: terminal applications and interfaces.

[2008] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[2009] Program processing

[2010] The components of the system and their specific operations are described below.

[2011] 1. Data Collection:

[2012] Device: GPS is used to collect location information, a microphone is used to record audio data, and a camera is used to capture facial expressions and photographs. In addition, an emotion engine recognizes the user's emotions from audio and camera data. For example, when a user is taking a walk, location information is collected along with the emotions (relaxed, stressed, etc.) at the time.

[2013] Users: Just use a smartphone or tablet as you normally would, with no special operations required.

[2014] 2. Data transmission:

[2015] Terminal: Periodically transmits collected behavioral and emotional data to the server.

[2016] User: Continues with daily life without performing any special operations.

[2017] 3. Data Receipt and Storage:

[2018] Server: Receives data sent from the device and stores it in a database. Collected data includes location information, voice data, photo data, and emotion data.

[2019] 4. Speech and emotion data transcription and recognition:

[2020] Server: Converts the voice data into text using a speech recognition algorithm and recognizes the user's emotions using an emotion engine. For example, emotion tags are assigned such as "happy," "sad," and "stressed."

[2021] 5. Data Analysis:

[2022] Server: Plots location data on a map and analyzes user behavior patterns. At the same time, converts voice data into text and analyzes emotion data to identify specific activity and emotional trends. For example, it analyzes whether a user feels stressed when in a specific location.

[2023] 6. Information generation:

[2024] Server: Generates appropriate information for the user based on the analysis results, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[2025] 7. Information Transmission:

[2026] Server: Sends server-generated information to the user's device, including notifications and reminders.

[2027] 8. Information Receipt and Notification Display:

[2028] Device: Displays the received information to the user and provides notifications or alarms as needed. For example, the user might see a message saying "Relax now, keep it up."

[2029] 9. Action Collection:

[2030] On the device: If the user takes action based on the notification, collect that action again. For example, record the action taken based on the reminder (e.g., take medicine, take a deep breath).

[2031] 10. Feedback Data Submission:

[2032] Device: Sends collected feedback data to the server, including data based on user behavior.

[2033] 11. Feedback data analysis and model updating:

[2034] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, it identifies the best time to relax from the feedback data and suggests relaxing music for the next time.

[2035] Specific examples

[2036] Example 1: Mr. B (65 years old) uses it daily:

[2037] Data collection and transmission: The emotion engine recognizes that Mr. B takes a walk every day at 1 p.m. and feels relaxed during that time. The device transmits this data to the server.

[2038] Data analysis: The server analyzes the location information and recognizes that you are walking in a specific park at 1 PM. The emotion engine detects your state of relaxation.

[2039] Information generation and provision: The server generates information about relaxation (e.g., a relaxation guide) and sends it to Mr. B's terminal.

[2040] Information presentation and action: Person B checks the information and takes action according to the relaxation guide.

[2041] Feedback collection and transmission: The actions taken by User B following the guide are recorded and the data is sent to the server.

[2042] Feedback analysis: The server analyzes this feedback and uses it for next time.

[2043] In this way, the system of the present invention can combine the user's behavioral data and emotional data to provide optimal information and improve the quality of life.

[2044] The processing flow will be explained below.

[2045] Step 1:

[2046] Data collection

[2047] Device: To record the user's daily activities, the device uses a GPS sensor to collect location information, a microphone to record voice data, and a camera to capture photos and facial expression data. In addition, an emotion engine recognizes the user's emotions from voice and facial expressions in real time.

[2048] User: The person uses a smartphone or tablet on a daily basis and data is collected without any special action.

[2049] Step 2:

[2050] Data transmission

[2051] Terminal: The collected location information, voice data, photo data, and emotion data are sent to the server at regular intervals or when a certain amount of data is reached.

[2052] Step 3:

[2053] Data reception and storage

[2054] Server: Receives data sent from the device and stores it in a database, including location information, audio data, photo data, and emotion data.

[2055] Step 4:

[2056] Converting audio data to text

[2057] Server: Converts the received voice data into text using a speech recognition algorithm. For example, the voice data "My walk this morning was pleasant" is converted into text.

[2058] Step 5:

[2059] emotion recognition

[2060] Server: Uses an emotion engine to recognize the user's emotions from voice data and photo data (mainly facial expressions). For example, it assigns emotion tags such as "relaxed," "stressed," and "happy" based on voice tone and facial expressions.

[2061] Step 6:

[2062] Data analysis

[2063] Server: Plots the collected location information on a map and analyzes the user's behavioral patterns. At the same time, it analyzes the converted voice data and recognized emotion data to understand trends in specific activities and emotions. For example, it analyzes whether the user always relaxes in a specific place (a park).

[2064] Step 7:

[2065] information generation

[2066] Server: Generates appropriate information for the user based on all the analysis results. For example, if the user is in the park at 1pm every day and the data shows that they are relaxing, it generates a message that provides them with tips on how to relax.

[2067] Step 8:

[2068] Information transmission

[2069] Server: Generates information messages and notifications and sends them to the user's device. The notification contents include reminders and emotional advice.

[2070] Step 9:

[2071] Receiving information and displaying notifications

[2072] Device: The device displays received information and notifications on the device screen and alerts the user with sound or vibration as needed. For example, a notification saying "Take deep breaths for relaxation" may appear on the screen.

[2073] Step 10:

[2074] Action Collection

[2075] Terminal: Collects feedback data based on user behavior. When a user follows a relaxation guide, their behavior is recorded. For example, data is collected by pressing a button that says "I took a deep breath."

[2076] Step 11:

[2077] Sending feedback data

[2078] Device: Sends collected feedback data to the server.

[2079] User: Follows instructions to take action and provides feedback information as a result.

[2080] Step 12:

[2081] Analysis of feedback data

[2082] Server: Analyzes the feedback data and optimizes the content and timing of the next information provision. For example, if it turns out that a user finds it easier to relax at a certain time of day, it will suggest relaxation methods that are appropriate for that time of day from the next time.

[2083] In this way, this system can improve the quality of life of users by comprehensively analyzing their behavioral and emotional data and providing optimal information tailored to their individual needs.

[2084] Example 2

[2085] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2086] Conventional information provision systems provide information based solely on user behavior data, making it impossible to perform detailed analysis of the user's emotional state or behavioral patterns, resulting in the problem that the information provided does not adequately meet the user's needs.In addition, there is a lack of a mechanism for integrating and analyzing emotional data in addition to voice data and location information, making it difficult to effectively provide the most appropriate information to the user.

[2087] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user behavioral data and emotional data, means for analyzing the collected data, and means for providing optimal information to the user based on the analysis results. This makes it possible to combine and analyze the user's behavioral data and emotional data, and to provide more personalized information. Please take your time to complete the information you want and add some extra features!

[2088] "Behavioral data" refers to information about various actions that a user takes in their daily life, and includes, for example, location information, movement history, information about applications used, and types of activities.

[2089] "Emotion data" refers to information about the user's emotional state, and includes, for example, data representing emotions such as "relaxed," "stressed," "joyed," and "sad" obtained by analyzing voice tone and facial expressions.

[2090] "Data collection means" refers to the devices and processes that collect behavioral and emotional data using sensors and applications built into the device.

[2091] "Data analysis means" refers to algorithms or software that process collected behavioral and emotional data and analyze the user's behavioral patterns and emotional state.

[2092] "Information provision means" refers to a process or device that generates optimal information for the user based on analyzed data and provides it to the terminal as a notification or alarm.

[2093] "Action collection means" refers to a device or process for recording and recollecting actions taken by users based on notifications.

[2094] "Feedback data" refers to data regarding the actions taken by users based on the information provided and the results of those actions, and is used to improve the accuracy of the next information provided.

[2095] An "emotion engine" refers to a software component that analyzes voice data, facial expression data, etc. to recognize the user's emotions.

[2096] "Reminder" refers to a notification or alarm sent to remind a user of a specific time or activity.

[2097] "Analysis results" refers to information obtained after analyzing collected data, including insights into the user's behavioral patterns and emotional state.

[2098] "Recommended information" refers to information or suggestions that are generated based on the analysis results and that correspond to the user's situation and needs.

[2099] This invention relates to a system that collects and analyzes user behavioral and emotional data and provides optimal information to users based on the results. The system aims to improve the user's quality of life and is implemented using devices such as smartphones and tablets.

[2100] Overall system configuration

[2101] This system is broadly composed of the following elements:

[2102] 1. Device: A smartphone or tablet that users use on a daily basis, used to collect behavioral and emotional data.

[2103] 2. Server: A central system that analyzes the collected data and provides information based on the analysis results.

[2104] 3. Data collection methods: GPS, microphone, camera, and emotion recognition sensors built into the device.

[2105] 4. Data analysis means: Analysis algorithms and software in the server.

[2106] 5. Information provision means: Notification functions and alarms installed on the device.

[2107] 6. Action collection means: terminal applications and interfaces.

[2108] 7. Emotion engine: A software component that analyzes the user's emotions based on voice data and facial expression data.

[2109] About program processing

[2110] Data collection

[2111] Device:

[2112] Location information is collected using GPS.

[2113] Record audio data using a microphone.

[2114] Facial expressions and photographic data are captured using a camera.

[2115] The emotion engine recognizes the user's emotions from voice data and camera data.

[2116] Example: When a user goes for a walk, their location information is collected along with their emotions (relaxed, stressed, etc.).

[2117] User:

[2118] No special operations are required; just use your smartphone or tablet as you normally would.

[2119] Data transmission

[2120] Device:

[2121] The collected behavioral data and emotional data are periodically sent to a server.

[2122] User:

[2123] No special operations are required while going about your daily life.

[2124] Data reception and storage

[2125] server:

[2126] The data sent from the terminal is received and stored in a database.

[2127] Data analysis and information provision

[2128] Text conversion and recognition of voice and emotion data

[2129] server:

[2130] It uses a speech recognition algorithm to convert voice data into text, and an emotion engine to recognize the user's emotions, tagging them as "happy," "sad," or "stressed," for example.

[2131] Data analysis

[2132] server:

[2133] The system plots location data on a map and analyzes user behavior patterns. At the same time, it analyzes specific activity and emotional trends based on the text content of the voice data and emotional data. For example, it analyzes whether a user feels stressed when in a specific location.

[2134] information generation

[2135] server:

[2136] Based on the analysis results, appropriate information is generated for the user, for example, providing a relaxation guide when the user is relaxed, or generating an encouraging message when the user is feeling stressed.

[2137] Information transmission

[2138] server:

[2139] Sends server-generated information to the user's device, including notifications and reminders.

[2140] Receiving information and displaying notifications

[2141] Device:

[2142] It displays the received information to the user and provides notifications or alarms as needed, for example, it displays a message to the user saying "You're relaxing now, keep it up."

[2143] Action collection and feedback analysis

[2144] Action Collection

[2145] Device:

[2146] If the user takes action based on the notification, that action is collected again, for example, recording the action taken based on the reminder (e.g., taking medicine, taking a deep breath, etc.).

[2147] Sending feedback data

[2148] Device:

[2149] The collected feedback data is sent to a server.

[2150] Analysis of feedback data and model updating

[2151] server:

[2152] The feedback data is analyzed to optimize the content and timing of the next information provision. For example, the feedback data can be used to understand the time of day when you want to relax, and the next time, relaxing music can be suggested.

[2153] Examples of concrete examples and prompts

[2154] Examples:

[2155] Example 1:

[2156] The emotion engine recognizes that the user takes a walk every day at 1 p.m. and feels relaxed during the walk. The device sends this data to the server.

[2157] The server analyzes the location information and recognizes that you are taking a walk in a particular park at 1 p.m., and the emotion engine detects your relaxed state.

[2158] The server generates information about relaxation (for example, a relaxation guide) and transmits it to the user's terminal.

[2159] The user reviews the information and takes action according to the relaxation guide.

[2160] The user's actions following the guide are recorded and the data is sent to a server.

[2161] The server analyzes this feedback and uses it for next time.

[2162] Example prompt sentence:

[2163] "Based on the fact that Person B relaxes in the park every day at 1 p.m., please suggest what kind of relaxation guide we should provide for Person B."

[2164] "If a user is feeling stressed, what kind of encouraging message would be effective to send them?"

[2165] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2166] Step 1: Data collection

[2167] Device:

[2168] Input: The smartphone or tablet that users use every day.

[2169] Specific operation:

[2170] Obtain location information using GPS.

[2171] Record audio data using a microphone.

[2172] Facial expressions and photographic data are captured using a camera.

[2173] The emotion engine recognizes the user's emotions (e.g., relaxed, stressed, etc.) from voice and camera data.

[2174] Data processing: Organize location information, audio data, camera data, and emotion recognition results and temporarily store them on the device.

[2175] Output: Organized behavioral and emotional data.

[2176] Step 2: Send data

[2177] Device:

[2178] Input: Organized behavioral and emotional data.

[2179] Specific operation:

[2180] The data is packetized and sent to the server at regular intervals.

[2181] Output: The data packets sent.

[2182] Step 3: Receiving and storing data

[2183] server:

[2184] Input: Data packets sent from the terminal.

[2185] Specific operation:

[2186] The data packets are received at the receiving port and decrypted.

[2187] Store the received data in a database.

[2188] Data processing: When storing data in a database, location information, audio data, photo data, and emotion data are stored separately in separate fields.

[2189] Output: Behavioral and emotional data stored in a database.

[2190] Step 4: Converting speech and emotion data into text and recognizing it

[2191] server:

[2192] Input: Audio data stored in a database.

[2193] Specific operation:

[2194] Convert the audio data into text using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API).

[2195] The text data is input into an emotion engine, which analyzes the user's emotions (e.g., "happy," "sad," "stressed").

[2196] Data processing: Convert the audio data into text and assign it an emotion tag.

[2197] Output: Translated speech data and emotion tags.

[2198] Step 5: Data analysis

[2199] server:

[2200] Input: location data, transcribed audio data, emotion tag.

[2201] Specific operation:

[2202] Location data is plotted on a map and behavioral patterns are analyzed.

[2203] Analyze specific activity and emotional trends based on text and emotion data.

[2204] For example, it analyzes whether a user feels stressed when in a particular place.

[2205] Data calculations: Statistical analysis of collected data to extract behavioral patterns and emotional trends.

[2206] Output: Analysis of behavioral patterns and emotional tendencies.

[2207] Step 6: Information Generation

[2208] server:

[2209] Input: Analysis results.

[2210] Specific operation:

[2211] Based on the analysis results, appropriate information is generated for the user.

[2212] For example, providing a relaxation guide when the user is relaxing.

[2213] Data processing: Generate appropriate notification messages and reminders based on the analysis results.

[2214] Output: The generated notification message or reminder.

[2215] Step 7: Send information

[2216] server:

[2217] Input: The generated notification message or reminder.

[2218] Specific operation:

[2219] The generated message is sent to the user's terminal.

[2220] Output: Notification message sent to the terminal.

[2221] Step 8: Receiving information and displaying notifications

[2222] Device:

[2223] Input: Notification messages and reminders sent from the server.

[2224] Specific operation:

[2225] The received information is displayed to the user.

[2226] Sound notifications and alarms as needed.

[2227] Output: Messages and reminders displayed on the user's device.

[2228] Step 9: Action Collection

[2229] Device:

[2230] Input: The action the user takes based on the notification.

[2231] Specific operation:

[2232] Record user actions within the device.

[2233] For example, record actions such as taking medicine or taking deep breaths based on reminders.

[2234] Output: Recorded user action data.

[2235] Step 10: Submitting feedback data

[2236] Device:

[2237] Input: Recorded user action data.

[2238] Specific operation:

[2239] Send the feedback data to the server.

[2240] Output: The feedback data sent.

[2241] Step 11: Analyze feedback data and update the model

[2242] server:

[2243] Input: Feedback data submitted by the user.

[2244] Specific operation:

[2245] Analyze feedback data to optimize the content and timing of the next information provision.

[2246] Generative AI models learn from feedback data to maintain a more accurate information delivery system.

[2247] Output: Updated AI model and optimized information delivery system.

[2248] (Application example 2)

[2249] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2250] Conventional information provision systems were limited to providing optimal information based on user behavior data, but no systems took the user's emotions into consideration. This made it difficult to provide appropriate information to alleviate the psychological challenges and stress that users actually experience. Furthermore, there were insufficient means to utilize user feedback on given information to improve the accuracy of future information provision.

[2251] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2252] In this invention, the server includes means for collecting and analyzing user behavioral data and emotional data, means for providing optimal content based on the analysis results, and means for collecting user actions based on the provided content. This makes it possible to provide optimal content that takes into account the user's emotional state, thereby reducing the user's psychological burden and improving their quality of life. Furthermore, analyzing the feedback data can improve the accuracy of future content provision.

[2253] "Behavioral data" refers to data related to the user's daily behavior, such as the user's movement history, application usage history, and interaction history.

[2254] "Emotional data" is data that indicates the emotional state of a user, obtained by analyzing the user's tone of voice, facial expressions, text messages, and the like.

[2255] "Content" includes information and entertainment elements provided to users, such as videos, music, articles, and guides.

[2256] "Location information" is data related to a user's current location or movement history, and is obtained using GPS, Wi-Fi information, etc.

[2257] "Analysis means" refers to algorithms or software that analyze collected data and infer user behavioral patterns and emotional states.

[2258] "Recommended content" is content that is provided in a form that is optimal for the user's emotional state and behavioral patterns based on the analysis results.

[2259] "Feedback data" refers to data that indicates the user's reaction and actions to the provided content.

[2260] This invention relates to a system that collects user behavioral data and emotional data, analyzes the data, and provides optimal content to the user. The main components of this system are a terminal that collects user behavioral data and emotional data, a server that analyzes the collected data, means for providing content based on the analysis results, means for collecting user reactions, data analysis means, and an emotion engine.

[2261] Overall system overview

[2262] 1. Device:

[2263] The main collection method is the smartphone used by users in their daily lives, and behavioral and emotional data is collected using GPS, microphones, cameras, and emotion recognition sensors. This allows users to provide data in a natural way without any special operations.

[2264] 2. Server:

[2265] The collected data is received and stored in a database. The server is equipped with a speech recognition algorithm that converts voice data into text, and an emotion analysis algorithm that uses an emotion engine. These functions enable detailed analysis of the user's behavioral patterns and emotional state.

[2266] 3. Emotion Engine:

[2267] This software component analyzes the user's emotions from voice and camera data. For example, if the user is feeling stressed, it can detect their stress level from their voice tone and facial expression.

[2268] 4. Data analysis methods:

[2269] The system plots location and behavioral data on a map and analyzes the user's behavioral patterns. From the analysis results, it identifies the times and places where the user feels relaxed and the situations in which they feel stressed.

[2270] 5. Information provision method:

[2271] The server generates content (e.g., relaxation music, stress relief guides) and sends it to the user's device, notifying them with notifications and alarms. For example, if the server determines that the user is relaxed, it provides music to help them maintain that state.

[2272] 6. How we collect user actions:

[2273] The device records the user's actions based on the provided content (e.g., following a relaxation guide) and sends them back to the server. This allows the system to receive user feedback and analyze the data to improve the accuracy of the information provided.

[2274] Specific Examples

[2275] example:

[2276] If the emotion engine detects through analysis that the user is feeling stressed at work, the server generates relaxation music and a guide video for stress relief and provides it to the user's smartphone. The results of the user's actions in response are collected and used to make the next recommendation.

[2277] Example prompt sentence:

[2278] "I want to build a system that provides recommended content when a user is feeling stressed. Please tell me how to analyze voice and location data and recommend relaxation guides and music based on the user's emotions."

[2279] This system uses TensorFlow / Keras to build an emotion analysis model and collects data using the smartphone's GPS, microphone, and camera. It also uses Amazon RDS and Google Firebase as databases, generating and providing appropriate content based on the analysis results. This enables highly accurate information provision based on the user's emotional state and behavior.

[2280] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2281] Step 1:

[2282] Data collection

[2283] The device (smartphone) collects the user's behavioral data (location information, application history) and emotional data (voice tone, facial expressions). The device's sensor data (GPS, microphone, camera) is used as input, and the behavioral data and emotional data are obtained as output. This data is temporarily stored within the device.

[2284] Step 2:

[2285] Data transmission

[2286] The terminal transmits the collected behavioral data and emotional data to the server. The input is the collected behavioral data and emotional data, and the output is the data transmitted to the server. The data is transmitted periodically.

[2287] Step 3:

[2288] Data reception and storage

[2289] The server receives behavioral and emotional data sent from the device. It uses the data sent as input and stores it in a database as output. It uses database systems such as Amazon RDS and Google Firebase.

[2290] Step 4:

[2291] Speech to text conversion and sentiment analysis

[2292] The server converts the received voice data into text using a speech recognition algorithm (e.g., Google Speech-to-Text) and analyzes emotions using an emotion analysis engine (e.g., TensorFlow / Keras). The input is voice data, and the output is text data and emotion tags (e.g., happy, sad, stressed).

[2293] Step 5:

[2294] Data analysis

[2295] The server plots the behavioral data on a map and analyzes the behavioral patterns. At the same time, it analyzes the user's specific activities and emotional tendencies from the textualized voice data and emotional data. Location data and textualized voice data are used as input, and the output is the user's behavioral patterns and emotional tendencies.

[2296] Step 6:

[2297] information generation

[2298] The server generates content (e.g., relaxation guides, music to relieve stress, etc.) to recommend to users based on the analysis results. The input is behavioral patterns and emotional tendency information, and the output is the generated content.

[2299] Step 7:

[2300] Information transmission

[2301] The server sends the generated content to the user's terminal. The input is the generated content, and the output is the content sent to the terminal.

[2302] Step 8:

[2303] Receiving information and displaying notifications

[2304] The terminal receives the content sent from the server and displays it to the user using a notification or alarm function. The input is the content sent from the server, and the output is the notification content displayed on the user's screen.

[2305] Step 9:

[2306] Action Collection

[2307] The terminal records the results of the user's actions based on the notification. The input is the user's actions, and the output is the action record data.

[2308] Step 10:

[2309] Sending feedback data

[2310] The terminal sends the collected action data to the server. The input is the collected action data, and the output is the feedback data sent to the server.

[2311] Step 11:

[2312] Analysis of feedback data and model updating

[2313] The server analyzes the feedback data and updates the model to optimize the content and timing of the next information provision. The input is the feedback data, and the output is the updated analytical model and information provision algorithm.

[2314] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2315] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2316] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2317] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2318] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2319] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2320] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2321] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2322] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2323] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2324] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2325] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2326] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2327] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2328] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2329] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2330] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2331] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2332] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2333] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2334] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2335] The following is further disclosed regarding the above embodiment.

[2336] (Claim 1)

[2337] a means for collecting user behavior data;

[2338] a means for analyzing the collected behavioral data;

[2339] A means for providing optimal information to users based on the analysis results;

[2340] means for collecting user actions based on the provided information;

[2341] A means of analyzing the collected action data and improving the accuracy of the next information provided;

[2342] A system including:

[2343] (Claim 2)

[2344] means for collecting user location information;

[2345] A means for analyzing a user's daily behavioral patterns based on location information;

[2346] a means for generating and providing reminders and notifications to the user based on the behavioral patterns;

[2347] The system of claim 1 further comprising:

[2348] (Claim 3)

[2349] a means for collecting voice data and converting it into text;

[2350] A means of analyzing the converted voice data and inferring the user's intentions and necessary information;

[2351] a means for providing recommended information to a user based on the estimated required information;

[2352] The system of claim 1 further comprising:

[2353] "Example 1"

[2354] (Claim 1)

[2355] a means for collecting user behavior data;

[2356] means for storing collected behavioral data and transmitting it under certain conditions;

[2357] means for analyzing the transmitted data;

[2358] A means for generating optimal information for a user based on the analysis results;

[2359] means for providing the generated information to a user;

[2360] means for collecting user actions based on the provided information;

[2361] A means of analyzing the collected action data and improving the accuracy of the next information provided;

[2362] A system including:

[2363] (Claim 2)

[2364] means for collecting user location information;

[2365] A means for analyzing a user's daily behavioral patterns based on location information;

[2366] a means for generating and providing reminders and notifications to the user based on the behavioral patterns;

[2367] The system of claim 1 further comprising:

[2368] (Claim 3)

[2369] a means for collecting voice data and converting it into text;

[2370] A means of analyzing the converted voice data and inferring the user's intentions and necessary information;

[2371] a means for providing recommended information to a user based on the estimated required information;

[2372] The system of claim 1 further comprising:

[2373] "Application Example 1"

[2374] (Claim 1)

[2375] a means for collecting user behavior data;

[2376] a means for analyzing the collected behavioral data;

[2377] A means for providing optimal information to users based on the analysis results;

[2378] means for collecting user actions based on the provided information;

[2379] A means of analyzing the collected action data and improving the accuracy of the next information provided;

[2380] means for simultaneously collecting and analyzing audio data, video data, and location information;

[2381] A means for providing customized product information to users in real time based on the generated analysis results;

[2382] A system including:

[2383] (Claim 2)

[2384] means for collecting user location information;

[2385] A means for analyzing a user's daily behavioral patterns based on location informa...

Claims

1. a means for collecting user behavior data; a means for analyzing the collected behavioral data; A means for providing optimal information to users based on the analysis results; means for collecting user actions based on the provided information; A means of analyzing the collected action data and improving the accuracy of the next information provided; A system including:

2. means for collecting user location information; A means for analyzing a user's daily behavioral patterns based on location information; a means for generating and providing reminders and notifications to the user based on the behavioral patterns; The system of claim 1 further comprising:

3. a means for collecting voice data and converting it into text; A means of analyzing the converted voice data and inferring the user's intentions and necessary information; a means for providing recommended information to a user based on the estimated required information; The system of claim 1 further comprising:

Images