system

Abstract

Provide a system. 【Solution means】 Means for the user to input or automatically record their own actions, Means for sending the action data to the server, Means for the user to set a role model to aim for, Means for sending the role model information to the server, Means for generating the next action to be taken using an AI generated based on the action data and role model information, Means for notifying the user of the generated action proposal, Means for recording the action result executed by the user again as a log and sending it to the server, Means for generating a new proposal based on the action result and sending feedback to the user, A system including the above.

Description

Technical Field

[0004] , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Many modern people have goals and role models to aim for in their daily lives, but there is a problem that it is difficult to plan and effectively execute specific actions. In particular, there is a lack of a system that can record one's own actions and propose appropriate actions to approach a role model based on that data, making it difficult to achieve goals. In such cases, if there is an effective support system, one can review one's own actions and efficiently progress towards the goal, but such a system did not exist in the prior art.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that includes means for a user to input or automatically record their actions, means for transmitting recorded action data to a server, means for setting a role model that the user aims to emulate, means for transmitting role model information to a server, means for generating the next action to be taken using a generated AI based on the action data and role model information, means for notifying the user of the generated action suggestions, means for recording the results of the user's actions again as logs and transmitting them to a server, and means for generating new suggestions based on the action results and sending feedback to the user.

[0006] Specifically, by including means of using GPS and sensors when collecting user behavior data, user behavior is accurately tracked. Furthermore, by including means of notifying users of action suggestions as time-based reminders, it helps users remember and perform the suggested actions. This enables users to plan their actions in line with their desired role models and effectively achieve their goals.

[0007] A "user" refers to an entity that uses this system to set their own activity logs and goals.

[0008] "Behavioral data" refers to data that includes information such as daily movements, shopping, exercise, and meals, which are entered or automatically recorded by the user.

[0009] A "server" refers to a computer system that receives behavioral data and role model information sent by users, analyzes this data, and generates the next course of action to take.

[0010] A "role model" refers to information that specifically defines the person or ideal state that a user aspires to be like.

[0011] "Generative AI" refers to artificial intelligence technology used to suggest the next course of action based on received behavioral data and role model information.

[0012] "Action suggestions" refer to the specific actions that the AI ​​generates for the user to take next.

[0013] "Feedback" refers to suggestions for improvement and next actions provided by the server based on the results of the user's actions.

[0014] "GPS" refers to a global positioning system used to obtain a user's location information.

[0015] A "sensor" refers to a device used to record user behavior. Specifically, this includes accelerometers and gyroscopes.

[0016] A "reminder" refers to a time-based notification designed to prompt a user to take a specific action. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9]Shows an emotion map to which a plurality of emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Modes for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The operation of a specific system will be described as an embodiment for carrying out the present invention.

[0039] This system collects users' activity logs and suggests their next course of action based on a pre-configured role model. It is implemented through the cooperation of the user, device, and server according to the following processing flow.

[0040] 1. Collection and transmission of user activity logs

[0041] Users enter or automatically record their activity logs.

[0042] Users record their daily activities, such as travel, shopping, exercise, and meals, using their smartphones or PCs. For example, they can automatically record running distance and time using their smartphone's accelerometer and GPS function as an exercise log. They can also manually enter meal details and shopping lists into the app.

[0043] The device sends behavioral data to the server.

[0044] The collected behavioral data is encrypted and then periodically sent to the server. This ensures the security of the data.

[0045] 2. Setting a role model

[0046] Users set role models

[0047] Users can specifically define their role model. For example, they can select a professional athlete or a successful business person and input their characteristics and goals.

[0048] The device sends role model information to the server.

[0049] The configured role model information is encrypted and sent to the server, just like the user's behavioral data.

[0050] 3. Generating the next course of action

[0051] The server analyzes behavioral data and role model information.

[0052] The server performs analysis based on the received behavioral data and role model information. In particular, it compares past behavioral patterns with current progress to generate the optimal actions for the user to move closer to their goals.

[0053] The server uses AI to generate action suggestions.

[0054] The generating AI considers behavioral data and role model information to generate specific next steps. For example, if it determines that adding weight training will bring the user closer to their goal, it will create a suggestion such as "Add 30 minutes of weight training to your next workout."

[0055] The server sends the proposal to the user's terminal.

[0056] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[0057] 4. Suggestions for Users

[0058] The device will notify you of the suggested content.

[0059] The user's device will notify them of the suggestions using its notification function. This ensures that the user does not forget what action they should take next.

[0060] 5. Re-logging of feedback and action results

[0061] The user performs an action and records the result.

[0062] The user performs the suggested action and enters the results into the app. For example, if they perform the suggested weight training, they will record the duration and details of the workout again.

[0063] The device sends the results of its actions to the server.

[0064] The collected behavioral data is sent back to the server and used for further analysis.

[0065] The server generates new suggestions and sends feedback.

[0066] The server performs a re-analysis based on the newly collected behavioral data and generates feedback and suggestions for the next action for the user. This is then sent to the user's terminal for notification.

[0067] Specific example

[0068] Example 1: If you aim to become a professional athlete

[0069] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[0070] Example 2: When balancing work and family life

[0071] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[0072] As described above, this system supports users in approaching their desired role models by providing action suggestions and feedback using generated AI based on user behavior data and goal settings.

[0073] The following describes the processing flow.

[0074] Step 1:

[0075] Users enter or automatically record their activity logs.

[0076] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. Alternatively, the system can automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[0077] Step 2:

[0078] The device sends behavioral data to the server.

[0079] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent at night when connected to Wi-Fi to minimize battery consumption.

[0080] Step 3:

[0081] Users set role models

[0082] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[0083] Step 4:

[0084] The device sends role model information to the server.

[0085] The configured role model information is encrypted before being sent to the server.

[0086] Step 5:

[0087] The server analyzes behavioral data and role model information.

[0088] The server performs analysis based on the received behavioral data and role model information. This involves analyzing the user's past behavioral patterns and progress to determine what actions are necessary to move closer to the role model.

[0089] Step 6:

[0090] The server uses AI to generate action suggestions.

[0091] The generating AI uses analyzed behavioral data and role model information to suggest the optimal next action for the user. For example, it can generate specific suggestions such as, "Add an hour of running next weekend."

[0092] Step 7:

[0093] The server sends the proposal to the user's terminal.

[0094] The generated action suggestions are sent to the user's device, allowing the user to see the suggested actions.

[0095] Step 8:

[0096] The device will notify you of the suggested content.

[0097] The user's device will notify them of the suggestion using a reminder function. For example, a notification might appear saying, "Please go for a 30-minute run every morning at 6:00 AM."

[0098] Step 9:

[0099] The user performs an action and records the result.

[0100] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[0101] Step 10:

[0102] The device sends the results of its actions to the server.

[0103] The collected behavioral data is sent back to the server. This data is also encrypted before transmission and used for further analysis.

[0104] Step 11:

[0105] The server generates new suggestions and sends feedback.

[0106] The server re-analyzes the new behavioral data and generates feedback and suggestions for the next action for the user. For example, this may include positive feedback such as, "Your running pace is increasing, keep it up!"

[0107] Step 12:

[0108] The device will notify you of the feedback.

[0109] The server sends feedback to the user's device and notifies the user again. This allows the user to effectively continue taking action towards their goal.

[0110] (Example 1)

[0111] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0112] Conventional behavioral suggestion systems had problems such as not being able to fully utilize user behavior data and not being able to effectively provide optimal behavioral suggestions that corresponded to the user's goals. In particular, ensuring the security of user behavior data and improving the accuracy of behavioral suggestions were challenges.

[0113] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0114] In this invention, the server includes means for the user to input or automatically record their actions; means for encrypting the action data and transmitting it to the server; means for setting a role model that the user aims to emulate; means for encrypting the role model information and transmitting it to the server; means for generating the next action to be taken using a generated AI model based on the action data and role model information; means for notifying the user's terminal of the generated action suggestion; means for logging the results of the user's actions again, encrypting them, and transmitting them to the server; and means for generating new suggestions based on the action results and transmitting feedback to the user's terminal. This makes it possible to provide highly accurate and optimal action suggestions that match the user's goals while ensuring the security of the user's action data.

[0115] A "user" refers to an individual who uses this system to record behavioral data and aims to achieve their goals.

[0116] "Behavioral data" refers to information about daily activities such as movement, exercise, and eating that users record.

[0117] "Encryption" refers to technologies used to protect behavioral data and role model information from being deciphered by third parties.

[0118] A "server" refers to a computer system that receives and analyzes behavioral data and role model information sent by users, and generates and sends appropriate action suggestions.

[0119] A "role model" refers to an ideal individual or ideal figure that a user aspires to be like, whose characteristics and goals are set for them.

[0120] A "generative AI model" refers to an artificial intelligence model that generates suggestions for the next course of action based on user behavior data and role model information.

[0121] "Action suggestions" refer to specific action instructions generated by the server to support the user in achieving their goals.

[0122] "Feedback" refers to notifying the user of newly generated suggestions or evaluations based on the results of the actions they have taken.

[0123] A "sensor" refers to an electronic device used to acquire user movement and location information.

[0124] This invention is a system that allows users to record their own actions, generate action suggestions to help them move closer to their target role model, and support their implementation. This system operates through the coordinated efforts of the user, terminal, and server.

[0125] First, users record their daily activity logs using their smartphones or PCs. Specifically, they can automatically record running distance and time using their smartphone's accelerometer and GPS function. They can also manually input meal details and shopping lists into the app. The device encrypts the collected activity data and periodically sends it to the server using secure protocols such as HTTPS.

[0126] Next, users set their desired role model. In the dedicated app's settings screen, they can select a role model such as a "professional athlete" or a "successful business person," and input their characteristics and goals. This role model information, like behavioral data, is encrypted and sent to the server.

[0127] The behavioral data and role model information sent to the server are subject to analysis. Based on this information, the server uses a generative AI model to generate specific actions that the user should take next. For example, if the user aims to become a "professional athlete," the AI ​​will generate a specific suggestion such as "add 30 minutes of weight training to your next training session." This suggestion is sent from the server to the user's device and notified to the user. The notification can also be set as a time-based reminder.

[0128] The user receives action suggestions from their device, performs them, and records the results again in the app. For example, if the user performs a suggested weight training exercise, the user records the duration and content of the exercise again and sends it to the server via their device. This action result data is then analyzed again on the server and sent back to the user as new suggestions or feedback. This allows the user to continuously perform optimal actions.

[0129] Specific example

[0130] Example 1: If you aim to become a professional athlete

[0131] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[0132] Example 2: When balancing work and family life

[0133] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[0134] Example of a prompt

[0135] Activity data: "Distance traveled: 5km, Exercise time: 30 minutes, Meal content: Salad"

[0136] Role model: "Goal: Professional athlete, Characteristics: Daily training: 2 hours, Diet: High protein, low calorie"

[0137] Please suggest the next action this user should take.

[0138] Thus, the system of the present invention supports users in approaching their desired role model by providing action suggestions and feedback using a generated AI model based on the user's behavioral data and goal settings.

[0139] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0140] Step 1:

[0141] Users enter or automatically record their activity logs.

[0142] Users record their daily activities using a dedicated app on their smartphone or PC. Specifically, one method involves automatically recording running distance and time using the smartphone's accelerometer and GPS function. Users can also manually enter details of meals and shopping lists into the app.

[0143] Input: User behavior data (e.g., distance traveled, exercise time, diet)

[0144] Output: Recorded behavioral data

[0145] Specific operation: The smartphone's accelerometer detects the user's steps, and the GPS function measures the distance traveled. The user also inputs details of their meals through the app's UI.

[0146] Step 2:

[0147] The device encrypts behavioral data and sends it to the server.

[0148] The device encrypts user behavior data and periodically sends it to the server using secure protocols such as HTTPS. This process ensures data security.

[0149] Input: Recorded behavioral data

[0150] Output: Encrypted behavioral data

[0151] Specific operation: A data encryption module operates within the app, encrypting behavioral data using encryption algorithms such as AES. The data is then sent to the server using HTTPS.

[0152] Step 3:

[0153] Set a role model that the user aspires to be.

[0154] Users select a role model, such as a "professional athlete" or a "successful business person," from the settings screen of the dedicated app and input their characteristics and goals. This information, like behavioral data, is encrypted.

[0155] Input: User role model information (e.g., goals, characteristics)

[0156] Output: Configured role model information

[0157] Specific operation: The user enters role model information using dropdown lists or text boxes in the app's settings screen.

[0158] Step 4:

[0159] The device encrypts the role model information and sends it to the server.

[0160] As mentioned above, the device encrypts the role model information and sends it to the server using the HTTPS protocol.

[0161] Input: Configured role model information

[0162] Output: Encrypted role model information

[0163] Specific operation: The in-app data encryption module encrypts the role model information using an encryption algorithm and sends it to the server via HTTPS.

[0164] Step 5:

[0165] The server analyzes behavioral data and role model information.

[0166] The server uses a dedicated analysis algorithm to analyze the received behavioral data and role model information. This helps determine the optimal actions for the user to take to achieve their goals.

[0167] Input: Encrypted behavioral data, role model information

[0168] Output: Data analysis results

[0169] Specific operation: The analysis engine runs on the server and compares behavioral data with role model information. It also compares past behavioral patterns with current progress.

[0170] Step 6:

[0171] The server uses a generated AI model to create action suggestions.

[0172] The server uses a generated AI model to create specific action suggestions based on the data analysis results described above.

[0173] Input: Data analysis results

[0174] Output: Action Suggestions

[0175] Specific operation: The generating AI model generates optimal action suggestions for the user based on the results of data analysis (e.g., "Add 30 minutes of weight training to your next workout").

[0176] Step 7:

[0177] The server sends the proposal to the user's terminal.

[0178] The generated action suggestions are sent from the server to the user's terminal in the form of a notification.

[0179] Input: Action suggestion

[0180] Output: Action suggestions sent to the user's device

[0181] Specific operation: The server sends an action suggestion to the device, and the device displays it to the user as a notification. The notification can also be set as a time-based reminder.

[0182] Step 8:

[0183] The user performs an action and records the result.

[0184] The user performs the suggested action and records the result again in the app.

[0185] Input: Proposed action

[0186] Output: Results of the actions performed

[0187] Specific actions: The user performs the suggested actions and enters the results using a smartphone or PC app. This includes details such as exercise time and content.

[0188] Step 9:

[0189] The device encrypts the results of its actions and sends them to the server.

[0190] The device encrypts the collected behavioral data and sends it back to the server.

[0191] Input: Results of the actions performed

[0192] Output: Encrypted behavioral result data

[0193] Specific operation: The in-app data encryption module runs, encrypts the action result data, and sends it to the server.

[0194] Step 10:

[0195] The server generates new suggestions and sends feedback.

[0196] The server re-analyzes the newly collected behavioral data, generates new suggestions and feedback, and sends them to the user's device.

[0197] Input: Encrypted behavioral data

[0198] Output: New suggestions and feedback

[0199] Specific operation: The server performs a re-analysis and generates new action suggestions and feedback. The generated information is sent to the user's device in the form of a notification.

[0200] (Application Example 1)

[0201] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0202] The challenge lies in creating an operational support system that safely and efficiently manages the operation of autonomous vehicles while also considering the driver's health. Furthermore, it is necessary to achieve continuously optimized operational management by specifically proposing the driving behaviors that drivers should aim for and incorporating the results of their execution.

[0203] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0204] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a role model that the user aspires to; means for transmitting the role model information to the server; means for generating the next action to be taken using a generative AI based on the action data and role model information; means for notifying the user of the generated action suggestions; means for logging the results of the user's actions again and transmitting them to the server; means for generating new suggestions based on the action results and sending feedback to the user; means for automatically collecting operation data of the autonomous vehicle and driver action data, encrypting them, and transmitting them to the server; means for generating operation and driver action suggestions using a generative AI model based on the optimal operation pattern and driver model set by the administrator; and means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses. This makes it possible to perform advanced operation management and driver support for autonomous vehicles and improve the safety and efficiency of operations.

[0205] "Means for users to input or automatically record their own actions" refers to devices or software that allow users to manually input their own behavioral data or to automatically record behavioral data using a smartphone or sensors.

[0206] "Means for transmitting the aforementioned behavioral data to the server" refers to a device or software that securely transmits the collected and recorded behavioral data to the server via the internet.

[0207] "Means for setting a role model that the user aspires to" refers to a device or software that allows the user to specifically define the person or ideal they aspire to be and input that information.

[0208] "Means for transmitting the role model information to the server" refers to a device or software that securely transmits the configured role model information to the server via the internet.

[0209] "Means for generating the next action to be taken using a generative AI based on the aforementioned behavioral data and role model information" refers to an algorithm and system that analyzes the collected behavioral data and role model information and calculates the optimal next action using a generative AI model.

[0210] "Means for notifying the user of generated action suggestions" refers to a notification system for informing the user of action suggestions created by the generating AI, and is a device or software that uses a smartphone, smart glasses, in-vehicle display, etc.

[0211] "Means for logging the results of user actions and sending them back to the server" refers to a device or software that records the results of user actions and sends that data back to the server.

[0212] "Means for generating new suggestions based on the aforementioned action results and sending feedback to the user" refers to a device or software that re-analyzes the recorded action result data, generates new action suggestions, and notifies the user.

[0213] "Means for automatically collecting operational data of autonomous vehicles and driver behavior data, encrypting them, and then transmitting them to a server" refers to a device or software that collects various data related to the operation of autonomous vehicles and driver behavior data using sensors and cameras, encrypts them, and then transmits them to a server.

[0214] "Means for generating operational and driver action suggestions using a generation AI model based on optimal operational patterns and driver models set by the administrator" refers to a device or software that generates operational and driver action suggestions using a generation AI model based on operational pattern and driver model information set by the administrator.

[0215] "Means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses" refers to a device or software that notifies the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses.

[0216] This invention provides a system for advanced operational management and driver assistance of autonomous vehicles. This system allows users (operation managers and drivers) to collect their own behavioral logs and propose the next course of action based on their configured optimal operating patterns and driver models. This is achieved through the cooperation of a server, terminals, and autonomous vehicles, following the processing flow outlined below.

[0217] 1. Collection and transmission of user activity logs

[0218] Various sensors (cameras, accelerometers, GPS, etc.) within the autonomous vehicle are used as a means to automatically record user activity logs. These sensors record operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. The collected data is encrypted and then periodically transmitted to a server. This ensures the security of the data.

[0219] 2. Setting a role model

[0220] As a means for administrators to set role models, they can specifically define the optimal driving patterns and driver models they aim for. For example, this could include patterns that prioritize safe driving or patterns that emphasize efficiency. The set role model information is encrypted and sent to the server, just like user behavior data.

[0221] 3. Generating the next course of action

[0222] The server analyzes behavioral data and role model information. This analysis uses a generated AI model based on the behavioral data and role model information. The server compares past behavioral patterns with current progress and generates the optimal actions for the user to get closer to their goal. For example, if a driver hasn't taken a two-hour break, it will create a specific suggestion such as "Take a 15-minute break at the next rest stop."

[0223] 4. Suggestions for Users

[0224] The generated action suggestions are communicated to the vehicle's information display system or the driver's smart glasses. This allows the driver to properly understand and act on what to do next, even while driving.

[0225] 5. Re-logging of feedback and action results

[0226] There is a mechanism for automatically re-logging data as a means of recording the results when the user (driver) performs a suggested action. For example, in-vehicle sensors detect and record the driver's resting status. The collected action result data is sent back to the server and used for the next analysis. The server performs a re-analysis based on the newly collected action result data, generates new suggestions, and sends feedback.

[0227] Technology for realizing functionality

[0228] This system is constructed using the following technologies:

[0229] Hardware:

[0230] Various sensors installed inside the vehicle (camera, accelerometer, GPS, etc.)

[0231] Smart glasses for drivers (e.g., Google Glass®)

[0232] Information display system for autonomous vehicles

[0233] software:

[0234] Custom API for data collection and transmission (encryption supported)

[0235] Cloud tools (e.g., Google Cloud AutoML, AWS® SageMaker) are used for analyzing behavioral data and operational models.

[0236] Generative AI models include natural language generation models (e.g., OpenAI's GPT model).

[0237] Specific example

[0238] For example, the following prompt statements are used to input into the generated AI model.

[0239] Prompt message:

[0240] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[0241] Output of the generative AI model:

[0242] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[0243] As described above, the system for implementing the present invention enables advanced operational management and driver assistance for autonomous vehicles. This is expected to improve the safety and efficiency of operations.

[0244] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0245] Step 1:

[0246] The user records their activity log. Various sensors in the autonomous vehicle (cameras, accelerometers, GPS, etc.) collect operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. Operational data and driver status data are obtained as input data. This data is encrypted.

[0247] Output: Encrypted operational data and driver status data.

[0248] Step 2:

[0249] The terminal sends the encrypted data to the server. The data is securely transferred to the server using a transmission protocol (e.g., HTTPS).

[0250] Input: Encrypted operational data and driver status data.

[0251] Output: Encrypted data stored on the server.

[0252] Step 3:

[0253] The user (flight manager) sets the role model they aspire to. The manager uses a web portal or application to specify the optimal ferry pattern and driver model (safe driving, efficient driving, etc.). The configuration information is encrypted and sent to the server.

[0254] Input: Role model configuration information.

[0255] Output: Role model configuration information stored on the server.

[0256] Step 4:

[0257] The server analyzes behavioral data and role model information. Cloud tools (such as Google Cloud AutoML and AWS SageMaker) are used to analyze and compare this data in real time.

[0258] Inputs: Operational data, driver status data, role model information.

[0259] Output: Analysis results.

[0260] Step 5:

[0261] The server uses a generative AI model to generate the next action to take. It inputs a prompt sentence into a natural language generation model (e.g., OpenAI's GPT model) and obtains an appropriate action suggestion.

[0262] Input: Analysis results.

[0263] Output: Generated action suggestions.

[0264] Step 6:

[0265] The terminal notifies the information display system in the autonomous vehicle or the driver's smart glasses of the generated action suggestions. It also provides the function to notify the user visually or audibly.

[0266] Input: Generated action suggestions.

[0267] Output: Notification to the driver.

[0268] Step 7:

[0269] When the user (driver) takes action based on the suggestion, the result of that action is recorded again as a log. The results of the actions are automatically collected by sensors and cameras inside the vehicle.

[0270] Input: Driver action result data.

[0271] Output: Log data.

[0272] Step 8:

[0273] The terminal sends the aforementioned log data to the server. The server analyzes it again, generates new action suggestions, and sends feedback.

[0274] Input: Log data.

[0275] Output: New action suggestions and feedback.

[0276] As a concrete example, enter the following prompt message into the generative AI model.

[0277] Prompt message:

[0278] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[0279] Output of the generative AI model:

[0280] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[0281] By following these steps, advanced operational management and driver assistance for autonomous vehicles can be achieved.

[0282] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0283] The operation of a specific system will be described as an embodiment of the present invention.

[0284] This system is for the user to collect their own behavior logs and propose the next actions to be taken based on the set role models. It also combines an emotion engine that recognizes the user's emotions and adjusts action proposals and feedback based on them. It is realized by the cooperation of the user, the terminal, and the server according to the processing flow shown below.

[0285] 1. Collection and transmission of user behavior logs

[0286] The user inputs or automatically records the behavior logs

[0287] The user manually inputs daily behaviors (e.g., movement, shopping, exercise, meals) using a smartphone or a PC. Also, using the GPS or acceleration sensor of the smartphone, it is possible to automatically record the movement distance, exercise time, etc.

[0288] The terminal transmits the behavior data to the server

[0289] The smartphone or PC periodically transmits the collected behavior data to the server. This transmission is performed encrypted to ensure data security. For example, by transmitting the data when connected to Wi-Fi at night, battery consumption is minimized.

[0290] 2. Setting of role models

[0291] The user sets the role model

[0292] The user specifically sets the target role model. For example, input specific goals such as "becoming a professional athlete" or "living a balanced life".

[0293] The terminal transmits the role model information to the server

[0294] The set role model information is transmitted to the server after being encrypted.

[0295] 3. How the Emotion Engine Works

[0296] Recognizing user emotions

[0297] The user's device is equipped with emotion recognition capabilities, allowing for real-time monitoring of the user's current emotional state through technologies such as facial recognition and voice analysis.

[0298] Send emotional data to the server.

[0299] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[0300] 4. Generating the next course of action

[0301] The server analyzes behavioral data, role model information, and emotional data.

[0302] The server analyzes the received behavioral data, role model information, and emotional data. In particular, it generates the most effective action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[0303] The server uses AI to generate action suggestions.

[0304] The generating AI uses multiple analyzed data points to suggest the optimal next action for the user. For example, if the user is feeling fatigued, it might suggest "setting a lighter training session for the next workout."

[0305] The server sends the proposal to the user's terminal.

[0306] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[0307] 5. Suggestions to Users

[0308] The terminal notifies the user of the proposed content

[0309] The user's terminal uses the reminder function to inform the user of the proposed content. For example, a notification such as "Please do 30 minutes of light running today" is displayed. Since the proposed content is adjusted based on the emotional data, the user can take appropriate actions

[0310] 6. Feedback and Re-logging of Action Results

[0311] The user executes an action and records the result

[0312] The user executes the proposed action and enters the result into the app. For example, after running, record "I ran for 30 minutes"

[0313] The terminal sends the action result and emotional data to the server

[0314] The collected action result data and emotional data are sent to the server again and used for the next analysis

[0315] The server generates a new proposal and sends feedback

[0316] The server re-analyzes based on the new action result data and emotional data, and generates feedback and the next action proposal for the user. For example, positive feedback such as "Your running pace is increasing, so keep it up like this" is included

[0317] For example, if a user aims to become a professional athlete and records daily training and emotional data, the server will adjust the training content based on the user's fatigue level and fluctuations in motivation. Similarly, for users aiming to balance work and family life, suggestions for relaxation will be provided when stress levels are high, enabling more effective stress management.

[0318] As described above, this system supports users in approaching their desired role models by providing behavioral suggestions and feedback using generated AI based on user behavioral data, goal settings, and emotional data.

[0319] The following describes the processing flow.

[0320] Step 1:

[0321] Users enter or automatically record their activity logs.

[0322] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. The system can also automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[0323] Step 2:

[0324] The device sends behavioral data to the server.

[0325] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent in batches when connected to Wi-Fi at night to minimize battery drain.

[0326] Step 3:

[0327] Users set role models

[0328] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[0329] Step 4:

[0330] The device sends role model information to the server.

[0331] The configured role model information is encrypted before being sent to the server.

[0332] Step 5:

[0333] Recognizing user emotions

[0334] The device uses emotion recognition to monitor the user's emotional state in real time through facial expression and voice analysis. Emotion recognition often utilizes the device's camera and microphone.

[0335] Step 6:

[0336] The device sends emotional data to the server.

[0337] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[0338] Step 7:

[0339] The server analyzes behavioral data, role model information, and emotional data.

[0340] The server analyzes the received behavioral data, role model information, and emotional data. For example, it considers the user's past behavioral patterns, progress, and current emotional state to suggest the most effective next action.

[0341] Step 8:

[0342] The server uses AI to generate action suggestions.

[0343] The generating AI suggests the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it will generate a specific suggestion such as "set a lighter training session next time."

[0344] Step 9:

[0345] The server sends the proposal to the user's terminal.

[0346] The generated action suggestions are sent to the user's device and displayed as notifications.

[0347] Step 10:

[0348] The device will notify you of the suggested content.

[0349] The user's device will notify them of the suggestions using a reminder function. For example, a notification might appear saying, "Please go for a light 30-minute run today." Because the suggestions are adjusted based on emotional data, users can take appropriate action.

[0350] Step 11:

[0351] The user performs an action and records the result.

[0352] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[0353] Step 12:

[0354] The device sends behavioral results and emotional data to the server.

[0355] The collected behavioral and emotional data are sent back to the server and used for further analysis.

[0356] Step 13:

[0357] The server generates new suggestions and sends feedback.

[0358] The server re-analyzes the newly collected behavioral and emotional data to generate feedback and suggestions for the next action for the user. For example, it might generate positive feedback such as, "Your running pace is increasing, keep it up!"

[0359] Step 14:

[0360] The device will notify you of the feedback.

[0361] The server sends feedback to the user's device and notifies the user again. This allows the user to continue taking effective action towards their goal.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0364] Conventional behavior suggestion systems provide suggestions based on user behavior data and goal setting, but they do not take into account the user's emotional state. As a result, it is difficult for users to effectively carry out the suggested actions, and there is a problem in that optimal behavior suggestions cannot be made according to individual circumstances.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a target model that the user aims for; means for transmitting the target model information to the server; means for recognizing the user's emotional state and transmitting that data to the server; means for performing analysis based on the action data, target model information, and emotional data and generating the next action to be taken using a generated AI model; means for notifying the user of the generated action suggestions; means for recording the results of the user's actions and transmitting them to the server; and means for generating new suggestions based on the action results and sending feedback to the user. This makes it possible to provide individual action suggestions that take into account the user's emotional state.

[0367] A "user" refers to an individual who uses the system to input and record their own behavioral and emotional data.

[0368] "Behavioral data" refers to information recorded by users about their daily activities, including distance traveled, exercise time, and diet.

[0369] A "server" refers to a computer system that receives behavioral data, role model information, and emotional data, and performs analysis, generates behavioral suggestions, and sends feedback.

[0370] A "goal model" refers to information that shows the specific goals and desired outcomes that a user aims to achieve.

[0371] "Emotional state" refers to data that represents the user's current emotions and mood, and is collected using facial recognition technology and voice analysis.

[0372] A "generative AI model" refers to an artificial intelligence model that generates the optimal next action to take based on the data it receives.

[0373] "Action suggestion" refers to the optimal next action that the user should take, as suggested by the generative AI model.

[0374] "Action results" refer to the data recorded after a user performs a suggested action.

[0375] "Feedback" refers to new suggestions or evaluations that a server generates for a user based on their actions.

[0376] This invention is a system that collects a user's own behavioral logs and suggests the next action to take based on a set goal model. Furthermore, it is equipped with an emotion engine that recognizes the user's emotional state and adjusts the action suggestions and feedback accordingly. This system is realized through the cooperation of the user, terminal, and server, as shown below.

[0377] Collection and transmission of user behavior logs

[0378] Users manually input their daily activities (e.g., travel, shopping, exercise, meals, etc.) through a dedicated application using their smartphone or PC. Alternatively, the system can automatically record distance traveled and exercise time using the GPS and accelerometer sensors (e.g., common location services and sensors) built into the smartphone.

[0379] The device periodically sends collected behavioral data to the server. AES256 encryption is used during transmission to ensure data security. The system is designed to minimize battery consumption, especially by transmitting data at night while connected to Wi-Fi.

[0380] Setting a role model

[0381] Through using the application, users set specific goal models they aspire to. For example, they might input goals such as "become a professional athlete" or "live a balanced life."

[0382] The terminal encrypts the configured target model information and sends it to the server.

[0383] How the emotion engine works

[0384] The user's device is equipped with emotion recognition capabilities, utilizing facial recognition technology (such as OpenCV) and speech analysis (such as Google Cloud Speech-to-Text API) to monitor the user's current emotional state in real time. This function allows for understanding the user's stress level and relaxation state.

[0385] Emotional data is periodically sent to the server and used for analysis there.

[0386] Generating the next course of action

[0387] The server analyzes the received behavioral data, target model information, and emotional data. Specifically, it generates action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[0388] The server uses a generative AI (such as GPT-3®) to suggest the best course of action next. For example, if the user is feeling fatigued, it might suggest "set a lighter training session for the next session." Examples of prompt messages in this case include:

[0389] "Our goal is for users to become professional athletes. Based on their current behavioral logs and emotional data, please suggest the optimal next course of action."

[0390] The generated action suggestions are sent from the server to the user's device and displayed as notifications. These suggestions can also be set as time-based reminders.

[0391] Suggestions and feedback for users

[0392] The user's device uses a reminder function to notify them of the suggestions. For example, a notification might appear saying, "Please go for a light 30-minute run today." Suggestions based on emotional data enable users to take appropriate actions.

[0393] The user performs the suggested action and logs the result to the application. For example, after running, they might type "I ran for 30 minutes."

[0394] The device sends the collected behavioral and emotional data back to the server for use in the next analysis. The server re-analyzes the data based on the new behavioral and emotional data and generates new behavioral suggestions and feedback. For example, it might send positive feedback such as, "Your running pace is increasing, keep it up!"

[0395] Specific example

[0396] Let's say a user's goal is to "become a professional athlete" and they are recording their daily training and emotional data. The server will adjust the next training session based on the user's fatigue level and fluctuations in motivation. Similarly, if another user's goal is to "balance work and family life," the server will suggest actions to relax when stress levels are high, enabling more effective stress management.

[0397] As described above, this system uses user behavior data, goal setting, and emotional data to provide optimal action suggestions and feedback using generative AI, thereby supporting users in approaching their desired target model.

[0398] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0399] Step 1:

[0400] Users enter or automatically record their activity logs.

[0401] Users manually input data on their daily activities (e.g., travel, shopping, exercise, meals) through a dedicated application using their smartphone or PC. Additionally, the system automatically records distance traveled and exercise time using the smartphone's GPS and accelerometer. This process collects detailed activity logs of the user as input data.

[0402] Step 2:

[0403] The device sends behavioral data to the server.

[0404] The device encrypts the collected behavioral data using AES256 and periodically sends it to the server. The destination is the server, and encrypted behavioral data is sent as input data. The output is the behavioral data securely stored on the server.

[0405] Step 3:

[0406] The user sets the target model.

[0407] Users input their desired goals (e.g., "become a professional athlete" or "live a balanced life") in text format through the application. This allows the application to obtain information about their goals as input data.

[0408] Step 4:

[0409] The device sends target model information to the server.

[0410] The terminal encrypts the configured target model information using AES256 and sends it to the server. The input data is the encrypted target model information, and the transmission result is the target model information securely stored on the server.

[0411] Step 5:

[0412] Recognizing user emotions

[0413] The device is equipped with emotion recognition capabilities, utilizing facial recognition technology (e.g., OpenCV) and speech analysis (e.g., Google Cloud Speech-to-Text API) to monitor the user's emotional state. This allows for the acquisition of real-time emotion data as input.

[0414] Step 6:

[0415] Send emotional data to the server.

[0416] The device periodically encrypts recognized emotion data using AES256 and sends it to the server. The input data is encrypted emotion data, and the transmission result is the emotion data stored on the server.

[0417] Step 7:

[0418] The server analyzes behavioral data, target model information, and emotional data.

[0419] The server integrates and analyzes the received behavioral data, target model information, and emotional data. Specifically, it performs data cleansing and normalization as a preprocessing step, and uses an algorithm that considers past behavioral patterns, progress, and emotional states as an analysis step. The input data consists of behavioral data, target model information, and emotional data, and the analysis results in analytical data regarding the next action to be taken.

[0420] Step 8:

[0421] The server uses a generated AI model to create action suggestions.

[0422] The server uses a generative AI model (e.g., GPT-3) to suggest the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it might suggest "set a lighter training session next time." The input data is the analyzed data, and the output is a specific action suggestion. Example prompt: "The user's goal is to become a professional athlete. Based on the current activity log and emotional data, please suggest the optimal next action."

[0423] Step 9:

[0424] The server sends the proposal to the user's terminal.

[0425] The server sends the generated action suggestions to the terminal, where they are notified. The input data includes the action suggestions, and the output is the action suggestions notified to the user's terminal.

[0426] Step 10:

[0427] The device will notify you of the suggested content.

[0428] The device uses a reminder function to inform the user of the suggested actions. For example, a notification might appear saying, "Please go for a light 30-minute run today." The input data is the suggested action, and the output is the notified action suggestion.

[0429] Step 11:

[0430] The user performs an action and records the result.

[0431] The user performs the suggested action and logs the result in the app. For example, after running, the user might input "I ran for 30 minutes." The input data is a log of the execution result, and the output is the recorded action result.

[0432] Step 12:

[0433] The device sends behavioral results and emotional data to the server.

[0434] The terminal encrypts the collected behavioral result data and emotional data using AES256 and sends it to the server. The input data consists of behavioral result data and emotional data, and the output is the data stored on the server.

[0435] Step 13:

[0436] The server generates new suggestions and sends feedback.

[0437] The server re-analyzes the new behavioral and emotional data to generate new behavioral suggestions and feedback for the user. For example, it might send feedback such as, "Your running pace is increasing, keep it up." The input data consists of behavioral and emotional data, and the output is new behavioral suggestions and feedback.

[0438] (Application Example 2)

[0439] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0440] Conventional user assistance systems in autonomous vehicles were unable to consider user behavior data and emotional states in real time, making it difficult to suggest appropriate actions. Furthermore, they lacked the ability to suggest the optimal next course of action based on user-defined goals (role models), resulting in insufficient support for users achieving their objectives.

[0441] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for the user to input or automatically record their actions, means for transmitting the action data to the server, means for setting a role model that the user aims to emulate, means for transmitting the role model information to the server, means for recognizing the user's emotional state in real time, means for transmitting the emotional data to the server, means for generating the next action to be taken using a generated AI based on the action data, role model information and emotional data, means for notifying the user of the generated action suggestion, means for logging the results of the user's actions again and transmitting them to the server, means for generating a new suggestion based on the action results and emotional data and sending feedback to the user, and means for providing the action suggestion in cooperation with an autonomous mobile system. This makes it possible to analyze the user's emotional state and action data in real time and to quickly and effectively provide optimal action suggestions based on the user's goals.

[0442] A "user" is an entity that uses this system to input and record their own activity logs and receive suggestions.

[0443] "Behavioral data" refers to various actions performed by users (e.g., travel, shopping, exercise, eating, etc.) and related information.

[0444] A "server" is a device that collects and analyzes behavioral data, role model information, and emotional data, and then generates and provides the next course of action to be taken.

[0445] A "role model" refers to the goal or ideal state that the user aspires to, and action suggestions are made based on that.

[0446] "Emotional state" refers to the user's current emotional state (e.g., sad, happy, tired, etc.), and is data recognized in real time using sensors, etc.

[0447] "Generative AI" refers to an artificial intelligence model that analyzes collected data and automatically generates optimal action suggestions.

[0448] An "autonomous mobility system" is a system that has autonomous driving technology used by users for transportation, such as self-driving vehicles.

[0449] "Action suggestions" are suggestions for the next action to take, generated by a generating AI based on the user's behavioral data, emotional state, and role model information.

[0450] "Feedback" refers to evaluations and suggestions for future actions generated based on the results of the user's actions.

[0451] The configuration and operation of a specific system will be described as an embodiment for carrying out the present invention. This system uses behavioral data, emotional data, and role model information to generate action suggestions based on the role model that the user aspires to.

[0452] 1. Prerequisite System Configuration

[0453] The system is built upon the following key hardware and software components.

[0454] Hardware:

[0455] 1. Head-mounted display (HMD): Recognizes the user's emotional state in real time and displays action suggestions.

[0456] 2. ECU (Electronic Control Unit) of an autonomous vehicle: Collects vehicle driving data.

[0457] 3. Sensors for emotion recognition: such as heart rate sensors and cameras for facial expression recognition.

[0458] 4. Server: Cloud-based, it performs data analysis and generates action suggestions.

[0459] software:

[0460] 1. Emotion recognition software: For example, use the Emotion API from Microsoft® Azure®.

[0461] 2. Data analysis and generative AI models: For example, using TENSORFLOW®, PyTorch, etc.

[0462] 3. Encryption protocol: For example, use TLS / SSL.

[0463] 4. Display software for the head-mounted display: For example, use Unity.

[0464] 2. Data collection and transmission

[0465] How users can input or automatically record behavioral data:

[0466] Users manually or automatically record their daily activities using smartphones, PCs, or HMDs.

[0467] For example, it can automatically record distance traveled and exercise time using GPS and accelerometers.

[0468] How devices send behavioral and emotional data to servers:

[0469] The collected behavioral and emotional data is encrypted using TLS / SSL and periodically sent to the server.

[0470] 3. Setting a role model

[0471] How users can set up role models:

[0472] Users set their own goals (for example, "develop safe driving habits") using a smartphone, PC, or HMD.

[0473] This configured role model information is encrypted before being sent to the server.

[0474] 4. How the Emotion Engine Works

[0475] How to recognize user emotions in real time:

[0476] The system uses sensors and cameras mounted on the HMD to monitor the user's facial expressions and heart rate.

[0477] For example, the Emotion API can be used to analyze a user's emotional state.

[0478] How to send emotional data to a server:

[0479] The recognized emotion data is periodically encrypted and sent to the server.

[0480] 5. Generating action proposals

[0481] How the server analyzes behavioral data, role model information, and sentiment data:

[0482] The server performs analysis based on past behavioral data and current emotional states. For example, it might use TensorFlow or PyTorch.

[0483] How a server uses generated AI to create action suggestions:

[0484] The generative AI proposes the optimal next course of action based on the analyzed data.

[0485] For example, if a user is feeling tired, the system might suggest, "We recommend taking a break at the next service area."

[0486] How the server sends the proposal to the user's terminal:

[0487] The generated action suggestions are notified to the user in real time via the HMD.

[0488] 6. Re-logging of feedback and action results

[0489] How users perform actions and record the results:

[0490] The user performs the suggested action and enters the result into the app.

[0491] How the device sends behavioral results and emotional data to the server:

[0492] The collected behavioral data and emotional data are sent back to the server.

[0493] How the server generates new suggestions and sends feedback:

[0494] The server re-analyzes the data based on the new information and generates new suggestions and feedback for the user.

[0495] Specific example:

[0496] Driver A set the goal of "reducing risks on the highway." When the driver feels fatigued while driving, the HMD detects an increase in heart rate and changes in facial expression. The server analyzes this and suggests, "The next rest area is 20 minutes away. We recommend taking a break soon."

[0497] Example of a prompt:

[0498] While driving an autonomous vehicle, suggest the optimal next course of action based on the role model set by the driver (e.g., safe driving).

[0499] Please consider the following data:

[0500] Driver's current emotional state (heart rate, facial recognition data)

[0501] Past driving behavior data (speed, braking, steering, etc.)

[0502] A role model set by the driver (e.g., risk reduction on highways)

[0503] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0504] (Process flow)

[0505] Step 1:

[0506] Input: User's daily activity data (e.g., distance traveled, exercise time).

[0507] Operation: The device collects activity data that the user inputs or automatically records using a smartphone, PC, or HMD. For example, it records distance traveled and exercise time using GPS and accelerometer sensors.

[0508] Output: Collected behavioral data.

[0509] Step 2:

[0510] Input: Collected behavioral data.

[0511] Operation: The device encrypts collected behavioral data and sends it to the server using TLS / SSL. For example, by sending data when connected to Wi-Fi at night, battery consumption can be minimized.

[0512] Output: Behavioral data sent to the server.

[0513] Step 3:

[0514] Input: User-defined role model information (e.g., goal of aiming for safe driving).

[0515] Operation: The user uses a smartphone, PC, or HMD to specifically define their target role model, and the device encrypts this information before sending it to the server.

[0516] Output: Role model information sent to the server.

[0517] Step 4:

[0518] Input: Behavioral data and role model information stored on the server side.

[0519] Operation: The server analyzes the user's past behavioral patterns and progress based on the received behavioral data and role model information. For example, it may use TensorFlow or PyTorch to analyze the data.

[0520] Output: Analysis results.

[0521] Step 5:

[0522] Input: Real-time user emotion data (e.g., heart rate, facial expression data).

[0523] Operation: The HMD uses sensors and cameras to recognize the user's emotional state (heart rate, facial expressions, etc.) in real time, encrypts the data, and sends it to the server.

[0524] Output: Emotional data sent to the server.

[0525] Step 6:

[0526] Input: Analysis results and real-time sentiment data.

[0527] Operation: The server uses a generative AI to generate the optimal next action based on analyzed behavioral data, role model information, and sentiment data. For example, it uses a generative AI model (e.g., GPT-4®) to create prompt statements.

[0528] Output: Generated action suggestions.

[0529] Step 7:

[0530] Input: Generated action suggestions.

[0531] Operation: The server encrypts the generated action suggestion, sends it to the HMD in real time, and the HMD notifies the user. For example, it might display "We recommend taking a break at the next service area."

[0532] Output: Notification of suggested actions to the user.

[0533] Step 8:

[0534] Input: The result of the user's actions based on the suggestion.

[0535] Operation: The user performs the suggested action and inputs the result back into the HMD, smartphone, or PC. For example, after exercising, they might record "I ran for 30 minutes."

[0536] Output: The result of the input action.

[0537] Step 9:

[0538] Input: Action results and sentiment data recorded again as logs.

[0539] Operation: The device encrypts the collected behavioral data and emotional data and sends it back to the server.

[0540] Output: Action results and sentiment data sent to the server.

[0541] Step 10:

[0542] Input: New behavioral result data and sentiment data sent to the server.

[0543] Operation: The server re-analyzes the new behavioral result data and sentiment data to generate new behavioral suggestions and feedback. For example, it might generate feedback such as, "Your running pace is increasing, so keep it up."

[0544] Output: New action suggestions and feedback.

[0545] The system operates in the manner described above, enabling it to provide real-time suggestions for optimal actions based on the role model the user aspires to.

[0546] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0547] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0548] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0549] [Second Embodiment]

[0550] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0551] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0552] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0553] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0554] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0555] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0556] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0557] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0558] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0559] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0560] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0561] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0562] The operation of a specific system will be described as an embodiment for carrying out the present invention.

[0563] This system collects users' activity logs and suggests their next course of action based on a pre-configured role model. It is implemented through the cooperation of the user, device, and server according to the following processing flow.

[0564] 1. Collection and transmission of user activity logs

[0565] Users enter or automatically record their activity logs.

[0566] Users record their daily activities, such as travel, shopping, exercise, and meals, using their smartphones or PCs. For example, they can automatically record running distance and time using their smartphone's accelerometer and GPS function as an exercise log. They can also manually enter meal details and shopping lists into the app.

[0567] The device sends behavioral data to the server.

[0568] The collected behavioral data is encrypted and then periodically sent to the server. This ensures the security of the data.

[0569] 2. Setting a role model

[0570] Users set role models

[0571] Users can specifically define their role model. For example, they can select a professional athlete or a successful business person and input their characteristics and goals.

[0572] The device sends role model information to the server.

[0573] The configured role model information is encrypted and sent to the server, just like the user's behavioral data.

[0574] 3. Generating the next course of action

[0575] The server analyzes behavioral data and role model information.

[0576] The server performs analysis based on the received behavioral data and role model information. In particular, it compares past behavioral patterns with current progress to generate the optimal actions for the user to move closer to their goals.

[0577] The server uses AI to generate action suggestions.

[0578] The generating AI considers behavioral data and role model information to generate specific next steps. For example, if it determines that adding weight training will bring the user closer to their goal, it will create a suggestion such as "Add 30 minutes of weight training to your next workout."

[0579] The server sends the proposal to the user's terminal.

[0580] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[0581] 4. Suggestions for Users

[0582] The device will notify you of the suggested content.

[0583] The user's device will notify them of the suggestions using its notification function. This ensures that the user does not forget what action they should take next.

[0584] 5. Re-logging of feedback and action results

[0585] The user performs an action and records the result.

[0586] The user performs the suggested action and enters the results into the app. For example, if they perform the suggested weight training, they will record the duration and details of the workout again.

[0587] The device sends the results of its actions to the server.

[0588] The collected behavioral data is sent back to the server and used for further analysis.

[0589] The server generates new suggestions and sends feedback.

[0590] The server performs a re-analysis based on the newly collected behavioral data and generates feedback and suggestions for the next action for the user. This is then sent to the user's terminal for notification.

[0591] Specific example

[0592] Example 1: If you aim to become a professional athlete

[0593] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[0594] Example 2: When balancing work and family life

[0595] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[0596] As described above, this system supports users in approaching their desired role models by providing action suggestions and feedback using generated AI based on user behavior data and goal settings.

[0597] The following describes the processing flow.

[0598] Step 1:

[0599] Users enter or automatically record their activity logs.

[0600] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. Alternatively, the system can automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[0601] Step 2:

[0602] The device sends behavioral data to the server.

[0603] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent at night when connected to Wi-Fi to minimize battery consumption.

[0604] Step 3:

[0605] Users set role models

[0606] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[0607] Step 4:

[0608] The device sends role model information to the server.

[0609] The configured role model information is encrypted before being sent to the server.

[0610] Step 5:

[0611] The server analyzes behavioral data and role model information.

[0612] The server performs analysis based on the received behavioral data and role model information. This involves analyzing the user's past behavioral patterns and progress to determine what actions are necessary to move closer to the role model.

[0613] Step 6:

[0614] The server uses AI to generate action suggestions.

[0615] The generating AI uses analyzed behavioral data and role model information to suggest the optimal next action for the user. For example, it can generate specific suggestions such as, "Add an hour of running next weekend."

[0616] Step 7:

[0617] The server sends the proposal to the user's terminal.

[0618] The generated action suggestions are sent to the user's device, allowing the user to see the suggested actions.

[0619] Step 8:

[0620] The device will notify you of the suggested content.

[0621] The user's device will notify them of the suggestion using a reminder function. For example, a notification might appear saying, "Please go for a 30-minute run every morning at 6:00 AM."

[0622] Step 9:

[0623] The user performs an action and records the result.

[0624] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[0625] Step 10:

[0626] The device sends the results of its actions to the server.

[0627] The collected behavioral data is sent back to the server. This data is also encrypted before transmission and used for further analysis.

[0628] Step 11:

[0629] The server generates new suggestions and sends feedback.

[0630] The server re-analyzes the new behavioral data and generates feedback and suggestions for the next action for the user. For example, this may include positive feedback such as, "Your running pace is increasing, keep it up!"

[0631] Step 12:

[0632] The device will notify you of the feedback.

[0633] The server sends feedback to the user's device and notifies the user again. This allows the user to effectively continue taking action towards their goal.

[0634] (Example 1)

[0635] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0636] Conventional behavioral suggestion systems had problems such as not being able to fully utilize user behavior data and not being able to effectively provide optimal behavioral suggestions that corresponded to the user's goals. In particular, ensuring the security of user behavior data and improving the accuracy of behavioral suggestions were challenges.

[0637] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0638] In this invention, the server includes means for the user to input or automatically record their actions; means for encrypting the action data and transmitting it to the server; means for setting a role model that the user aims to emulate; means for encrypting the role model information and transmitting it to the server; means for generating the next action to be taken using a generated AI model based on the action data and role model information; means for notifying the user's terminal of the generated action suggestion; means for logging the results of the user's actions again, encrypting them, and transmitting them to the server; and means for generating new suggestions based on the action results and transmitting feedback to the user's terminal. This makes it possible to provide highly accurate and optimal action suggestions that match the user's goals while ensuring the security of the user's action data.

[0639] A "user" refers to an individual who uses this system to record behavioral data and aims to achieve their goals.

[0640] "Behavioral data" refers to information about daily activities such as movement, exercise, and eating that users record.

[0641] "Encryption" refers to technologies used to protect behavioral data and role model information from being deciphered by third parties.

[0642] A "server" refers to a computer system that receives and analyzes behavioral data and role model information sent by users, and generates and sends appropriate action suggestions.

[0643] A "role model" refers to an ideal individual or ideal figure that a user aspires to be like, whose characteristics and goals are set for them.

[0644] A "generative AI model" refers to an artificial intelligence model that generates suggestions for the next course of action based on user behavior data and role model information.

[0645] "Action suggestions" refer to specific action instructions generated by the server to support the user in achieving their goals.

[0646] "Feedback" refers to notifying the user of newly generated suggestions or evaluations based on the results of the actions they have taken.

[0647] A "sensor" refers to an electronic device used to acquire user movement and location information.

[0648] This invention is a system that allows users to record their own actions, generate action suggestions to help them move closer to their target role model, and support their implementation. This system operates through the coordinated efforts of the user, terminal, and server.

[0649] First, users record their daily activity logs using their smartphones or PCs. Specifically, they can automatically record running distance and time using their smartphone's accelerometer and GPS function. They can also manually input meal details and shopping lists into the app. The device encrypts the collected activity data and periodically sends it to the server using secure protocols such as HTTPS.

[0650] Next, users set their desired role model. In the dedicated app's settings screen, they can select a role model such as a "professional athlete" or a "successful business person," and input their characteristics and goals. This role model information, like behavioral data, is encrypted and sent to the server.

[0651] The behavioral data and role model information sent to the server are subject to analysis. Based on this information, the server uses a generative AI model to generate specific actions that the user should take next. For example, if the user aims to become a "professional athlete," the AI ​​will generate a specific suggestion such as "add 30 minutes of weight training to your next training session." This suggestion is sent from the server to the user's device and notified to the user. The notification can also be set as a time-based reminder.

[0652] The user receives action suggestions from their device, performs them, and records the results again in the app. For example, if the user performs a suggested weight training exercise, the user records the duration and content of the exercise again and sends it to the server via their device. This action result data is then analyzed again on the server and sent back to the user as new suggestions or feedback. This allows the user to continuously perform optimal actions.

[0653] Specific example

[0654] Example 1: If you aim to become a professional athlete

[0655] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[0656] Example 2: When balancing work and family life

[0657] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[0658] Example of a prompt

[0659] Activity data: "Distance traveled: 5km, Exercise time: 30 minutes, Meal content: Salad"

[0660] Role model: "Goal: Professional athlete, Characteristics: Daily training: 2 hours, Diet: High protein, low calorie"

[0661] Please suggest the next action this user should take.

[0662] Thus, the system of the present invention supports users in approaching their desired role model by providing action suggestions and feedback using a generated AI model based on the user's behavioral data and goal settings.

[0663] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0664] Step 1:

[0665] Users enter or automatically record their activity logs.

[0666] Users record their daily activities using a dedicated app on their smartphone or PC. Specifically, one method involves automatically recording running distance and time using the smartphone's accelerometer and GPS function. Users can also manually enter details of meals and shopping lists into the app.

[0667] Input: User behavior data (e.g., distance traveled, exercise time, diet)

[0668] Output: Recorded behavioral data

[0669] Specific operation: The smartphone's accelerometer detects the user's steps, and the GPS function measures the distance traveled. The user also inputs details of their meals through the app's UI.

[0670] Step 2:

[0671] The device encrypts behavioral data and sends it to the server.

[0672] The device encrypts user behavior data and periodically sends it to the server using secure protocols such as HTTPS. This process ensures data security.

[0673] Input: Recorded behavioral data

[0674] Output: Encrypted behavioral data

[0675] Specific operation: A data encryption module operates within the app, encrypting behavioral data using encryption algorithms such as AES. The data is then sent to the server using HTTPS.

[0676] Step 3:

[0677] Set a role model that the user aspires to be.

[0678] Users select a role model, such as a "professional athlete" or a "successful business person," from the settings screen of the dedicated app and input their characteristics and goals. This information, like behavioral data, is encrypted.

[0679] Input: User role model information (e.g., goals, characteristics)

[0680] Output: Configured role model information

[0681] Specific operation: The user enters role model information using dropdown lists or text boxes in the app's settings screen.

[0682] Step 4:

[0683] The device encrypts the role model information and sends it to the server.

[0684] As mentioned above, the device encrypts the role model information and sends it to the server using the HTTPS protocol.

[0685] Input: Configured role model information

[0686] Output: Encrypted role model information

[0687] Specific operation: The in-app data encryption module encrypts the role model information using an encryption algorithm and sends it to the server via HTTPS.

[0688] Step 5:

[0689] The server analyzes behavioral data and role model information.

[0690] The server uses a dedicated analysis algorithm to analyze the received behavioral data and role model information. This helps determine the optimal actions for the user to take to achieve their goals.

[0691] Input: Encrypted behavioral data, role model information

[0692] Output: Data analysis results

[0693] Specific operation: The analysis engine runs on the server and compares behavioral data with role model information. It also compares past behavioral patterns with current progress.

[0694] Step 6:

[0695] The server uses a generated AI model to create action suggestions.

[0696] The server uses a generated AI model to create specific action suggestions based on the data analysis results described above.

[0697] Input: Data analysis results

[0698] Output: Action Suggestions

[0699] Specific operation: The generating AI model generates optimal action suggestions for the user based on the results of data analysis (e.g., "Add 30 minutes of weight training to your next workout").

[0700] Step 7:

[0701] The server sends the proposal to the user's terminal.

[0702] The generated action suggestions are sent from the server to the user's terminal in the form of a notification.

[0703] Input: Action suggestion

[0704] Output: Action suggestions sent to the user's device

[0705] Specific operation: The server sends an action suggestion to the device, and the device displays it to the user as a notification. The notification can also be set as a time-based reminder.

[0706] Step 8:

[0707] The user performs an action and records the result.

[0708] The user performs the suggested action and records the result again in the app.

[0709] Input: Proposed action

[0710] Output: Results of the actions performed

[0711] Specific actions: The user performs the suggested actions and enters the results using a smartphone or PC app. This includes details such as exercise time and content.

[0712] Step 9:

[0713] The device encrypts the results of its actions and sends them to the server.

[0714] The device encrypts the collected behavioral data and sends it back to the server.

[0715] Input: Results of the actions performed

[0716] Output: Encrypted behavioral result data

[0717] Specific operation: The in-app data encryption module runs, encrypts the action result data, and sends it to the server.

[0718] Step 10:

[0719] The server generates new suggestions and sends feedback.

[0720] The server re-analyzes the newly collected behavioral data, generates new suggestions and feedback, and sends them to the user's device.

[0721] Input: Encrypted behavioral data

[0722] Output: New suggestions and feedback

[0723] Specific operation: The server performs a re-analysis and generates new action suggestions and feedback. The generated information is sent to the user's device in the form of a notification.

[0724] (Application Example 1)

[0725] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0726] The challenge lies in creating an operational support system that safely and efficiently manages the operation of autonomous vehicles while also considering the driver's health. Furthermore, it is necessary to achieve continuously optimized operational management by specifically proposing the driving behaviors that drivers should aim for and incorporating the results of their execution.

[0727] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0728] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a role model that the user aspires to; means for transmitting the role model information to the server; means for generating the next action to be taken using a generative AI based on the action data and role model information; means for notifying the user of the generated action suggestions; means for logging the results of the user's actions again and transmitting them to the server; means for generating new suggestions based on the action results and sending feedback to the user; means for automatically collecting operation data of the autonomous vehicle and driver action data, encrypting them, and transmitting them to the server; means for generating operation and driver action suggestions using a generative AI model based on the optimal operation pattern and driver model set by the administrator; and means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses. This makes it possible to perform advanced operation management and driver support for autonomous vehicles and improve the safety and efficiency of operations.

[0729] "Means for users to input or automatically record their own actions" refers to devices or software that allow users to manually input their own behavioral data or to automatically record behavioral data using a smartphone or sensors.

[0730] "Means for transmitting the aforementioned behavioral data to the server" refers to a device or software that securely transmits the collected and recorded behavioral data to the server via the internet.

[0731] "Means for setting a role model that the user aspires to" refers to a device or software that allows the user to specifically define the person or ideal they aspire to be and input that information.

[0732] "Means for transmitting the role model information to the server" refers to a device or software that securely transmits the configured role model information to the server via the internet.

[0733] "Means for generating the next action to be taken using a generative AI based on the aforementioned behavioral data and role model information" refers to an algorithm and system that analyzes the collected behavioral data and role model information and calculates the optimal next action using a generative AI model.

[0734] "Means for notifying the user of generated action suggestions" refers to a notification system for informing the user of action suggestions created by the generating AI, and is a device or software that uses a smartphone, smart glasses, in-vehicle display, etc.

[0735] "Means for logging the results of user actions and sending them back to the server" refers to a device or software that records the results of user actions and sends that data back to the server.

[0736] "Means for generating new suggestions based on the aforementioned action results and sending feedback to the user" refers to a device or software that re-analyzes the recorded action result data, generates new action suggestions, and notifies the user.

[0737] "Means for automatically collecting operational data of autonomous vehicles and driver behavior data, encrypting them, and then transmitting them to a server" refers to a device or software that collects various data related to the operation of autonomous vehicles and driver behavior data using sensors and cameras, encrypts them, and then transmits them to a server.

[0738] "Means for generating operational and driver action suggestions using a generation AI model based on optimal operational patterns and driver models set by the administrator" refers to a device or software that generates operational and driver action suggestions using a generation AI model based on operational pattern and driver model information set by the administrator.

[0739] "Means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses" refers to a device or software that notifies the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses.

[0740] This invention provides a system for advanced operational management and driver assistance of autonomous vehicles. This system allows users (operation managers and drivers) to collect their own behavioral logs and propose the next course of action based on their configured optimal operating patterns and driver models. This is achieved through the cooperation of a server, terminals, and autonomous vehicles, following the processing flow outlined below.

[0741] 1. Collection and transmission of user activity logs

[0742] Various sensors (cameras, accelerometers, GPS, etc.) within the autonomous vehicle are used as a means to automatically record user activity logs. These sensors record operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. The collected data is encrypted and then periodically transmitted to a server. This ensures the security of the data.

[0743] 2. Setting a role model

[0744] As a means for administrators to set role models, they can specifically define the optimal driving patterns and driver models they aim for. For example, this could include patterns that prioritize safe driving or patterns that emphasize efficiency. The set role model information is encrypted and sent to the server, just like user behavior data.

[0745] 3. Generating the next course of action

[0746] The server analyzes behavioral data and role model information. This analysis uses a generated AI model based on the behavioral data and role model information. The server compares past behavioral patterns with current progress and generates the optimal actions for the user to get closer to their goal. For example, if a driver hasn't taken a two-hour break, it will create a specific suggestion such as "Take a 15-minute break at the next rest stop."

[0747] 4. Suggestions for Users

[0748] The generated action suggestions are communicated to the vehicle's information display system or the driver's smart glasses. This allows the driver to properly understand and act on what to do next, even while driving.

[0749] 5. Re-logging of feedback and action results

[0750] There is a mechanism for automatically re-logging data as a means of recording the results when the user (driver) performs a suggested action. For example, in-vehicle sensors detect and record the driver's resting status. The collected action result data is sent back to the server and used for the next analysis. The server performs a re-analysis based on the newly collected action result data, generates new suggestions, and sends feedback.

[0751] Technology for realizing functionality

[0752] This system is constructed using the following technologies:

[0753] Hardware:

[0754] Various sensors installed inside the vehicle (camera, accelerometer, GPS, etc.)

[0755] Smart glasses for drivers (e.g., Google Glass)

[0756] Information display system for autonomous vehicles

[0757] software:

[0758] Custom API for data collection and transmission (encryption supported)

[0759] Cloud tools (e.g., Google Cloud AutoML, AWS SageMaker) are used to analyze behavioral data and operational models.

[0760] Generative AI models include natural language generation models (e.g., OpenAI's GPT model).

[0761] Specific example

[0762] For example, the following prompt statements are used to input into the generated AI model.

[0763] Prompt message:

[0764] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[0765] Output of the generative AI model:

[0766] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[0767] As described above, the system for implementing the present invention enables advanced operational management and driver assistance for autonomous vehicles. This is expected to improve the safety and efficiency of operations.

[0768] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0769] Step 1:

[0770] The user records their activity log. Various sensors in the autonomous vehicle (cameras, accelerometers, GPS, etc.) collect operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. Operational data and driver status data are obtained as input data. This data is encrypted.

[0771] Output: Encrypted operational data and driver status data.

[0772] Step 2:

[0773] The terminal sends the encrypted data to the server. The data is securely transferred to the server using a transmission protocol (e.g., HTTPS).

[0774] Input: Encrypted operational data and driver status data.

[0775] Output: Encrypted data stored on the server.

[0776] Step 3:

[0777] The user (flight manager) sets the role model they aspire to. The manager uses a web portal or application to specify the optimal ferry pattern and driver model (safe driving, efficient driving, etc.). The configuration information is encrypted and sent to the server.

[0778] Input: Role model configuration information.

[0779] Output: Role model configuration information stored on the server.

[0780] Step 4:

[0781] The server analyzes behavioral data and role model information. Cloud tools (such as Google Cloud AutoML and AWS SageMaker) are used to analyze and compare this data in real time.

[0782] Inputs: Operational data, driver status data, role model information.

[0783] Output: Analysis results.

[0784] Step 5:

[0785] The server uses a generative AI model to generate the next action to take. It inputs a prompt sentence into a natural language generation model (e.g., OpenAI's GPT model) and obtains an appropriate action suggestion.

[0786] Input: Analysis results.

[0787] Output: Generated action suggestions.

[0788] Step 6:

[0789] The terminal notifies the information display system in the autonomous vehicle or the driver's smart glasses of the generated action suggestions. It also provides the function to notify the user visually or audibly.

[0790] Input: Generated action suggestions.

[0791] Output: Notification to the driver.

[0792] Step 7:

[0793] When the user (driver) takes action based on the suggestion, the result of that action is recorded again as a log. The results of the actions are automatically collected by sensors and cameras inside the vehicle.

[0794] Input: Driver action result data.

[0795] Output: Log data.

[0796] Step 8:

[0797] The terminal sends the aforementioned log data to the server. The server analyzes it again, generates new action suggestions, and sends feedback.

[0798] Input: Log data.

[0799] Output: New action suggestions and feedback.

[0800] As a concrete example, enter the following prompt message into the generative AI model.

[0801] Prompt message:

[0802] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[0803] Output of the generative AI model:

[0804] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[0805] By following these steps, advanced operational management and driver assistance for autonomous vehicles can be achieved.

[0806] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0807] The operation of a specific system will be described as an embodiment of the present invention.

[0808] This system collects the user's own behavioral logs and suggests the next course of action based on a pre-set role model. It also incorporates an emotion engine that recognizes the user's emotions and adjusts the action suggestions and feedback accordingly. This is achieved through the cooperation of the user, device, and server according to the processing flow outlined below.

[0809] 1. Collection and transmission of user activity logs

[0810] Users enter or automatically record their activity logs.

[0811] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. Alternatively, the system can automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[0812] The device sends behavioral data to the server.

[0813] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent at night when connected to Wi-Fi to minimize battery consumption.

[0814] 2. Setting a role model

[0815] Users set role models

[0816] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[0817] The device sends role model information to the server.

[0818] The configured role model information is encrypted before being sent to the server.

[0819] 3. How the Emotion Engine Works

[0820] Recognizing user emotions

[0821] The user's device is equipped with emotion recognition capabilities, allowing for real-time monitoring of the user's current emotional state through technologies such as facial recognition and voice analysis.

[0822] Send emotional data to the server.

[0823] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[0824] 4. Generating the next course of action

[0825] The server analyzes behavioral data, role model information, and emotional data.

[0826] The server analyzes the received behavioral data, role model information, and emotional data. In particular, it generates the most effective action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[0827] The server uses AI to generate action suggestions.

[0828] The generating AI uses multiple analyzed data points to suggest the optimal next action for the user. For example, if the user is feeling fatigued, it might suggest "setting a lighter training session for the next workout."

[0829] The server sends the proposal to the user's terminal.

[0830] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[0831] 5. Suggestions for Users

[0832] The device will notify you of the suggested content.

[0833] The user's device uses a reminder function to notify them of the suggestions. For example, a notification might appear saying, "Please go for a light 30-minute run today." Because the suggestions are adjusted based on emotional data, users can take appropriate action.

[0834] 6. Re-logging of feedback and action results

[0835] The user performs an action and records the result.

[0836] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[0837] The device sends behavioral results and emotional data to the server.

[0838] The collected behavioral and emotional data are sent back to the server and used for further analysis.

[0839] The server generates new suggestions and sends feedback.

[0840] The server re-analyzes the new behavioral and emotional data to generate feedback and suggestions for the next action for the user. For example, this might include positive feedback such as, "Your running pace is increasing, keep it up!"

[0841] For example, if a user aims to become a professional athlete and records daily training and emotional data, the server will adjust the training content based on the user's fatigue level and fluctuations in motivation. Similarly, for users aiming to balance work and family life, suggestions for relaxation will be provided when stress levels are high, enabling more effective stress management.

[0842] As described above, this system supports users in approaching their desired role models by providing behavioral suggestions and feedback using generated AI based on user behavioral data, goal settings, and emotional data.

[0843] The following describes the processing flow.

[0844] Step 1:

[0845] Users enter or automatically record their activity logs.

[0846] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. The system can also automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[0847] Step 2:

[0848] The device sends behavioral data to the server.

[0849] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent in batches when connected to Wi-Fi at night to minimize battery drain.

[0850] Step 3:

[0851] Users set role models

[0852] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[0853] Step 4:

[0854] The device sends role model information to the server.

[0855] The configured role model information is encrypted before being sent to the server.

[0856] Step 5:

[0857] Recognizing user emotions

[0858] The device uses emotion recognition to monitor the user's emotional state in real time through facial expression and voice analysis. Emotion recognition often utilizes the device's camera and microphone.

[0859] Step 6:

[0860] The device sends emotional data to the server.

[0861] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[0862] Step 7:

[0863] The server analyzes behavioral data, role model information, and emotional data.

[0864] The server analyzes the received behavioral data, role model information, and emotional data. For example, it considers the user's past behavioral patterns, progress, and current emotional state to suggest the most effective next action.

[0865] Step 8:

[0866] The server uses AI to generate action suggestions.

[0867] The generating AI suggests the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it will generate a specific suggestion such as "set a lighter training session next time."

[0868] Step 9:

[0869] The server sends the proposal to the user's terminal.

[0870] The generated action suggestions are sent to the user's device and displayed as notifications.

[0871] Step 10:

[0872] The device will notify you of the suggested content.

[0873] The user's device will notify them of the suggestions using a reminder function. For example, a notification might appear saying, "Please go for a light 30-minute run today." Because the suggestions are adjusted based on emotional data, users can take appropriate action.

[0874] Step 11:

[0875] The user performs an action and records the result.

[0876] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[0877] Step 12:

[0878] The device sends behavioral results and emotional data to the server.

[0879] The collected behavioral and emotional data are sent back to the server and used for further analysis.

[0880] Step 13:

[0881] The server generates new suggestions and sends feedback.

[0882] The server re-analyzes the newly collected behavioral and emotional data to generate feedback and suggestions for the next action for the user. For example, it might generate positive feedback such as, "Your running pace is increasing, keep it up!"

[0883] Step 14:

[0884] The device will notify you of the feedback.

[0885] The server sends feedback to the user's device and notifies the user again. This allows the user to continue taking effective action towards their goal.

[0886] (Example 2)

[0887] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0888] Conventional behavior suggestion systems provide suggestions based on user behavior data and goal setting, but they do not take into account the user's emotional state. As a result, it is difficult for users to effectively carry out the suggested actions, and there is a problem in that optimal behavior suggestions cannot be made according to individual circumstances.

[0889] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0890] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a target model that the user aims for; means for transmitting the target model information to the server; means for recognizing the user's emotional state and transmitting that data to the server; means for performing analysis based on the action data, target model information, and emotional data and generating the next action to be taken using a generated AI model; means for notifying the user of the generated action suggestions; means for recording the results of the user's actions and transmitting them to the server; and means for generating new suggestions based on the action results and sending feedback to the user. This makes it possible to provide individual action suggestions that take into account the user's emotional state.

[0891] A "user" refers to an individual who uses the system to input and record their own behavioral and emotional data.

[0892] "Behavioral data" refers to information recorded by users about their daily activities, including distance traveled, exercise time, and diet.

[0893] A "server" refers to a computer system that receives behavioral data, role model information, and emotional data, and performs analysis, generates behavioral suggestions, and sends feedback.

[0894] A "goal model" refers to information that shows the specific goals and desired outcomes that a user aims to achieve.

[0895] "Emotional state" refers to data that represents the user's current emotions and mood, and is collected using facial recognition technology and voice analysis.

[0896] A "generative AI model" refers to an artificial intelligence model that generates the optimal next action to take based on the data it receives.

[0897] "Action suggestion" refers to the optimal next action that the user should take, as suggested by the generative AI model.

[0898] "Action results" refer to the data recorded after a user performs a suggested action.

[0899] "Feedback" refers to new suggestions or evaluations that a server generates for a user based on their actions.

[0900] This invention is a system that collects a user's own behavioral logs and suggests the next action to take based on a set goal model. Furthermore, it is equipped with an emotion engine that recognizes the user's emotional state and adjusts the action suggestions and feedback accordingly. This system is realized through the cooperation of the user, terminal, and server, as shown below.

[0901] Collection and transmission of user behavior logs

[0902] Users manually input their daily activities (e.g., travel, shopping, exercise, meals, etc.) through a dedicated application using their smartphone or PC. Alternatively, the system can automatically record distance traveled and exercise time using the GPS and accelerometer sensors (e.g., common location services and sensors) built into the smartphone.

[0903] The device periodically sends collected behavioral data to the server. AES256 encryption is used during transmission to ensure data security. The system is designed to minimize battery consumption, especially by transmitting data at night while connected to Wi-Fi.

[0904] Setting a role model

[0905] Through using the application, users set specific goal models they aspire to. For example, they might input goals such as "become a professional athlete" or "live a balanced life."

[0906] The terminal encrypts the configured target model information and sends it to the server.

[0907] How the emotion engine works

[0908] The user's device is equipped with emotion recognition capabilities, utilizing facial recognition technology (such as OpenCV) and speech analysis (such as Google Cloud Speech-to-Text API) to monitor the user's current emotional state in real time. This function allows for understanding the user's stress level and relaxation state.

[0909] Emotional data is periodically sent to the server and used for analysis there.

[0910] Generating the next course of action

[0911] The server analyzes the received behavioral data, target model information, and emotional data. Specifically, it generates action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[0912] The server uses a generative AI (such as GPT-3) to suggest the best course of action next. For example, if the user is feeling fatigued, it might suggest "set a lighter training session next time." Examples of prompts in this case include:

[0913] "Our goal is for users to become professional athletes. Based on their current behavioral logs and emotional data, please suggest the optimal next course of action."

[0914] The generated action suggestions are sent from the server to the user's device and displayed as notifications. These suggestions can also be set as time-based reminders.

[0915] Suggestions and feedback for users

[0916] The user's device uses a reminder function to notify them of the suggestions. For example, a notification might appear saying, "Please go for a light 30-minute run today." Suggestions based on emotional data enable users to take appropriate actions.

[0917] The user performs the suggested action and logs the result to the application. For example, after running, they might type "I ran for 30 minutes."

[0918] The device sends the collected behavioral and emotional data back to the server for use in the next analysis. The server re-analyzes the data based on the new behavioral and emotional data and generates new behavioral suggestions and feedback. For example, it might send positive feedback such as, "Your running pace is increasing, keep it up!"

[0919] Specific example

[0920] Let's say a user's goal is to "become a professional athlete" and they are recording their daily training and emotional data. The server will adjust the next training session based on the user's fatigue level and fluctuations in motivation. Similarly, if another user's goal is to "balance work and family life," the server will suggest actions to relax when stress levels are high, enabling more effective stress management.

[0921] As described above, this system uses user behavior data, goal setting, and emotional data to provide optimal action suggestions and feedback using generative AI, thereby supporting users in approaching their desired target model.

[0922] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0923] Step 1:

[0924] Users enter or automatically record their activity logs.

[0925] Users manually input data on their daily activities (e.g., travel, shopping, exercise, meals) through a dedicated application using their smartphone or PC. Additionally, the system automatically records distance traveled and exercise time using the smartphone's GPS and accelerometer. This process collects detailed activity logs of the user as input data.

[0926] Step 2:

[0927] The device sends behavioral data to the server.

[0928] The device encrypts the collected behavioral data using AES256 and periodically sends it to the server. The destination is the server, and encrypted behavioral data is sent as input data. The output is the behavioral data securely stored on the server.

[0929] Step 3:

[0930] The user sets the target model.

[0931] Users input their desired goals (e.g., "become a professional athlete" or "live a balanced life") in text format through the application. This allows the application to obtain information about their goals as input data.

[0932] Step 4:

[0933] The device sends target model information to the server.

[0934] The terminal encrypts the configured target model information using AES256 and sends it to the server. The input data is the encrypted target model information, and the transmission result is the target model information securely stored on the server.

[0935] Step 5:

[0936] Recognizing user emotions

[0937] The device is equipped with emotion recognition capabilities, utilizing facial recognition technology (e.g., OpenCV) and speech analysis (e.g., Google Cloud Speech-to-Text API) to monitor the user's emotional state. This allows for the acquisition of real-time emotion data as input.

[0938] Step 6:

[0939] Send emotional data to the server.

[0940] The device periodically encrypts recognized emotion data using AES256 and sends it to the server. The input data is encrypted emotion data, and the transmission result is the emotion data stored on the server.

[0941] Step 7:

[0942] The server analyzes behavioral data, target model information, and emotional data.

[0943] The server integrates and analyzes the received behavioral data, target model information, and emotional data. Specifically, it performs data cleansing and normalization as a preprocessing step, and uses an algorithm that considers past behavioral patterns, progress, and emotional states as an analysis step. The input data consists of behavioral data, target model information, and emotional data, and the analysis results in analytical data regarding the next action to be taken.

[0944] Step 8:

[0945] The server uses a generated AI model to create action suggestions.

[0946] The server uses a generative AI model (e.g., GPT-3) to suggest the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it might suggest "set a lighter training session next time." The input data is the analyzed data, and the output is a specific action suggestion. Example prompt: "The user's goal is to become a professional athlete. Based on the current activity log and emotional data, please suggest the optimal next action."

[0947] Step 9:

[0948] The server sends the proposal to the user's terminal.

[0949] The server sends the generated action suggestions to the terminal, where they are notified. The input data includes the action suggestions, and the output is the action suggestions notified to the user's terminal.

[0950] Step 10:

[0951] The device will notify you of the suggested content.

[0952] The device uses a reminder function to inform the user of the suggested actions. For example, a notification might appear saying, "Please go for a light 30-minute run today." The input data is the suggested action, and the output is the notified action suggestion.

[0953] Step 11:

[0954] The user performs an action and records the result.

[0955] The user performs the suggested action and logs the result in the app. For example, after running, the user might input "I ran for 30 minutes." The input data is a log of the execution result, and the output is the recorded action result.

[0956] Step 12:

[0957] The device sends behavioral results and emotional data to the server.

[0958] The terminal encrypts the collected behavioral result data and emotional data using AES256 and sends it to the server. The input data consists of behavioral result data and emotional data, and the output is the data stored on the server.

[0959] Step 13:

[0960] The server generates new suggestions and sends feedback.

[0961] The server re-analyzes the new behavioral and emotional data to generate new behavioral suggestions and feedback for the user. For example, it might send feedback such as, "Your running pace is increasing, keep it up." The input data consists of behavioral and emotional data, and the output is new behavioral suggestions and feedback.

[0962] (Application Example 2)

[0963] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0964] Conventional user assistance systems in autonomous vehicles were unable to consider user behavior data and emotional states in real time, making it difficult to suggest appropriate actions. Furthermore, they lacked the ability to suggest the optimal next course of action based on user-defined goals (role models), resulting in insufficient support for users achieving their objectives.

[0965] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for the user to input or automatically record their actions, means for transmitting the action data to the server, means for setting a role model that the user aims to emulate, means for transmitting the role model information to the server, means for recognizing the user's emotional state in real time, means for transmitting the emotional data to the server, means for generating the next action to be taken using a generated AI based on the action data, role model information and emotional data, means for notifying the user of the generated action suggestion, means for logging the results of the user's actions again and transmitting them to the server, means for generating a new suggestion based on the action results and emotional data and sending feedback to the user, and means for providing the action suggestion in cooperation with an autonomous mobile system. This makes it possible to analyze the user's emotional state and action data in real time and to quickly and effectively provide optimal action suggestions based on the user's goals.

[0966] A "user" is an entity that uses this system to input and record their own activity logs and receive suggestions.

[0967] "Behavioral data" refers to various actions performed by users (e.g., travel, shopping, exercise, eating, etc.) and related information.

[0968] A "server" is a device that collects and analyzes behavioral data, role model information, and emotional data, and then generates and provides the next course of action to be taken.

[0969] A "role model" refers to the goal or ideal state that the user aspires to, and action suggestions are made based on that.

[0970] "Emotional state" refers to the user's current emotional state (e.g., sad, happy, tired, etc.), and is data recognized in real time using sensors, etc.

[0971] "Generative AI" refers to an artificial intelligence model that analyzes collected data and automatically generates optimal action suggestions.

[0972] An "autonomous mobility system" is a system that has autonomous driving technology used by users for transportation, such as self-driving vehicles.

[0973] "Action suggestions" are suggestions for the next action to take, generated by a generating AI based on the user's behavioral data, emotional state, and role model information.

[0974] "Feedback" refers to evaluations and suggestions for future actions generated based on the results of the user's actions.

[0975] The configuration and operation of a specific system will be described as an embodiment for carrying out the present invention. This system uses behavioral data, emotional data, and role model information to generate action suggestions based on the role model that the user aspires to.

[0976] 1. Prerequisite System Configuration

[0977] The system is built upon the following key hardware and software components.

[0978] Hardware:

[0979] 1. Head-mounted display (HMD): Recognizes the user's emotional state in real time and displays action suggestions.

[0980] 2. ECU (Electronic Control Unit) of an autonomous vehicle: Collects vehicle driving data.

[0981] 3. Sensors for emotion recognition: such as heart rate sensors and cameras for facial expression recognition.

[0982] 4. Server: Cloud-based, it performs data analysis and generates action suggestions.

[0983] software:

[0984] 1. Emotion recognition software: For example, use the Emotion API from Microsoft Azure.

[0985] 2. Data analysis and generative AI models: for example, using TensorFlow, PyTorch, etc.

[0986] 3. Encryption protocol: For example, use TLS / SSL.

[0987] 4. Display software for the head-mounted display: For example, use Unity.

[0988] 2. Data collection and transmission

[0989] How users can input or automatically record behavioral data:

[0990] Users manually or automatically record their daily activities using smartphones, PCs, or HMDs.

[0991] For example, it can automatically record distance traveled and exercise time using GPS and accelerometers.

[0992] How devices send behavioral and emotional data to servers:

[0993] The collected behavioral and emotional data is encrypted using TLS / SSL and periodically sent to the server.

[0994] 3. Setting a role model

[0995] How users can set up role models:

[0996] Users set their own goals (for example, "develop safe driving habits") using a smartphone, PC, or HMD.

[0997] This configured role model information is encrypted before being sent to the server.

[0998] 4. How the Emotion Engine Works

[0999] How to recognize user emotions in real time:

[1000] The system uses sensors and cameras mounted on the HMD to monitor the user's facial expressions and heart rate.

[1001] For example, the Emotion API can be used to analyze a user's emotional state.

[1002] How to send emotional data to a server:

[1003] The recognized emotion data is periodically encrypted and sent to the server.

[1004] 5. Generating action proposals

[1005] How the server analyzes behavioral data, role model information, and sentiment data:

[1006] The server performs analysis based on past behavioral data and current emotional states. For example, it might use TensorFlow or PyTorch.

[1007] How a server uses generated AI to create action suggestions:

[1008] The generative AI proposes the optimal next course of action based on the analyzed data.

[1009] For example, if a user is feeling tired, the system might suggest, "We recommend taking a break at the next service area."

[1010] How the server sends the proposal to the user's terminal:

[1011] The generated action suggestions are notified to the user in real time via the HMD.

[1012] 6. Re-logging of feedback and action results

[1013] How users perform actions and record the results:

[1014] The user performs the suggested action and enters the result into the app.

[1015] How the device sends behavioral results and emotional data to the server:

[1016] The collected behavioral data and emotional data are sent back to the server.

[1017] How the server generates new suggestions and sends feedback:

[1018] The server re-analyzes the data based on the new information and generates new suggestions and feedback for the user.

[1019] Specific example:

[1020] Driver A set the goal of "reducing risks on the highway." When the driver feels fatigued while driving, the HMD detects an increase in heart rate and changes in facial expression. The server analyzes this and suggests, "The next rest area is 20 minutes away. We recommend taking a break soon."

[1021] Example of a prompt:

[1022] While driving an autonomous vehicle, suggest the optimal next course of action based on the role model set by the driver (e.g., safe driving).

[1023] Please consider the following data:

[1024] Driver's current emotional state (heart rate, facial recognition data)

[1025] Past driving behavior data (speed, braking, steering, etc.)

[1026] A role model set by the driver (e.g., risk reduction on highways)

[1027] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1028] (Process flow)

[1029] Step 1:

[1030] Input: User's daily activity data (e.g., distance traveled, exercise time).

[1031] Operation: The device collects activity data that the user inputs or automatically records using a smartphone, PC, or HMD. For example, it records distance traveled and exercise time using GPS and accelerometer sensors.

[1032] Output: Collected behavioral data.

[1033] Step 2:

[1034] Input: Collected behavioral data.

[1035] Operation: The device encrypts collected behavioral data and sends it to the server using TLS / SSL. For example, by sending data when connected to Wi-Fi at night, battery consumption can be minimized.

[1036] Output: Behavioral data sent to the server.

[1037] Step 3:

[1038] Input: User-defined role model information (e.g., goal of aiming for safe driving).

[1039] Operation: The user uses a smartphone, PC, or HMD to specifically define their target role model, and the device encrypts this information before sending it to the server.

[1040] Output: Role model information sent to the server.

[1041] Step 4:

[1042] Input: Behavioral data and role model information stored on the server side.

[1043] Operation: The server analyzes the user's past behavioral patterns and progress based on the received behavioral data and role model information. For example, it may use TensorFlow or PyTorch to analyze the data.

[1044] Output: Analysis results.

[1045] Step 5:

[1046] Input: Real-time user emotion data (e.g., heart rate, facial expression data).

[1047] Operation: The HMD uses sensors and cameras to recognize the user's emotional state (heart rate, facial expressions, etc.) in real time, encrypts the data, and sends it to the server.

[1048] Output: Emotional data sent to the server.

[1049] Step 6:

[1050] Input: Analysis results and real-time sentiment data.

[1051] Operation: The server uses a generative AI to generate the optimal next action based on analyzed behavioral data, role model information, and sentiment data. For example, it uses a generative AI model (e.g., GPT-4) to create prompt statements.

[1052] Output: Generated action suggestions.

[1053] Step 7:

[1054] Input: Generated action suggestions.

[1055] Operation: The server encrypts the generated action suggestion, sends it to the HMD in real time, and the HMD notifies the user. For example, it might display "We recommend taking a break at the next service area."

[1056] Output: Notification of suggested actions to the user.

[1057] Step 8:

[1058] Input: The result of the user's actions based on the suggestion.

[1059] Operation: The user performs the suggested action and inputs the result back into the HMD, smartphone, or PC. For example, after exercising, they might record "I ran for 30 minutes."

[1060] Output: The result of the input action.

[1061] Step 9:

[1062] Input: Action results and sentiment data recorded again as logs.

[1063] Operation: The device encrypts the collected behavioral data and emotional data and sends it back to the server.

[1064] Output: Action results and sentiment data sent to the server.

[1065] Step 10:

[1066] Input: New behavioral result data and sentiment data sent to the server.

[1067] Operation: The server re-analyzes the new behavioral result data and sentiment data to generate new behavioral suggestions and feedback. For example, it might generate feedback such as, "Your running pace is increasing, so keep it up."

[1068] Output: New action suggestions and feedback.

[1069] The system operates in the manner described above, enabling it to provide real-time suggestions for optimal actions based on the role model the user aspires to.

[1070] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1071] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1072] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1073] [Third Embodiment]

[1074] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1075] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1076] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1077] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1078] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1079] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1080] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1081] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1082] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1083] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1084] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1085] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1086] The operation of a specific system will be described as an embodiment for carrying out the present invention.

[1087] This system collects users' activity logs and suggests their next course of action based on a pre-configured role model. It is implemented through the cooperation of the user, device, and server according to the following processing flow.

[1088] 1. Collection and transmission of user activity logs

[1089] Users enter or automatically record their activity logs.

[1090] Users record their daily activities, such as travel, shopping, exercise, and meals, using their smartphones or PCs. For example, they can automatically record running distance and time using their smartphone's accelerometer and GPS function as an exercise log. They can also manually enter meal details and shopping lists into the app.

[1091] The device sends behavioral data to the server.

[1092] The collected behavioral data is encrypted and then periodically sent to the server. This ensures the security of the data.

[1093] 2. Setting a role model

[1094] Users set role models

[1095] Users can specifically define their role model. For example, they can select a professional athlete or a successful business person and input their characteristics and goals.

[1096] The device sends role model information to the server.

[1097] The configured role model information is encrypted and sent to the server, just like the user's behavioral data.

[1098] 3. Generating the next course of action

[1099] The server analyzes behavioral data and role model information.

[1100] The server performs analysis based on the received behavioral data and role model information. In particular, it compares past behavioral patterns with current progress to generate the optimal actions for the user to move closer to their goals.

[1101] The server uses AI to generate action suggestions.

[1102] The generating AI considers behavioral data and role model information to generate specific next steps. For example, if it determines that adding weight training will bring the user closer to their goal, it will create a suggestion such as "Add 30 minutes of weight training to your next workout."

[1103] The server sends the proposal to the user's terminal.

[1104] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[1105] 4. Suggestions for Users

[1106] The device will notify you of the suggested content.

[1107] The user's device will notify them of the suggestions using its notification function. This ensures that the user does not forget what action they should take next.

[1108] 5. Re-logging of feedback and action results

[1109] The user performs an action and records the result.

[1110] The user performs the suggested action and enters the results into the app. For example, if they perform the suggested weight training, they will record the duration and details of the workout again.

[1111] The device sends the results of its actions to the server.

[1112] The collected behavioral data is sent back to the server and used for further analysis.

[1113] The server generates new suggestions and sends feedback.

[1114] The server performs a re-analysis based on the newly collected behavioral data and generates feedback and suggestions for the next action for the user. This is then sent to the user's terminal for notification.

[1115] Specific example

[1116] Example 1: If you aim to become a professional athlete

[1117] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[1118] Example 2: When balancing work and family life

[1119] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[1120] As described above, this system supports users in approaching their desired role models by providing action suggestions and feedback using generated AI based on user behavior data and goal settings.

[1121] The following describes the processing flow.

[1122] Step 1:

[1123] Users enter or automatically record their activity logs.

[1124] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. Alternatively, the system can automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[1125] Step 2:

[1126] The device sends behavioral data to the server.

[1127] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent at night when connected to Wi-Fi to minimize battery consumption.

[1128] Step 3:

[1129] Users set role models

[1130] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[1131] Step 4:

[1132] The device sends role model information to the server.

[1133] The configured role model information is encrypted before being sent to the server.

[1134] Step 5:

[1135] The server analyzes behavioral data and role model information.

[1136] The server performs analysis based on the received behavioral data and role model information. This involves analyzing the user's past behavioral patterns and progress to determine what actions are necessary to move closer to the role model.

[1137] Step 6:

[1138] The server uses AI to generate action suggestions.

[1139] The generating AI uses analyzed behavioral data and role model information to suggest the optimal next action for the user. For example, it can generate specific suggestions such as, "Add an hour of running next weekend."

[1140] Step 7:

[1141] The server sends the proposal to the user's terminal.

[1142] The generated action suggestions are sent to the user's device, allowing the user to see the suggested actions.

[1143] Step 8:

[1144] The device will notify you of the suggested content.

[1145] The user's device will notify them of the suggestion using a reminder function. For example, a notification might appear saying, "Please go for a 30-minute run every morning at 6:00 AM."

[1146] Step 9:

[1147] The user performs an action and records the result.

[1148] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[1149] Step 10:

[1150] The device sends the results of its actions to the server.

[1151] The collected behavioral data is sent back to the server. This data is also encrypted before transmission and used for further analysis.

[1152] Step 11:

[1153] The server generates new suggestions and sends feedback.

[1154] The server re-analyzes the new behavioral data and generates feedback and suggestions for the next action for the user. For example, this may include positive feedback such as, "Your running pace is increasing, keep it up!"

[1155] Step 12:

[1156] The device will notify you of the feedback.

[1157] The server sends feedback to the user's device and notifies the user again. This allows the user to effectively continue taking action towards their goal.

[1158] (Example 1)

[1159] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1160] Conventional behavioral suggestion systems had problems such as not being able to fully utilize user behavior data and not being able to effectively provide optimal behavioral suggestions that corresponded to the user's goals. In particular, ensuring the security of user behavior data and improving the accuracy of behavioral suggestions were challenges.

[1161] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1162] In this invention, the server includes means for the user to input or automatically record their actions; means for encrypting the action data and transmitting it to the server; means for setting a role model that the user aims to emulate; means for encrypting the role model information and transmitting it to the server; means for generating the next action to be taken using a generated AI model based on the action data and role model information; means for notifying the user's terminal of the generated action suggestion; means for logging the results of the user's actions again, encrypting them, and transmitting them to the server; and means for generating new suggestions based on the action results and transmitting feedback to the user's terminal. This makes it possible to provide highly accurate and optimal action suggestions that match the user's goals while ensuring the security of the user's action data.

[1163] A "user" refers to an individual who uses this system to record behavioral data and aims to achieve their goals.

[1164] "Behavioral data" refers to information about daily activities such as movement, exercise, and eating that users record.

[1165] "Encryption" refers to technologies used to protect behavioral data and role model information from being deciphered by third parties.

[1166] A "server" refers to a computer system that receives and analyzes behavioral data and role model information sent by users, and generates and sends appropriate action suggestions.

[1167] A "role model" refers to an ideal individual or ideal figure that a user aspires to be like, whose characteristics and goals are set for them.

[1168] A "generative AI model" refers to an artificial intelligence model that generates suggestions for the next course of action based on user behavior data and role model information.

[1169] "Action suggestions" refer to specific action instructions generated by the server to support the user in achieving their goals.

[1170] "Feedback" refers to notifying the user of newly generated suggestions or evaluations based on the results of the actions they have taken.

[1171] A "sensor" refers to an electronic device used to acquire user movement and location information.

[1172] This invention is a system that allows users to record their own actions, generate action suggestions to help them move closer to their target role model, and support their implementation. This system operates through the coordinated efforts of the user, terminal, and server.

[1173] First, users record their daily activity logs using their smartphones or PCs. Specifically, they can automatically record running distance and time using their smartphone's accelerometer and GPS function. They can also manually input meal details and shopping lists into the app. The device encrypts the collected activity data and periodically sends it to the server using secure protocols such as HTTPS.

[1174] Next, users set their desired role model. In the dedicated app's settings screen, they can select a role model such as a "professional athlete" or a "successful business person," and input their characteristics and goals. This role model information, like behavioral data, is encrypted and sent to the server.

[1175] The behavioral data and role model information sent to the server are subject to analysis. Based on this information, the server uses a generative AI model to generate specific actions that the user should take next. For example, if the user aims to become a "professional athlete," the AI ​​will generate a specific suggestion such as "add 30 minutes of weight training to your next training session." This suggestion is sent from the server to the user's device and notified to the user. The notification can also be set as a time-based reminder.

[1176] The user receives action suggestions from their device, performs them, and records the results again in the app. For example, if the user performs a suggested weight training exercise, the user records the duration and content of the exercise again and sends it to the server via their device. This action result data is then analyzed again on the server and sent back to the user as new suggestions or feedback. This allows the user to continuously perform optimal actions.

[1177] Specific example

[1178] Example 1: If you aim to become a professional athlete

[1179] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[1180] Example 2: When balancing work and family life

[1181] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[1182] Example of a prompt

[1183] Activity data: "Distance traveled: 5km, Exercise time: 30 minutes, Meal content: Salad"

[1184] Role model: "Goal: Professional athlete, Characteristics: Daily training: 2 hours, Diet: High protein, low calorie"

[1185] Please suggest the next action this user should take.

[1186] Thus, the system of the present invention supports users in approaching their desired role model by providing action suggestions and feedback using a generated AI model based on the user's behavioral data and goal settings.

[1187] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1188] Step 1:

[1189] Users enter or automatically record their activity logs.

[1190] Users record their daily activities using a dedicated app on their smartphone or PC. Specifically, one method involves automatically recording running distance and time using the smartphone's accelerometer and GPS function. Users can also manually enter details of meals and shopping lists into the app.

[1191] Input: User behavior data (e.g., distance traveled, exercise time, diet)

[1192] Output: Recorded behavioral data

[1193] Specific operation: The smartphone's accelerometer detects the user's steps, and the GPS function measures the distance traveled. The user also inputs details of their meals through the app's UI.

[1194] Step 2:

[1195] The device encrypts behavioral data and sends it to the server.

[1196] The device encrypts user behavior data and periodically sends it to the server using secure protocols such as HTTPS. This process ensures data security.

[1197] Input: Recorded behavioral data

[1198] Output: Encrypted behavioral data

[1199] Specific operation: A data encryption module operates within the app, encrypting behavioral data using encryption algorithms such as AES. The data is then sent to the server using HTTPS.

[1200] Step 3:

[1201] Set a role model that the user aspires to be.

[1202] Users select a role model, such as a "professional athlete" or a "successful business person," from the settings screen of the dedicated app and input their characteristics and goals. This information, like behavioral data, is encrypted.

[1203] Input: User role model information (e.g., goals, characteristics)

[1204] Output: Configured role model information

[1205] Specific operation: The user enters role model information using dropdown lists or text boxes in the app's settings screen.

[1206] Step 4:

[1207] The device encrypts the role model information and sends it to the server.

[1208] As mentioned above, the device encrypts the role model information and sends it to the server using the HTTPS protocol.

[1209] Input: Configured role model information

[1210] Output: Encrypted role model information

[1211] Specific operation: The in-app data encryption module encrypts the role model information using an encryption algorithm and sends it to the server via HTTPS.

[1212] Step 5:

[1213] The server analyzes behavioral data and role model information.

[1214] The server uses a dedicated analysis algorithm to analyze the received behavioral data and role model information. This helps determine the optimal actions for the user to take to achieve their goals.

[1215] Input: Encrypted behavioral data, role model information

[1216] Output: Data analysis results

[1217] Specific operation: The analysis engine runs on the server and compares behavioral data with role model information. It also compares past behavioral patterns with current progress.

[1218] Step 6:

[1219] The server uses a generated AI model to create action suggestions.

[1220] The server uses a generated AI model to create specific action suggestions based on the data analysis results described above.

[1221] Input: Data analysis results

[1222] Output: Action Suggestions

[1223] Specific operation: The generating AI model generates optimal action suggestions for the user based on the results of data analysis (e.g., "Add 30 minutes of weight training to your next workout").

[1224] Step 7:

[1225] The server sends the proposal to the user's terminal.

[1226] The generated action suggestions are sent from the server to the user's terminal in the form of a notification.

[1227] Input: Action suggestion

[1228] Output: Action suggestions sent to the user's device

[1229] Specific operation: The server sends an action suggestion to the device, and the device displays it to the user as a notification. The notification can also be set as a time-based reminder.

[1230] Step 8:

[1231] The user performs an action and records the result.

[1232] The user performs the suggested action and records the result again in the app.

[1233] Input: Proposed action

[1234] Output: Results of the actions performed

[1235] Specific actions: The user performs the suggested actions and enters the results using a smartphone or PC app. This includes details such as exercise time and content.

[1236] Step 9:

[1237] The device encrypts the results of its actions and sends them to the server.

[1238] The device encrypts the collected behavioral data and sends it back to the server.

[1239] Input: Results of the actions performed

[1240] Output: Encrypted behavioral result data

[1241] Specific operation: The in-app data encryption module runs, encrypts the action result data, and sends it to the server.

[1242] Step 10:

[1243] The server generates new suggestions and sends feedback.

[1244] The server re-analyzes the newly collected behavioral data, generates new suggestions and feedback, and sends them to the user's device.

[1245] Input: Encrypted behavioral data

[1246] Output: New suggestions and feedback

[1247] Specific operation: The server performs a re-analysis and generates new action suggestions and feedback. The generated information is sent to the user's device in the form of a notification.

[1248] (Application Example 1)

[1249] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1250] The challenge lies in creating an operational support system that safely and efficiently manages the operation of autonomous vehicles while also considering the driver's health. Furthermore, it is necessary to achieve continuously optimized operational management by specifically proposing the driving behaviors that drivers should aim for and incorporating the results of their execution.

[1251] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1252] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a role model that the user aspires to; means for transmitting the role model information to the server; means for generating the next action to be taken using a generative AI based on the action data and role model information; means for notifying the user of the generated action suggestions; means for logging the results of the user's actions again and transmitting them to the server; means for generating new suggestions based on the action results and sending feedback to the user; means for automatically collecting operation data of the autonomous vehicle and driver action data, encrypting them, and transmitting them to the server; means for generating operation and driver action suggestions using a generative AI model based on the optimal operation pattern and driver model set by the administrator; and means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses. This makes it possible to perform advanced operation management and driver support for autonomous vehicles and improve the safety and efficiency of operations.

[1253] "Means for users to input or automatically record their own actions" refers to devices or software that allow users to manually input their own behavioral data or to automatically record behavioral data using a smartphone or sensors.

[1254] "Means for transmitting the aforementioned behavioral data to the server" refers to a device or software that securely transmits the collected and recorded behavioral data to the server via the internet.

[1255] "Means for setting a role model that the user aspires to" refers to a device or software that allows the user to specifically define the person or ideal they aspire to be and input that information.

[1256] "Means for transmitting the role model information to the server" refers to a device or software that securely transmits the configured role model information to the server via the internet.

[1257] "Means for generating the next action to be taken using a generative AI based on the aforementioned behavioral data and role model information" refers to an algorithm and system that analyzes the collected behavioral data and role model information and calculates the optimal next action using a generative AI model.

[1258] "Means for notifying the user of generated action suggestions" refers to a notification system for informing the user of action suggestions created by the generating AI, and is a device or software that uses a smartphone, smart glasses, in-vehicle display, etc.

[1259] "Means for logging the results of user actions and sending them back to the server" refers to a device or software that records the results of user actions and sends that data back to the server.

[1260] "Means for generating new suggestions based on the aforementioned action results and sending feedback to the user" refers to a device or software that re-analyzes the recorded action result data, generates new action suggestions, and notifies the user.

[1261] "Means for automatically collecting operational data of autonomous vehicles and driver behavior data, encrypting them, and then transmitting them to a server" refers to a device or software that collects various data related to the operation of autonomous vehicles and driver behavior data using sensors and cameras, encrypts them, and then transmits them to a server.

[1262] "Means for generating operational and driver action suggestions using a generation AI model based on optimal operational patterns and driver models set by the administrator" refers to a device or software that generates operational and driver action suggestions using a generation AI model based on operational pattern and driver model information set by the administrator.

[1263] "Means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses" refers to a device or software that notifies the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses.

[1264] This invention provides a system for advanced operational management and driver assistance of autonomous vehicles. This system allows users (operation managers and drivers) to collect their own behavioral logs and propose the next course of action based on their configured optimal operating patterns and driver models. This is achieved through the cooperation of a server, terminals, and autonomous vehicles, following the processing flow outlined below.

[1265] 1. Collection and transmission of user activity logs

[1266] Various sensors (cameras, accelerometers, GPS, etc.) within the autonomous vehicle are used as a means to automatically record user activity logs. These sensors record operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. The collected data is encrypted and then periodically transmitted to a server. This ensures the security of the data.

[1267] 2. Setting a role model

[1268] As a means for administrators to set role models, they can specifically define the optimal driving patterns and driver models they aim for. For example, this could include patterns that prioritize safe driving or patterns that emphasize efficiency. The set role model information is encrypted and sent to the server, just like user behavior data.

[1269] 3. Generating the next course of action

[1270] The server analyzes behavioral data and role model information. This analysis uses a generated AI model based on the behavioral data and role model information. The server compares past behavioral patterns with current progress and generates the optimal actions for the user to get closer to their goal. For example, if a driver hasn't taken a two-hour break, it will create a specific suggestion such as "Take a 15-minute break at the next rest stop."

[1271] 4. Suggestions for Users

[1272] The generated action suggestions are communicated to the vehicle's information display system or the driver's smart glasses. This allows the driver to properly understand and act on what to do next, even while driving.

[1273] 5. Re-logging of feedback and action results

[1274] There is a mechanism for automatically re-logging data as a means of recording the results when the user (driver) performs a suggested action. For example, in-vehicle sensors detect and record the driver's resting status. The collected action result data is sent back to the server and used for the next analysis. The server performs a re-analysis based on the newly collected action result data, generates new suggestions, and sends feedback.

[1275] Technology for realizing functionality

[1276] This system is constructed using the following technologies:

[1277] Hardware:

[1278] Various sensors installed inside the vehicle (camera, accelerometer, GPS, etc.)

[1279] Smart glasses for drivers (e.g., Google Glass)

[1280] Information display system for autonomous vehicles

[1281] software:

[1282] Custom API for data collection and transmission (encryption supported)

[1283] Cloud tools (e.g., Google Cloud AutoML, AWS SageMaker) are used to analyze behavioral data and operational models.

[1284] Generative AI models include natural language generation models (e.g., OpenAI's GPT model).

[1285] Specific example

[1286] For example, the following prompt statements are used to input into the generated AI model.

[1287] Prompt message:

[1288] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[1289] Output of the generative AI model:

[1290] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[1291] As described above, the system for implementing the present invention enables advanced operational management and driver assistance for autonomous vehicles. This is expected to improve the safety and efficiency of operations.

[1292] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1293] Step 1:

[1294] The user records their activity log. Various sensors in the autonomous vehicle (cameras, accelerometers, GPS, etc.) collect operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. Operational data and driver status data are obtained as input data. This data is encrypted.

[1295] Output: Encrypted operational data and driver status data.

[1296] Step 2:

[1297] The terminal sends the encrypted data to the server. The data is securely transferred to the server using a transmission protocol (e.g., HTTPS).

[1298] Input: Encrypted operational data and driver status data.

[1299] Output: Encrypted data stored on the server.

[1300] Step 3:

[1301] The user (flight manager) sets the role model they aspire to. The manager uses a web portal or application to specify the optimal ferry pattern and driver model (safe driving, efficient driving, etc.). The configuration information is encrypted and sent to the server.

[1302] Input: Role model configuration information.

[1303] Output: Role model configuration information stored on the server.

[1304] Step 4:

[1305] The server analyzes behavioral data and role model information. Cloud tools (such as Google Cloud AutoML and AWS SageMaker) are used to analyze and compare this data in real time.

[1306] Inputs: Operational data, driver status data, role model information.

[1307] Output: Analysis results.

[1308] Step 5:

[1309] The server uses a generative AI model to generate the next action to take. It inputs a prompt sentence into a natural language generation model (e.g., OpenAI's GPT model) and obtains an appropriate action suggestion.

[1310] Input: Analysis results.

[1311] Output: Generated action suggestions.

[1312] Step 6:

[1313] The terminal notifies the information display system in the autonomous vehicle or the driver's smart glasses of the generated action suggestions. It also provides the function to notify the user visually or audibly.

[1314] Input: Generated action suggestions.

[1315] Output: Notification to the driver.

[1316] Step 7:

[1317] When the user (driver) takes action based on the suggestion, the result of that action is recorded again as a log. The results of the actions are automatically collected by sensors and cameras inside the vehicle.

[1318] Input: Driver action result data.

[1319] Output: Log data.

[1320] Step 8:

[1321] The terminal sends the aforementioned log data to the server. The server analyzes it again, generates new action suggestions, and sends feedback.

[1322] Input: Log data.

[1323] Output: New action suggestions and feedback.

[1324] As a concrete example, enter the following prompt message into the generative AI model.

[1325] Prompt message:

[1326] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[1327] Output of the generative AI model:

[1328] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[1329] By following these steps, advanced operational management and driver assistance for autonomous vehicles can be achieved.

[1330] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1331] The operation of a specific system will be described as an embodiment of the present invention.

[1332] This system collects the user's own behavioral logs and suggests the next course of action based on a pre-set role model. It also incorporates an emotion engine that recognizes the user's emotions and adjusts the action suggestions and feedback accordingly. This is achieved through the cooperation of the user, device, and server according to the processing flow outlined below.

[1333] 1. Collection and transmission of user activity logs

[1334] Users enter or automatically record their activity logs.

[1335] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. Alternatively, the system can automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[1336] The device sends behavioral data to the server.

[1337] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent at night when connected to Wi-Fi to minimize battery consumption.

[1338] 2. Setting a role model

[1339] Users set role models

[1340] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[1341] The device sends role model information to the server.

[1342] The configured role model information is encrypted before being sent to the server.

[1343] 3. How the Emotion Engine Works

[1344] Recognizing user emotions

[1345] The user's device is equipped with emotion recognition capabilities, allowing for real-time monitoring of the user's current emotional state through technologies such as facial recognition and voice analysis.

[1346] Send emotional data to the server.

[1347] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[1348] 4. Generating the next course of action

[1349] The server analyzes behavioral data, role model information, and emotional data.

[1350] The server analyzes the received behavioral data, role model information, and emotional data. In particular, it generates the most effective action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[1351] The server uses AI to generate action suggestions.

[1352] The generating AI uses multiple analyzed data points to suggest the optimal next action for the user. For example, if the user is feeling fatigued, it might suggest "setting a lighter training session for the next workout."

[1353] The server sends the proposal to the user's terminal.

[1354] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[1355] 5. Suggestions for Users

[1356] The device will notify you of the suggested content.

[1357] The user's device uses a reminder function to notify them of the suggestions. For example, a notification might appear saying, "Please go for a light 30-minute run today." Because the suggestions are adjusted based on emotional data, users can take appropriate action.

[1358] 6. Re-logging of feedback and action results

[1359] The user performs an action and records the result.

[1360] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[1361] The device sends behavioral results and emotional data to the server.

[1362] The collected behavioral and emotional data are sent back to the server and used for further analysis.

[1363] The server generates new suggestions and sends feedback.

[1364] The server re-analyzes the new behavioral and emotional data to generate feedback and suggestions for the next action for the user. For example, this might include positive feedback such as, "Your running pace is increasing, keep it up!"

[1365] For example, if a user aims to become a professional athlete and records daily training and emotional data, the server will adjust the training content based on the user's fatigue level and fluctuations in motivation. Similarly, for users aiming to balance work and family life, suggestions for relaxation will be provided when stress levels are high, enabling more effective stress management.

[1366] As described above, this system supports users in approaching their desired role models by providing behavioral suggestions and feedback using generated AI based on user behavioral data, goal settings, and emotional data.

[1367] The following describes the processing flow.

[1368] Step 1:

[1369] Users enter or automatically record their activity logs.

[1370] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. The system can also automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[1371] Step 2:

[1372] The device sends behavioral data to the server.

[1373] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent in batches when connected to Wi-Fi at night to minimize battery drain.

[1374] Step 3:

[1375] Users set role models

[1376] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[1377] Step 4:

[1378] The device sends role model information to the server.

[1379] The configured role model information is encrypted before being sent to the server.

[1380] Step 5:

[1381] Recognizing user emotions

[1382] The device uses emotion recognition to monitor the user's emotional state in real time through facial expression and voice analysis. Emotion recognition often utilizes the device's camera and microphone.

[1383] Step 6:

[1384] The device sends emotional data to the server.

[1385] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[1386] Step 7:

[1387] The server analyzes behavioral data, role model information, and emotional data.

[1388] The server analyzes the received behavioral data, role model information, and emotional data. For example, it considers the user's past behavioral patterns, progress, and current emotional state to suggest the most effective next action.

[1389] Step 8:

[1390] The server uses AI to generate action suggestions.

[1391] The generating AI suggests the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it will generate a specific suggestion such as "set a lighter training session next time."

[1392] Step 9:

[1393] The server sends the proposal to the user's terminal.

[1394] The generated action suggestions are sent to the user's device and displayed as notifications.

[1395] Step 10:

[1396] The device will notify you of the suggested content.

[1397] The user's device will notify them of the suggestions using a reminder function. For example, a notification might appear saying, "Please go for a light 30-minute run today." Because the suggestions are adjusted based on emotional data, users can take appropriate action.

[1398] Step 11:

[1399] The user performs an action and records the result.

[1400] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[1401] Step 12:

[1402] The device sends behavioral results and emotional data to the server.

[1403] The collected behavioral and emotional data are sent back to the server and used for further analysis.

[1404] Step 13:

[1405] The server generates new suggestions and sends feedback.

[1406] The server re-analyzes the newly collected behavioral and emotional data to generate feedback and suggestions for the next action for the user. For example, it might generate positive feedback such as, "Your running pace is increasing, keep it up!"

[1407] Step 14:

[1408] The device will notify you of the feedback.

[1409] The server sends feedback to the user's device and notifies the user again. This allows the user to continue taking effective action towards their goal.

[1410] (Example 2)

[1411] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1412] Conventional behavior suggestion systems provide suggestions based on user behavior data and goal setting, but they do not take into account the user's emotional state. As a result, it is difficult for users to effectively carry out the suggested actions, and there is a problem in that optimal behavior suggestions cannot be made according to individual circumstances.

[1413] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1414] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a target model that the user aims for; means for transmitting the target model information to the server; means for recognizing the user's emotional state and transmitting that data to the server; means for performing analysis based on the action data, target model information, and emotional data and generating the next action to be taken using a generated AI model; means for notifying the user of the generated action suggestions; means for recording the results of the user's actions and transmitting them to the server; and means for generating new suggestions based on the action results and sending feedback to the user. This makes it possible to provide individual action suggestions that take into account the user's emotional state.

[1415] A "user" refers to an individual who uses the system to input and record their own behavioral and emotional data.

[1416] "Behavioral data" refers to information recorded by users about their daily activities, including distance traveled, exercise time, and diet.

[1417] A "server" refers to a computer system that receives behavioral data, role model information, and emotional data, and performs analysis, generates behavioral suggestions, and sends feedback.

[1418] A "goal model" refers to information that shows the specific goals and desired outcomes that a user aims to achieve.

[1419] "Emotional state" refers to data that represents the user's current emotions and mood, and is collected using facial recognition technology and voice analysis.

[1420] A "generative AI model" refers to an artificial intelligence model that generates the optimal next action to take based on the data it receives.

[1421] "Action suggestion" refers to the optimal next action that the user should take, as suggested by the generative AI model.

[1422] "Action results" refer to the data recorded after a user performs a suggested action.

[1423] "Feedback" refers to new suggestions or evaluations that a server generates for a user based on their actions.

[1424] This invention is a system that collects a user's own behavioral logs and suggests the next action to take based on a set goal model. Furthermore, it is equipped with an emotion engine that recognizes the user's emotional state and adjusts the action suggestions and feedback accordingly. This system is realized through the cooperation of the user, terminal, and server, as shown below.

[1425] Collection and transmission of user behavior logs

[1426] Users manually input their daily activities (e.g., travel, shopping, exercise, meals, etc.) through a dedicated application using their smartphone or PC. Alternatively, the system can automatically record distance traveled and exercise time using the GPS and accelerometer sensors (e.g., common location services and sensors) built into the smartphone.

[1427] The device periodically sends collected behavioral data to the server. AES256 encryption is used during transmission to ensure data security. The system is designed to minimize battery consumption, especially by transmitting data at night while connected to Wi-Fi.

[1428] Setting a role model

[1429] Through using the application, users set specific goal models they aspire to. For example, they might input goals such as "become a professional athlete" or "live a balanced life."

[1430] The terminal encrypts the configured target model information and sends it to the server.

[1431] How the emotion engine works

[1432] The user's device is equipped with emotion recognition capabilities, utilizing facial recognition technology (such as OpenCV) and speech analysis (such as Google Cloud Speech-to-Text API) to monitor the user's current emotional state in real time. This function allows for understanding the user's stress level and relaxation state.

[1433] Emotional data is periodically sent to the server and used for analysis there.

[1434] Generating the next course of action

[1435] The server analyzes the received behavioral data, target model information, and emotional data. Specifically, it generates action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[1436] The server uses a generative AI (such as GPT-3) to suggest the best course of action next. For example, if the user is feeling fatigued, it might suggest "set a lighter training session next time." Examples of prompts in this case include:

[1437] "Our goal is for users to become professional athletes. Based on their current behavioral logs and emotional data, please suggest the optimal next course of action."

[1438] The generated action suggestions are sent from the server to the user's device and displayed as notifications. These suggestions can also be set as time-based reminders.

[1439] Suggestions and feedback for users

[1440] The user's device uses a reminder function to notify them of the suggestions. For example, a notification might appear saying, "Please go for a light 30-minute run today." Suggestions based on emotional data enable users to take appropriate actions.

[1441] The user performs the suggested action and logs the result to the application. For example, after running, they might type "I ran for 30 minutes."

[1442] The device sends the collected behavioral and emotional data back to the server for use in the next analysis. The server re-analyzes the data based on the new behavioral and emotional data and generates new behavioral suggestions and feedback. For example, it might send positive feedback such as, "Your running pace is increasing, keep it up!"

[1443] Specific example

[1444] Let's say a user's goal is to "become a professional athlete" and they are recording their daily training and emotional data. The server will adjust the next training session based on the user's fatigue level and fluctuations in motivation. Similarly, if another user's goal is to "balance work and family life," the server will suggest actions to relax when stress levels are high, enabling more effective stress management.

[1445] As described above, this system uses user behavior data, goal setting, and emotional data to provide optimal action suggestions and feedback using generative AI, thereby supporting users in approaching their desired target model.

[1446] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1447] Step 1:

[1448] Users enter or automatically record their activity logs.

[1449] Users manually input data on their daily activities (e.g., travel, shopping, exercise, meals) through a dedicated application using their smartphone or PC. Additionally, the system automatically records distance traveled and exercise time using the smartphone's GPS and accelerometer. This process collects detailed activity logs of the user as input data.

[1450] Step 2:

[1451] The device sends behavioral data to the server.

[1452] The device encrypts the collected behavioral data using AES256 and periodically sends it to the server. The destination is the server, and encrypted behavioral data is sent as input data. The output is the behavioral data securely stored on the server.

[1453] Step 3:

[1454] The user sets the target model.

[1455] Users input their desired goals (e.g., "become a professional athlete" or "live a balanced life") in text format through the application. This allows the application to obtain information about their goals as input data.

[1456] Step 4:

[1457] The device sends target model information to the server.

[1458] The terminal encrypts the configured target model information using AES256 and sends it to the server. The input data is the encrypted target model information, and the transmission result is the target model information securely stored on the server.

[1459] Step 5:

[1460] Recognizing user emotions

[1461] The device is equipped with emotion recognition capabilities, utilizing facial recognition technology (e.g., OpenCV) and speech analysis (e.g., Google Cloud Speech-to-Text API) to monitor the user's emotional state. This allows for the acquisition of real-time emotion data as input.

[1462] Step 6:

[1463] Send emotional data to the server.

[1464] The device periodically encrypts recognized emotion data using AES256 and sends it to the server. The input data is encrypted emotion data, and the transmission result is the emotion data stored on the server.

[1465] Step 7:

[1466] The server analyzes behavioral data, target model information, and emotional data.

[1467] The server integrates and analyzes the received behavioral data, target model information, and emotional data. Specifically, it performs data cleansing and normalization as a preprocessing step, and uses an algorithm that considers past behavioral patterns, progress, and emotional states as an analysis step. The input data consists of behavioral data, target model information, and emotional data, and the analysis results in analytical data regarding the next action to be taken.

[1468] Step 8:

[1469] The server uses a generated AI model to create action suggestions.

[1470] The server uses a generative AI model (e.g., GPT-3) to suggest the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it might suggest "set a lighter training session next time." The input data is the analyzed data, and the output is a specific action suggestion. Example prompt: "The user's goal is to become a professional athlete. Based on the current activity log and emotional data, please suggest the optimal next action."

[1471] Step 9:

[1472] The server sends the proposal to the user's terminal.

[1473] The server sends the generated action suggestions to the terminal, where they are notified. The input data includes the action suggestions, and the output is the action suggestions notified to the user's terminal.

[1474] Step 10:

[1475] The device will notify you of the suggested content.

[1476] The device uses a reminder function to inform the user of the suggested actions. For example, a notification might appear saying, "Please go for a light 30-minute run today." The input data is the suggested action, and the output is the notified action suggestion.

[1477] Step 11:

[1478] The user performs an action and records the result.

[1479] The user performs the suggested action and logs the result in the app. For example, after running, the user might input "I ran for 30 minutes." The input data is a log of the execution result, and the output is the recorded action result.

[1480] Step 12:

[1481] The device sends behavioral results and emotional data to the server.

[1482] The terminal encrypts the collected behavioral result data and emotional data using AES256 and sends it to the server. The input data consists of behavioral result data and emotional data, and the output is the data stored on the server.

[1483] Step 13:

[1484] The server generates new suggestions and sends feedback.

[1485] The server re-analyzes the new behavioral and emotional data to generate new behavioral suggestions and feedback for the user. For example, it might send feedback such as, "Your running pace is increasing, keep it up." The input data consists of behavioral and emotional data, and the output is new behavioral suggestions and feedback.

[1486] (Application Example 2)

[1487] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1488] Conventional user assistance systems in autonomous vehicles were unable to consider user behavior data and emotional states in real time, making it difficult to suggest appropriate actions. Furthermore, they lacked the ability to suggest the optimal next course of action based on user-defined goals (role models), resulting in insufficient support for users achieving their objectives.

[1489] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for the user to input or automatically record their actions, means for transmitting the action data to the server, means for setting a role model that the user aims to emulate, means for transmitting the role model information to the server, means for recognizing the user's emotional state in real time, means for transmitting the emotional data to the server, means for generating the next action to be taken using a generated AI based on the action data, role model information and emotional data, means for notifying the user of the generated action suggestion, means for logging the results of the user's actions again and transmitting them to the server, means for generating a new suggestion based on the action results and emotional data and sending feedback to the user, and means for providing the action suggestion in cooperation with an autonomous mobile system. This makes it possible to analyze the user's emotional state and action data in real time and to quickly and effectively provide optimal action suggestions based on the user's goals.

[1490] A "user" is an entity that uses this system to input and record their own activity logs and receive suggestions.

[1491] "Behavioral data" refers to various actions performed by users (e.g., travel, shopping, exercise, eating, etc.) and related information.

[1492] A "server" is a device that collects and analyzes behavioral data, role model information, and emotional data, and then generates and provides the next course of action to be taken.

[1493] A "role model" refers to the goal or ideal state that the user aspires to, and action suggestions are made based on that.

[1494] "Emotional state" refers to the user's current emotional state (e.g., sad, happy, tired, etc.), and is data recognized in real time using sensors, etc.

[1495] "Generative AI" refers to an artificial intelligence model that analyzes collected data and automatically generates optimal action suggestions.

[1496] An "autonomous mobility system" is a system that has autonomous driving technology used by users for transportation, such as self-driving vehicles.

[1497] "Action suggestions" are suggestions for the next action to take, generated by a generating AI based on the user's behavioral data, emotional state, and role model information.

[1498] "Feedback" refers to evaluations and suggestions for future actions generated based on the results of the user's actions.

[1499] The configuration and operation of a specific system will be described as an embodiment for carrying out the present invention. This system uses behavioral data, emotional data, and role model information to generate action suggestions based on the role model that the user aspires to.

[1500] 1. Prerequisite System Configuration

[1501] The system is built upon the following key hardware and software components.

[1502] Hardware:

[1503] 1. Head-mounted display (HMD): Recognizes the user's emotional state in real time and displays action suggestions.

[1504] 2. ECU (Electronic Control Unit) of an autonomous vehicle: Collects vehicle driving data.

[1505] 3. Sensors for emotion recognition: such as heart rate sensors and cameras for facial expression recognition.

[1506] 4. Server: Cloud-based, it performs data analysis and generates action suggestions.

[1507] software:

[1508] 1. Emotion recognition software: For example, use the Emotion API from Microsoft Azure.

[1509] 2. Data analysis and generative AI models: for example, using TensorFlow, PyTorch, etc.

[1510] 3. Encryption protocol: For example, use TLS / SSL.

[1511] 4. Display software for the head-mounted display: For example, use Unity.

[1512] 2. Data collection and transmission

[1513] How users can input or automatically record behavioral data:

[1514] Users manually or automatically record their daily activities using smartphones, PCs, or HMDs.

[1515] For example, it can automatically record distance traveled and exercise time using GPS and accelerometers.

[1516] How devices send behavioral and emotional data to servers:

[1517] The collected behavioral and emotional data is encrypted using TLS / SSL and periodically sent to the server.

[1518] 3. Setting a role model

[1519] How users can set up role models:

[1520] Users set their own goals (for example, "develop safe driving habits") using a smartphone, PC, or HMD.

[1521] This configured role model information is encrypted before being sent to the server.

[1522] 4. How the Emotion Engine Works

[1523] How to recognize user emotions in real time:

[1524] The system uses sensors and cameras mounted on the HMD to monitor the user's facial expressions and heart rate.

[1525] For example, the Emotion API can be used to analyze a user's emotional state.

[1526] How to send emotional data to a server:

[1527] The recognized emotion data is periodically encrypted and sent to the server.

[1528] 5. Generating action proposals

[1529] How the server analyzes behavioral data, role model information, and sentiment data:

[1530] The server performs analysis based on past behavioral data and current emotional states. For example, it might use TensorFlow or PyTorch.

[1531] How a server uses generated AI to create action suggestions:

[1532] The generative AI proposes the optimal next course of action based on the analyzed data.

[1533] For example, if a user is feeling tired, the system might suggest, "We recommend taking a break at the next service area."

[1534] How the server sends the proposal to the user's terminal:

[1535] The generated action suggestions are notified to the user in real time via the HMD.

[1536] 6. Re-logging of feedback and action results

[1537] How users perform actions and record the results:

[1538] The user performs the suggested action and enters the result into the app.

[1539] How the device sends behavioral results and emotional data to the server:

[1540] The collected behavioral data and emotional data are sent back to the server.

[1541] How the server generates new suggestions and sends feedback:

[1542] The server re-analyzes the data based on the new information and generates new suggestions and feedback for the user.

[1543] Specific example:

[1544] Driver A set the goal of "reducing risks on the highway." When the driver feels fatigued while driving, the HMD detects an increase in heart rate and changes in facial expression. The server analyzes this and suggests, "The next rest area is 20 minutes away. We recommend taking a break soon."

[1545] Example of a prompt:

[1546] While driving an autonomous vehicle, suggest the optimal next course of action based on the role model set by the driver (e.g., safe driving).

[1547] Please consider the following data:

[1548] Driver's current emotional state (heart rate, facial recognition data)

[1549] Past driving behavior data (speed, braking, steering, etc.)

[1550] A role model set by the driver (e.g., risk reduction on highways)

[1551] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1552] (Process flow)

[1553] Step 1:

[1554] Input: User's daily activity data (e.g., distance traveled, exercise time).

[1555] Operation: The device collects activity data that the user inputs or automatically records using a smartphone, PC, or HMD. For example, it records distance traveled and exercise time using GPS and accelerometer sensors.

[1556] Output: Collected behavioral data.

[1557] Step 2:

[1558] Input: Collected behavioral data.

[1559] Operation: The device encrypts collected behavioral data and sends it to the server using TLS / SSL. For example, by sending data when connected to Wi-Fi at night, battery consumption can be minimized.

[1560] Output: Behavioral data sent to the server.

[1561] Step 3:

[1562] Input: User-defined role model information (e.g., goal of aiming for safe driving).

[1563] Operation: The user uses a smartphone, PC, or HMD to specifically define their target role model, and the device encrypts this information before sending it to the server.

[1564] Output: Role model information sent to the server.

[1565] Step 4:

[1566] Input: Behavioral data and role model information stored on the server side.

[1567] Operation: The server analyzes the user's past behavioral patterns and progress based on the received behavioral data and role model information. For example, it may use TensorFlow or PyTorch to analyze the data.

[1568] Output: Analysis results.

[1569] Step 5:

[1570] Input: Real-time user emotion data (e.g., heart rate, facial expression data).

[1571] Operation: The HMD uses sensors and cameras to recognize the user's emotional state (heart rate, facial expressions, etc.) in real time, encrypts the data, and sends it to the server.

[1572] Output: Emotional data sent to the server.

[1573] Step 6:

[1574] Input: Analysis results and real-time sentiment data.

[1575] Operation: The server uses a generative AI to generate the optimal next action based on analyzed behavioral data, role model information, and sentiment data. For example, it uses a generative AI model (e.g., GPT-4) to create prompt statements.

[1576] Output: Generated action suggestions.

[1577] Step 7:

[1578] Input: Generated action suggestions.

[1579] Operation: The server encrypts the generated action suggestion, sends it to the HMD in real time, and the HMD notifies the user. For example, it might display "We recommend taking a break at the next service area."

[1580] Output: Notification of suggested actions to the user.

[1581] Step 8:

[1582] Input: The result of the user's actions based on the suggestion.

[1583] Operation: The user performs the suggested action and inputs the result back into the HMD, smartphone, or PC. For example, after exercising, they might record "I ran for 30 minutes."

[1584] Output: The result of the input action.

[1585] Step 9:

[1586] Input: Action results and sentiment data recorded again as logs.

[1587] Operation: The device encrypts the collected behavioral data and emotional data and sends it back to the server.

[1588] Output: Action results and sentiment data sent to the server.

[1589] Step 10:

[1590] Input: New behavioral result data and sentiment data sent to the server.

[1591] Operation: The server re-analyzes the new behavioral result data and sentiment data to generate new behavioral suggestions and feedback. For example, it might generate feedback such as, "Your running pace is increasing, so keep it up."

[1592] Output: New action suggestions and feedback.

[1593] The system operates in the manner described above, enabling it to provide real-time suggestions for optimal actions based on the role model the user aspires to.

[1594] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1595] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1596] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1597] [Fourth Embodiment]

[1598] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1599] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1600] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1601] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1602] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1603] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1604] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1605] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1606] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1607] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1608] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1609] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1610] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1611] The operation of a specific system will be described as an embodiment for carrying out the present invention.

[1612] This system collects users' activity logs and suggests their next course of action based on a pre-configured role model. It is implemented through the cooperation of the user, device, and server according to the following processing flow.

[1613] 1. Collection and transmission of user activity logs

[1614] Users enter or automatically record their activity logs.

[1615] Users record their daily activities, such as travel, shopping, exercise, and meals, using their smartphones or PCs. For example, they can automatically record running distance and time using their smartphone's accelerometer and GPS function as an exercise log. They can also manually enter meal details and shopping lists into the app.

[1616] The device sends behavioral data to the server.

[1617] The collected behavioral data is encrypted and then periodically sent to the server. This ensures the security of the data.

[1618] 2. Setting a role model

[1619] Users set role models

[1620] Users can specifically define their role model. For example, they can select a professional athlete or a successful business person and input their characteristics and goals.

[1621] The device sends role model information to the server.

[1622] The configured role model information is encrypted and sent to the server, just like the user's behavioral data.

[1623] 3. Generating the next course of action

[1624] The server analyzes behavioral data and role model information.

[1625] The server performs analysis based on the received behavioral data and role model information. In particular, it compares past behavioral patterns with current progress to generate the optimal actions for the user to move closer to their goals.

[1626] The server uses AI to generate action suggestions.

[1627] The generating AI considers behavioral data and role model information to generate specific next steps. For example, if it determines that adding weight training will bring the user closer to their goal, it will create a suggestion such as "Add 30 minutes of weight training to your next workout."

[1628] The server sends the proposal to the user's terminal.

[1629] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[1630] 4. Suggestions for Users

[1631] The device will notify you of the suggested content.

[1632] The user's device will notify them of the suggestions using its notification function. This ensures that the user does not forget what action they should take next.

[1633] 5. Re-logging of feedback and action results

[1634] The user performs an action and records the result.

[1635] The user performs the suggested action and enters the results into the app. For example, if they perform the suggested weight training, they will record the duration and details of the workout again.

[1636] The device sends the results of its actions to the server.

[1637] The collected behavioral data is sent back to the server and used for further analysis.

[1638] The server generates new suggestions and sends feedback.

[1639] The server performs a re-analysis based on the newly collected behavioral data and generates feedback and suggestions for the next action for the user. This is then sent to the user's terminal for notification.

[1640] Specific example

[1641] Example 1: If you aim to become a professional athlete

[1642] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[1643] Example 2: When balancing work and family life

[1644] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[1645] As described above, this system supports users in approaching their desired role models by providing action suggestions and feedback using generated AI based on user behavior data and goal settings.

[1646] The following describes the processing flow.

[1647] Step 1:

[1648] Users enter or automatically record their activity logs.

[1649] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. Alternatively, the system can automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[1650] Step 2:

[1651] The device sends behavioral data to the server.

[1652] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent at night when connected to Wi-Fi to minimize battery consumption.

[1653] Step 3:

[1654] Users set role models

[1655] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[1656] Step 4:

[1657] The device sends role model information to the server.

[1658] The configured role model information is encrypted before being sent to the server.

[1659] Step 5:

[1660] The server analyzes behavioral data and role model information.

[1661] The server performs analysis based on the received behavioral data and role model information. This involves analyzing the user's past behavioral patterns and progress to determine what actions are necessary to move closer to the role model.

[1662] Step 6:

[1663] The server uses AI to generate action suggestions.

[1664] The generating AI uses analyzed behavioral data and role model information to suggest the optimal next action for the user. For example, it can generate specific suggestions such as, "Add an hour of running next weekend."

[1665] Step 7:

[1666] The server sends the proposal to the user's terminal.

[1667] The generated action suggestions are sent to the user's device, allowing the user to see the suggested actions.

[1668] Step 8:

[1669] The device will notify you of the suggested content.

[1670] The user's device will notify them of the suggestion using a reminder function. For example, a notification might appear saying, "Please go for a 30-minute run every morning at 6:00 AM."

[1671] Step 9:

[1672] The user performs an action and records the result.

[1673] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[1674] Step 10:

[1675] The device sends the results of its actions to the server.

[1676] The collected behavioral data is sent back to the server. This data is also encrypted before transmission and used for further analysis.

[1677] Step 11:

[1678] The server generates new suggestions and sends feedback.

[1679] The server re-analyzes the new behavioral data and generates feedback and suggestions for the next action for the user. For example, this may include positive feedback such as, "Your running pace is increasing, keep it up!"

[1680] Step 12:

[1681] The device will notify you of the feedback.

[1682] The server sends feedback to the user's device and notifies the user again. This allows the user to effectively continue taking action towards their goal.

[1683] (Example 1)

[1684] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1685] Conventional behavioral suggestion systems had problems such as not being able to fully utilize user behavior data and not being able to effectively provide optimal behavioral suggestions that corresponded to the user's goals. In particular, ensuring the security of user behavior data and improving the accuracy of behavioral suggestions were challenges.

[1686] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1687] In this invention, the server includes means for the user to input or automatically record their actions; means for encrypting the action data and transmitting it to the server; means for setting a role model that the user aims to emulate; means for encrypting the role model information and transmitting it to the server; means for generating the next action to be taken using a generated AI model based on the action data and role model information; means for notifying the user's terminal of the generated action suggestion; means for logging the results of the user's actions again, encrypting them, and transmitting them to the server; and means for generating new suggestions based on the action results and transmitting feedback to the user's terminal. This makes it possible to provide highly accurate and optimal action suggestions that match the user's goals while ensuring the security of the user's action data.

[1688] A "user" refers to an individual who uses this system to record behavioral data and aims to achieve their goals.

[1689] "Behavioral data" refers to information about daily activities such as movement, exercise, and eating that users record.

[1690] "Encryption" refers to technologies used to protect behavioral data and role model information from being deciphered by third parties.

[1691] A "server" refers to a computer system that receives and analyzes behavioral data and role model information sent by users, and generates and sends appropriate action suggestions.

[1692] A "role model" refers to an ideal individual or ideal figure that a user aspires to be like, whose characteristics and goals are set for them.

[1693] A "generative AI model" refers to an artificial intelligence model that generates suggestions for the next course of action based on user behavior data and role model information.

[1694] "Action suggestions" refer to specific action instructions generated by the server to support the user in achieving their goals.

[1695] "Feedback" refers to notifying the user of newly generated suggestions or evaluations based on the results of the actions they have taken.

[1696] A "sensor" refers to an electronic device used to acquire user movement and location information.

[1697] This invention is a system that allows users to record their own actions, generate action suggestions to help them move closer to their target role model, and support their implementation. This system operates through the coordinated efforts of the user, terminal, and server.

[1698] First, users record their daily activity logs using their smartphones or PCs. Specifically, they can automatically record running distance and time using their smartphone's accelerometer and GPS function. They can also manually input meal details and shopping lists into the app. The device encrypts the collected activity data and periodically sends it to the server using secure protocols such as HTTPS.

[1699] Next, users set their desired role model. In the dedicated app's settings screen, they can select a role model such as a "professional athlete" or a "successful business person," and input their characteristics and goals. This role model information, like behavioral data, is encrypted and sent to the server.

[1700] The behavioral data and role model information sent to the server are subject to analysis. Based on this information, the server uses a generative AI model to generate specific actions that the user should take next. For example, if the user aims to become a "professional athlete," the AI ​​will generate a specific suggestion such as "add 30 minutes of weight training to your next training session." This suggestion is sent from the server to the user's device and notified to the user. The notification can also be set as a time-based reminder.

[1701] The user receives action suggestions from their device, performs them, and records the results again in the app. For example, if the user performs a suggested weight training exercise, the user records the duration and content of the exercise again and sends it to the server via their device. This action result data is then analyzed again on the server and sent back to the user as new suggestions or feedback. This allows the user to continuously perform optimal actions.

[1702] Specific example

[1703] Example 1: If you aim to become a professional athlete

[1704] When a user sets a goal of "becoming a professional athlete" and records daily training and dietary data, the server compares the user's current fitness level with the training habits of a professional athlete and generates suggestions for the next necessary training and meals. This allows the user to continue training effectively.

[1705] Example 2: When balancing work and family life

[1706] If a user sets a goal of "achieving results at work while prioritizing family," and records their daily meetings, project progress, and time spent at home, the server analyzes this data and notifies them with specific suggestions, such as "avoid working overtime on Friday to spend more time with family next weekend." This allows the user to achieve a balanced life.

[1707] Example of a prompt

[1708] Activity data: "Distance traveled: 5km, Exercise time: 30 minutes, Meal content: Salad"

[1709] Role model: "Goal: Professional athlete, Characteristics: Daily training: 2 hours, Diet: High protein, low calorie"

[1710] Please suggest the next action this user should take.

[1711] Thus, the system of the present invention supports users in approaching their desired role model by providing action suggestions and feedback using a generated AI model based on the user's behavioral data and goal settings.

[1712] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1713] Step 1:

[1714] Users enter or automatically record their activity logs.

[1715] Users record their daily activities using a dedicated app on their smartphone or PC. Specifically, one method involves automatically recording running distance and time using the smartphone's accelerometer and GPS function. Users can also manually enter details of meals and shopping lists into the app.

[1716] Input: User behavior data (e.g., distance traveled, exercise time, diet)

[1717] Output: Recorded behavioral data

[1718] Specific operation: The smartphone's accelerometer detects the user's steps, and the GPS function measures the distance traveled. The user also inputs details of their meals through the app's UI.

[1719] Step 2:

[1720] The device encrypts behavioral data and sends it to the server.

[1721] The device encrypts user behavior data and periodically sends it to the server using secure protocols such as HTTPS. This process ensures data security.

[1722] Input: Recorded behavioral data

[1723] Output: Encrypted behavioral data

[1724] Specific operation: A data encryption module operates within the app, encrypting behavioral data using encryption algorithms such as AES. The data is then sent to the server using HTTPS.

[1725] Step 3:

[1726] Set a role model that the user aspires to be.

[1727] Users select a role model, such as a "professional athlete" or a "successful business person," from the settings screen of the dedicated app and input their characteristics and goals. This information, like behavioral data, is encrypted.

[1728] Input: User role model information (e.g., goals, characteristics)

[1729] Output: Configured role model information

[1730] Specific operation: The user enters role model information using dropdown lists or text boxes in the app's settings screen.

[1731] Step 4:

[1732] The device encrypts the role model information and sends it to the server.

[1733] As mentioned above, the device encrypts the role model information and sends it to the server using the HTTPS protocol.

[1734] Input: Configured role model information

[1735] Output: Encrypted role model information

[1736] Specific operation: The in-app data encryption module encrypts the role model information using an encryption algorithm and sends it to the server via HTTPS.

[1737] Step 5:

[1738] The server analyzes behavioral data and role model information.

[1739] The server uses a dedicated analysis algorithm to analyze the received behavioral data and role model information. This helps determine the optimal actions for the user to take to achieve their goals.

[1740] Input: Encrypted behavioral data, role model information

[1741] Output: Data analysis results

[1742] Specific operation: The analysis engine runs on the server and compares behavioral data with role model information. It also compares past behavioral patterns with current progress.

[1743] Step 6:

[1744] The server uses a generated AI model to create action suggestions.

[1745] The server uses a generated AI model to create specific action suggestions based on the data analysis results described above.

[1746] Input: Data analysis results

[1747] Output: Action Suggestions

[1748] Specific operation: The generating AI model generates optimal action suggestions for the user based on the results of data analysis (e.g., "Add 30 minutes of weight training to your next workout").

[1749] Step 7:

[1750] The server sends the proposal to the user's terminal.

[1751] The generated action suggestions are sent from the server to the user's terminal in the form of a notification.

[1752] Input: Action suggestion

[1753] Output: Action suggestions sent to the user's device

[1754] Specific operation: The server sends an action suggestion to the device, and the device displays it to the user as a notification. The notification can also be set as a time-based reminder.

[1755] Step 8:

[1756] The user performs an action and records the result.

[1757] The user performs the suggested action and records the result again in the app.

[1758] Input: Proposed action

[1759] Output: Results of the actions performed

[1760] Specific actions: The user performs the suggested actions and enters the results using a smartphone or PC app. This includes details such as exercise time and content.

[1761] Step 9:

[1762] The device encrypts the results of its actions and sends them to the server.

[1763] The device encrypts the collected behavioral data and sends it back to the server.

[1764] Input: Results of the actions performed

[1765] Output: Encrypted behavioral result data

[1766] Specific operation: The in-app data encryption module runs, encrypts the action result data, and sends it to the server.

[1767] Step 10:

[1768] The server generates new suggestions and sends feedback.

[1769] The server re-analyzes the newly collected behavioral data, generates new suggestions and feedback, and sends them to the user's device.

[1770] Input: Encrypted behavioral data

[1771] Output: New suggestions and feedback

[1772] Specific operation: The server performs a re-analysis and generates new action suggestions and feedback. The generated information is sent to the user's device in the form of a notification.

[1773] (Application Example 1)

[1774] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1775] The challenge lies in creating an operational support system that safely and efficiently manages the operation of autonomous vehicles while also considering the driver's health. Furthermore, it is necessary to achieve continuously optimized operational management by specifically proposing the driving behaviors that drivers should aim for and incorporating the results of their execution.

[1776] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1777] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a role model that the user aspires to; means for transmitting the role model information to the server; means for generating the next action to be taken using a generative AI based on the action data and role model information; means for notifying the user of the generated action suggestions; means for logging the results of the user's actions again and transmitting them to the server; means for generating new suggestions based on the action results and sending feedback to the user; means for automatically collecting operation data of the autonomous vehicle and driver action data, encrypting them, and transmitting them to the server; means for generating operation and driver action suggestions using a generative AI model based on the optimal operation pattern and driver model set by the administrator; and means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses. This makes it possible to perform advanced operation management and driver support for autonomous vehicles and improve the safety and efficiency of operations.

[1778] "Means for users to input or automatically record their own actions" refers to devices or software that allow users to manually input their own behavioral data or to automatically record behavioral data using a smartphone or sensors.

[1779] "Means for transmitting the aforementioned behavioral data to the server" refers to a device or software that securely transmits the collected and recorded behavioral data to the server via the internet.

[1780] "Means for setting a role model that the user aspires to" refers to a device or software that allows the user to specifically define the person or ideal they aspire to be and input that information.

[1781] "Means for transmitting the role model information to the server" refers to a device or software that securely transmits the configured role model information to the server via the internet.

[1782] "Means for generating the next action to be taken using a generative AI based on the aforementioned behavioral data and role model information" refers to an algorithm and system that analyzes the collected behavioral data and role model information and calculates the optimal next action using a generative AI model.

[1783] "Means for notifying the user of generated action suggestions" refers to a notification system for informing the user of action suggestions created by the generating AI, and is a device or software that uses a smartphone, smart glasses, in-vehicle display, etc.

[1784] "Means for logging the results of user actions and sending them back to the server" refers to a device or software that records the results of user actions and sends that data back to the server.

[1785] "Means for generating new suggestions based on the aforementioned action results and sending feedback to the user" refers to a device or software that re-analyzes the recorded action result data, generates new action suggestions, and notifies the user.

[1786] "Means for automatically collecting operational data of autonomous vehicles and driver behavior data, encrypting them, and then transmitting them to a server" refers to a device or software that collects various data related to the operation of autonomous vehicles and driver behavior data using sensors and cameras, encrypts them, and then transmits them to a server.

[1787] "Means for generating operational and driver action suggestions using a generation AI model based on optimal operational patterns and driver models set by the administrator" refers to a device or software that generates operational and driver action suggestions using a generation AI model based on operational pattern and driver model information set by the administrator.

[1788] "Means for notifying the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses" refers to a device or software that notifies the generated operation and driver action suggestions to the in-vehicle information display system or the driver's smart glasses.

[1789] This invention provides a system for advanced operational management and driver assistance of autonomous vehicles. This system allows users (operation managers and drivers) to collect their own behavioral logs and propose the next course of action based on their configured optimal operating patterns and driver models. This is achieved through the cooperation of a server, terminals, and autonomous vehicles, following the processing flow outlined below.

[1790] 1. Collection and transmission of user activity logs

[1791] Various sensors (cameras, accelerometers, GPS, etc.) within the autonomous vehicle are used as a means to automatically record user activity logs. These sensors record operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. The collected data is encrypted and then periodically transmitted to a server. This ensures the security of the data.

[1792] 2. Setting a role model

[1793] As a means for administrators to set role models, they can specifically define the optimal driving patterns and driver models they aim for. For example, this could include patterns that prioritize safe driving or patterns that emphasize efficiency. The set role model information is encrypted and sent to the server, just like user behavior data.

[1794] 3. Generating the next course of action

[1795] The server analyzes behavioral data and role model information. This analysis uses a generated AI model based on the behavioral data and role model information. The server compares past behavioral patterns with current progress and generates the optimal actions for the user to get closer to their goal. For example, if a driver hasn't taken a two-hour break, it will create a specific suggestion such as "Take a 15-minute break at the next rest stop."

[1796] 4. Suggestions for Users

[1797] The generated action suggestions are communicated to the vehicle's information display system or the driver's smart glasses. This allows the driver to properly understand and act on what to do next, even while driving.

[1798] 5. Re-logging of feedback and action results

[1799] There is a mechanism for automatically re-logging data as a means of recording the results when the user (driver) performs a suggested action. For example, in-vehicle sensors detect and record the driver's resting status. The collected action result data is sent back to the server and used for the next analysis. The server performs a re-analysis based on the newly collected action result data, generates new suggestions, and sends feedback.

[1800] Technology for realizing functionality

[1801] This system is constructed using the following technologies:

[1802] Hardware:

[1803] Various sensors installed inside the vehicle (camera, accelerometer, GPS, etc.)

[1804] Smart glasses for drivers (e.g., Google Glass)

[1805] Information display system for autonomous vehicles

[1806] software:

[1807] Custom API for data collection and transmission (encryption supported)

[1808] Cloud tools (e.g., Google Cloud AutoML, AWS SageMaker) are used to analyze behavioral data and operational models.

[1809] Generative AI models include natural language generation models (e.g., OpenAI's GPT model).

[1810] Specific example

[1811] For example, the following prompt statements are used to input into the generated AI model.

[1812] Prompt message:

[1813] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[1814] Output of the generative AI model:

[1815] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[1816] As described above, the system for implementing the present invention enables advanced operational management and driver assistance for autonomous vehicles. This is expected to improve the safety and efficiency of operations.

[1817] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1818] Step 1:

[1819] The user records their activity log. Various sensors in the autonomous vehicle (cameras, accelerometers, GPS, etc.) collect operational data (distance traveled, time, route) and driver status (rest time, driving time, health data) in real time. Operational data and driver status data are obtained as input data. This data is encrypted.

[1820] Output: Encrypted operational data and driver status data.

[1821] Step 2:

[1822] The terminal sends the encrypted data to the server. The data is securely transferred to the server using a transmission protocol (e.g., HTTPS).

[1823] Input: Encrypted operational data and driver status data.

[1824] Output: Encrypted data stored on the server.

[1825] Step 3:

[1826] The user (flight manager) sets the role model they aspire to. The manager uses a web portal or application to specify the optimal ferry pattern and driver model (safe driving, efficient driving, etc.). The configuration information is encrypted and sent to the server.

[1827] Input: Role model configuration information.

[1828] Output: Role model configuration information stored on the server.

[1829] Step 4:

[1830] The server analyzes behavioral data and role model information. Cloud tools (such as Google Cloud AutoML and AWS SageMaker) are used to analyze and compare this data in real time.

[1831] Inputs: Operational data, driver status data, role model information.

[1832] Output: Analysis results.

[1833] Step 5:

[1834] The server uses a generative AI model to generate the next action to take. It inputs a prompt sentence into a natural language generation model (e.g., OpenAI's GPT model) and obtains an appropriate action suggestion.

[1835] Input: Analysis results.

[1836] Output: Generated action suggestions.

[1837] Step 6:

[1838] The terminal notifies the information display system in the autonomous vehicle or the driver's smart glasses of the generated action suggestions. It also provides the function to notify the user visually or audibly.

[1839] Input: Generated action suggestions.

[1840] Output: Notification to the driver.

[1841] Step 7:

[1842] When the user (driver) takes action based on the suggestion, the result of that action is recorded again as a log. The results of the actions are automatically collected by sensors and cameras inside the vehicle.

[1843] Input: Driver action result data.

[1844] Output: Log data.

[1845] Step 8:

[1846] The terminal sends the aforementioned log data to the server. The server analyzes it again, generates new action suggestions, and sends feedback.

[1847] Input: Log data.

[1848] Output: New action suggestions and feedback.

[1849] As a concrete example, enter the following prompt message into the generative AI model.

[1850] Prompt message:

[1851] "The driver's operational data shows that they haven't taken a break in the last two hours, and their current heart rate and driving stress level are increasing. Furthermore, the role model has set a pattern that prioritizes safe driving. Considering this situation, what should be the next course of action?"

[1852] Output of the generative AI model:

[1853] "I suggest taking a 15-minute break at the next rest stop. You should also rehydrate and do some light stretching."

[1854] By following these steps, advanced operational management and driver assistance for autonomous vehicles can be achieved.

[1855] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1856] The operation of a specific system will be described as an embodiment of the present invention.

[1857] This system collects the user's own behavioral logs and suggests the next course of action based on a pre-set role model. It also incorporates an emotion engine that recognizes the user's emotions and adjusts the action suggestions and feedback accordingly. This is achieved through the cooperation of the user, device, and server according to the processing flow outlined below.

[1858] 1. Collection and transmission of user activity logs

[1859] Users enter or automatically record their activity logs.

[1860] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. Alternatively, the system can automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[1861] The device sends behavioral data to the server.

[1862] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent at night when connected to Wi-Fi to minimize battery consumption.

[1863] 2. Setting a role model

[1864] Users set role models

[1865] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[1866] The device sends role model information to the server.

[1867] The configured role model information is encrypted before being sent to the server.

[1868] 3. How the Emotion Engine Works

[1869] Recognizing user emotions

[1870] The user's device is equipped with emotion recognition capabilities, allowing for real-time monitoring of the user's current emotional state through technologies such as facial recognition and voice analysis.

[1871] Send emotional data to the server.

[1872] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[1873] 4. Generating the next course of action

[1874] The server analyzes behavioral data, role model information, and emotional data.

[1875] The server analyzes the received behavioral data, role model information, and emotional data. In particular, it generates the most effective action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[1876] The server uses AI to generate action suggestions.

[1877] The generating AI uses multiple analyzed data points to suggest the optimal next action for the user. For example, if the user is feeling fatigued, it might suggest "setting a lighter training session for the next workout."

[1878] The server sends the proposal to the user's terminal.

[1879] The generated action suggestions are sent to the user's device and displayed as notifications. These notifications may also be set as time-based reminders.

[1880] 5. Suggestions for Users

[1881] The device will notify you of the suggested content.

[1882] The user's device uses a reminder function to notify them of the suggestions. For example, a notification might appear saying, "Please go for a light 30-minute run today." Because the suggestions are adjusted based on emotional data, users can take appropriate action.

[1883] 6. Re-logging of feedback and action results

[1884] The user performs an action and records the result.

[1885] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[1886] The device sends behavioral results and emotional data to the server.

[1887] The collected behavioral and emotional data are sent back to the server and used for further analysis.

[1888] The server generates new suggestions and sends feedback.

[1889] The server re-analyzes the new behavioral and emotional data to generate feedback and suggestions for the next action for the user. For example, this might include positive feedback such as, "Your running pace is increasing, keep it up!"

[1890] For example, if a user aims to become a professional athlete and records daily training and emotional data, the server will adjust the training content based on the user's fatigue level and fluctuations in motivation. Similarly, for users aiming to balance work and family life, suggestions for relaxation will be provided when stress levels are high, enabling more effective stress management.

[1891] As described above, this system supports users in approaching their desired role models by providing behavioral suggestions and feedback using generated AI based on user behavioral data, goal settings, and emotional data.

[1892] The following describes the processing flow.

[1893] Step 1:

[1894] Users enter or automatically record their activity logs.

[1895] Users manually input their daily activities (e.g., travel, shopping, exercise, meals) using their smartphones or PCs. The system can also automatically record distance traveled, exercise time, and other data using the smartphone's GPS and accelerometer.

[1896] Step 2:

[1897] The device sends behavioral data to the server.

[1898] Smartphones and PCs periodically send collected behavioral data to a server. This transmission is encrypted to ensure data security. For example, data is sent in batches when connected to Wi-Fi at night to minimize battery drain.

[1899] Step 3:

[1900] Users set role models

[1901] Users set specific role models they aspire to. For example, they might enter specific goals such as "becoming a professional athlete" or "living a balanced life."

[1902] Step 4:

[1903] The device sends role model information to the server.

[1904] The configured role model information is encrypted before being sent to the server.

[1905] Step 5:

[1906] Recognizing user emotions

[1907] The device uses emotion recognition to monitor the user's emotional state in real time through facial expression and voice analysis. Emotion recognition often utilizes the device's camera and microphone.

[1908] Step 6:

[1909] The device sends emotional data to the server.

[1910] The recognized emotion data is periodically sent to the server. This allows for real-time monitoring of the user's emotional changes.

[1911] Step 7:

[1912] The server analyzes behavioral data, role model information, and emotional data.

[1913] The server analyzes the received behavioral data, role model information, and emotional data. For example, it considers the user's past behavioral patterns, progress, and current emotional state to suggest the most effective next action.

[1914] Step 8:

[1915] The server uses AI to generate action suggestions.

[1916] The generating AI suggests the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it will generate a specific suggestion such as "set a lighter training session next time."

[1917] Step 9:

[1918] The server sends the proposal to the user's terminal.

[1919] The generated action suggestions are sent to the user's device and displayed as notifications.

[1920] Step 10:

[1921] The device will notify you of the suggested content.

[1922] The user's device will notify them of the suggestions using a reminder function. For example, a notification might appear saying, "Please go for a light 30-minute run today." Because the suggestions are adjusted based on emotional data, users can take appropriate action.

[1923] Step 11:

[1924] The user performs an action and records the result.

[1925] The user performs the suggested action and enters the result into the app. For example, after a run, they might record, "I ran for 30 minutes."

[1926] Step 12:

[1927] The device sends behavioral results and emotional data to the server.

[1928] The collected behavioral and emotional data are sent back to the server and used for further analysis.

[1929] Step 13:

[1930] The server generates new suggestions and sends feedback.

[1931] The server re-analyzes the newly collected behavioral and emotional data to generate feedback and suggestions for the next action for the user. For example, it might generate positive feedback such as, "Your running pace is increasing, keep it up!"

[1932] Step 14:

[1933] The device will notify you of the feedback.

[1934] The server sends feedback to the user's device and notifies the user again. This allows the user to continue taking effective action towards their goal.

[1935] (Example 2)

[1936] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1937] Conventional behavior suggestion systems provide suggestions based on user behavior data and goal setting, but they do not take into account the user's emotional state. As a result, it is difficult for users to effectively carry out the suggested actions, and there is a problem in that optimal behavior suggestions cannot be made according to individual circumstances.

[1938] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1939] In this invention, the server includes means for the user to input or automatically record their actions; means for transmitting the action data to the server; means for setting a target model that the user aims for; means for transmitting the target model information to the server; means for recognizing the user's emotional state and transmitting that data to the server; means for performing analysis based on the action data, target model information, and emotional data and generating the next action to be taken using a generated AI model; means for notifying the user of the generated action suggestions; means for recording the results of the user's actions and transmitting them to the server; and means for generating new suggestions based on the action results and sending feedback to the user. This makes it possible to provide individual action suggestions that take into account the user's emotional state.

[1940] A "user" refers to an individual who uses the system to input and record their own behavioral and emotional data.

[1941] "Behavioral data" refers to information recorded by users about their daily activities, including distance traveled, exercise time, and diet.

[1942] A "server" refers to a computer system that receives behavioral data, role model information, and emotional data, and performs analysis, generates behavioral suggestions, and sends feedback.

[1943] A "goal model" refers to information that shows the specific goals and desired outcomes that a user aims to achieve.

[1944] "Emotional state" refers to data that represents the user's current emotions and mood, and is collected using facial recognition technology and voice analysis.

[1945] A "generative AI model" refers to an artificial intelligence model that generates the optimal next action to take based on the data it receives.

[1946] "Action suggestion" refers to the optimal next action that the user should take, as suggested by the generative AI model.

[1947] "Action results" refer to the data recorded after a user performs a suggested action.

[1948] "Feedback" refers to new suggestions or evaluations that a server generates for a user based on their actions.

[1949] This invention is a system that collects a user's own behavioral logs and suggests the next action to take based on a set goal model. Furthermore, it is equipped with an emotion engine that recognizes the user's emotional state and adjusts the action suggestions and feedback accordingly. This system is realized through the cooperation of the user, terminal, and server, as shown below.

[1950] Collection and transmission of user behavior logs

[1951] Users manually input their daily activities (e.g., travel, shopping, exercise, meals, etc.) through a dedicated application using their smartphone or PC. Alternatively, the system can automatically record distance traveled and exercise time using the GPS and accelerometer sensors (e.g., common location services and sensors) built into the smartphone.

[1952] The device periodically sends collected behavioral data to the server. AES256 encryption is used during transmission to ensure data security. The system is designed to minimize battery consumption, especially by transmitting data at night while connected to Wi-Fi.

[1953] Setting a role model

[1954] Through using the application, users set specific goal models they aspire to. For example, they might input goals such as "become a professional athlete" or "live a balanced life."

[1955] The terminal encrypts the configured target model information and sends it to the server.

[1956] How the emotion engine works

[1957] The user's device is equipped with emotion recognition capabilities, utilizing facial recognition technology (such as OpenCV) and speech analysis (such as Google Cloud Speech-to-Text API) to monitor the user's current emotional state in real time. This function allows for understanding the user's stress level and relaxation state.

[1958] Emotional data is periodically sent to the server and used for analysis there.

[1959] Generating the next course of action

[1960] The server analyzes the received behavioral data, target model information, and emotional data. Specifically, it generates action suggestions by considering past behavioral patterns, current progress, and the user's emotional state.

[1961] The server uses a generative AI (such as GPT-3) to suggest the best course of action next. For example, if the user is feeling fatigued, it might suggest "set a lighter training session next time." Examples of prompts in this case include:

[1962] "Our goal is for users to become professional athletes. Based on their current behavioral logs and emotional data, please suggest the optimal next course of action."

[1963] The generated action suggestions are sent from the server to the user's device and displayed as notifications. These suggestions can also be set as time-based reminders.

[1964] Suggestions and feedback for users

[1965] The user's device uses a reminder function to notify them of the suggestions. For example, a notification might appear saying, "Please go for a light 30-minute run today." Suggestions based on emotional data enable users to take appropriate actions.

[1966] The user performs the suggested action and logs the result to the application. For example, after running, they might type "I ran for 30 minutes."

[1967] The device sends the collected behavioral and emotional data back to the server for use in the next analysis. The server re-analyzes the data based on the new behavioral and emotional data and generates new behavioral suggestions and feedback. For example, it might send positive feedback such as, "Your running pace is increasing, keep it up!"

[1968] Specific example

[1969] Let's say a user's goal is to "become a professional athlete" and they are recording their daily training and emotional data. The server will adjust the next training session based on the user's fatigue level and fluctuations in motivation. Similarly, if another user's goal is to "balance work and family life," the server will suggest actions to relax when stress levels are high, enabling more effective stress management.

[1970] As described above, this system uses user behavior data, goal setting, and emotional data to provide optimal action suggestions and feedback using generative AI, thereby supporting users in approaching their desired target model.

[1971] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1972] Step 1:

[1973] Users enter or automatically record their activity logs.

[1974] Users manually input data on their daily activities (e.g., travel, shopping, exercise, meals) through a dedicated application using their smartphone or PC. Additionally, the system automatically records distance traveled and exercise time using the smartphone's GPS and accelerometer. This process collects detailed activity logs of the user as input data.

[1975] Step 2:

[1976] The device sends behavioral data to the server.

[1977] The device encrypts the collected behavioral data using AES256 and periodically sends it to the server. The destination is the server, and encrypted behavioral data is sent as input data. The output is the behavioral data securely stored on the server.

[1978] Step 3:

[1979] The user sets the target model.

[1980] Users input their desired goals (e.g., "become a professional athlete" or "live a balanced life") in text format through the application. This allows the application to obtain information about their goals as input data.

[1981] Step 4:

[1982] The device sends target model information to the server.

[1983] The terminal encrypts the configured target model information using AES256 and sends it to the server. The input data is the encrypted target model information, and the transmission result is the target model information securely stored on the server.

[1984] Step 5:

[1985] Recognizing user emotions

[1986] The device is equipped with emotion recognition capabilities, utilizing facial recognition technology (e.g., OpenCV) and speech analysis (e.g., Google Cloud Speech-to-Text API) to monitor the user's emotional state. This allows for the acquisition of real-time emotion data as input.

[1987] Step 6:

[1988] Send emotional data to the server.

[1989] The device periodically encrypts recognized emotion data using AES256 and sends it to the server. The input data is encrypted emotion data, and the transmission result is the emotion data stored on the server.

[1990] Step 7:

[1991] The server analyzes behavioral data, target model information, and emotional data.

[1992] The server integrates and analyzes the received behavioral data, target model information, and emotional data. Specifically, it performs data cleansing and normalization as a preprocessing step, and uses an algorithm that considers past behavioral patterns, progress, and emotional states as an analysis step. The input data consists of behavioral data, target model information, and emotional data, and the analysis results in analytical data regarding the next action to be taken.

[1993] Step 8:

[1994] The server uses a generated AI model to create action suggestions.

[1995] The server uses a generative AI model (e.g., GPT-3) to suggest the optimal next action for the user based on the analyzed data. For example, if the user is feeling fatigued, it might suggest "set a lighter training session next time." The input data is the analyzed data, and the output is a specific action suggestion. Example prompt: "The user's goal is to become a professional athlete. Based on the current activity log and emotional data, please suggest the optimal next action."

[1996] Step 9:

[1997] The server sends the proposal to the user's terminal.

[1998] The server sends the generated action suggestions to the terminal, where they are notified. The input data includes the action suggestions, and the output is the action suggestions notified to the user's terminal.

[1999] Step 10:

[2000] The device will notify you of the suggested content.

[2001] The device uses a reminder function to inform the user of the suggested actions. For example, a notification might appear saying, "Please go for a light 30-minute run today." The input data is the suggested action, and the output is the notified action suggestion.

[2002] Step 11:

[2003] The user performs an action and records the result.

[2004] The user performs the suggested action and logs the result in the app. For example, after running, the user might input "I ran for 30 minutes." The input data is a log of the execution result, and the output is the recorded action result.

[2005] Step 12:

[2006] The device sends behavioral results and emotional data to the server.

[2007] The terminal encrypts the collected behavioral result data and emotional data using AES256 and sends it to the server. The input data consists of behavioral result data and emotional data, and the output is the data stored on the server.

[2008] Step 13:

[2009] The server generates new suggestions and sends feedback.

[2010] The server re-analyzes the new behavioral and emotional data to generate new behavioral suggestions and feedback for the user. For example, it might send feedback such as, "Your running pace is increasing, keep it up." The input data consists of behavioral and emotional data, and the output is new behavioral suggestions and feedback.

[2011] (Application Example 2)

[2012] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2013] Conventional user assistance systems in autonomous vehicles were unable to consider user behavior data and emotional states in real time, making it difficult to suggest appropriate actions. Furthermore, they lacked the ability to suggest the optimal next course of action based on user-defined goals (role models), resulting in insufficient support for users achieving their objectives.

[2014] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for the user to input or automatically record their actions, means for transmitting the action data to the server, means for setting a role model that the user aims to emulate, means for transmitting the role model information to the server, means for recognizing the user's emotional state in real time, means for transmitting the emotional data to the server, means for generating the next action to be taken using a generated AI based on the action data, role model information and emotional data, means for notifying the user of the generated action suggestion, means for logging the results of the user's actions again and transmitting them to the server, means for generating a new suggestion based on the action results and emotional data and sending feedback to the user, and means for providing the action suggestion in cooperation with an autonomous mobile system. This makes it possible to analyze the user's emotional state and action data in real time and to quickly and effectively provide optimal action suggestions based on the user's goals.

[2015] A "user" is an entity that uses this system to input and record their own activity logs and receive suggestions.

[2016] "Behavioral data" refers to various actions performed by users (e.g., travel, shopping, exercise, eating, etc.) and related information.

[2017] A "server" is a device that collects and analyzes behavioral data, role model information, and emotional data, and then generates and provides the next course of action to be taken.

[2018] A "role model" refers to the goal or ideal state that the user aspires to, and action suggestions are made based on that.

[2019] "Emotional state" refers to the user's current emotional state (e.g., sad, happy, tired, etc.), and is data recognized in real time using sensors, etc.

[2020] "Generative AI" refers to an artificial intelligence model that analyzes collected data and automatically generates optimal action suggestions.

[2021] An "autonomous mobility system" is a system that has autonomous driving technology used by users for transportation, such as self-driving vehicles.

[2022] "Action suggestions" are suggestions for the next action to take, generated by a generating AI based on the user's behavioral data, emotional state, and role model information.

[2023] "Feedback" refers to evaluations and suggestions for future actions generated based on the results of the user's actions.

[2024] The configuration and operation of a specific system will be described as an embodiment for carrying out the present invention. This system uses behavioral data, emotional data, and role model information to generate action suggestions based on the role model that the user aspires to.

[2025] 1. Prerequisite System Configuration

[2026] The system is built upon the following key hardware and software components.

[2027] Hardware:

[2028] 1. Head-mounted display (HMD): Recognizes the user's emotional state in real time and displays action suggestions.

[2029] 2. ECU (Electronic Control Unit) of an autonomous vehicle: Collects vehicle driving data.

[2030] 3. Sensors for emotion recognition: such as heart rate sensors and cameras for facial expression recognition.

[2031] 4. Server: Cloud-based, it performs data analysis and generates action suggestions.

[2032] software:

[2033] 1. Emotion recognition software: For example, use the Emotion API from Microsoft Azure.

[2034] 2. Data analysis and generative AI models: for example, using TensorFlow, PyTorch, etc.

[2035] 3. Encryption protocol: For example, use TLS / SSL.

[2036] 4. Display software for the head-mounted display: For example, use Unity.

[2037] 2. Data collection and transmission

[2038] How users can input or automatically record behavioral data:

[2039] Users manually or automatically record their daily activities using smartphones, PCs, or HMDs.

[2040] For example, it can automatically record distance traveled and exercise time using GPS and accelerometers.

[2041] How devices send behavioral and emotional data to servers:

[2042] The collected behavioral and emotional data is encrypted using TLS / SSL and periodically sent to the server.

[2043] 3. Setting a role model

[2044] How users can set up role models:

[2045] Users set their own goals (for example, "develop safe driving habits") using a smartphone, PC, or HMD.

[2046] This configured role model information is encrypted before being sent to the server.

[2047] 4. How the Emotion Engine Works

[2048] How to recognize user emotions in real time:

[2049] The system uses sensors and cameras mounted on the HMD to monitor the user's facial expressions and heart rate.

[2050] For example, the Emotion API can be used to analyze a user's emotional state.

[2051] How to send emotional data to a server:

[2052] The recognized emotion data is periodically encrypted and sent to the server.

[2053] 5. Generating action proposals

[2054] How the server analyzes behavioral data, role model information, and sentiment data:

[2055] The server performs analysis based on past behavioral data and current emotional states. For example, it might use TensorFlow or PyTorch.

[2056] How a server uses generated AI to create action suggestions:

[2057] The generative AI proposes the optimal next course of action based on the analyzed data.

[2058] For example, if a user is feeling tired, the system might suggest, "We recommend taking a break at the next service area."

[2059] How the server sends the proposal to the user's terminal:

[2060] The generated action suggestions are notified to the user in real time via the HMD.

[2061] 6. Re-logging of feedback and action results

[2062] How users perform actions and record the results:

[2063] The user performs the suggested action and enters the result into the app.

[2064] How the device sends behavioral results and emotional data to the server:

[2065] The collected behavioral data and emotional data are sent back to the server.

[2066] How the server generates new suggestions and sends feedback:

[2067] The server re-analyzes the data based on the new information and generates new suggestions and feedback for the user.

[2068] Specific example:

[2069] Driver A set the goal of "reducing risks on the highway." When the driver feels fatigued while driving, the HMD detects an increase in heart rate and changes in facial expression. The server analyzes this and suggests, "The next rest area is 20 minutes away. We recommend taking a break soon."

[2070] Example of a prompt:

[2071] While driving an autonomous vehicle, suggest the optimal next course of action based on the role model set by the driver (e.g., safe driving).

[2072] Please consider the following data:

[2073] Driver's current emotional state (heart rate, facial recognition data)

[2074] Past driving behavior data (speed, braking, steering, etc.)

[2075] A role model set by the driver (e.g., risk reduction on highways)

[2076] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2077] (Process flow)

[2078] Step 1:

[2079] Input: User's daily activity data (e.g., distance traveled, exercise time).

[2080] Operation: The device collects activity data that the user inputs or automatically records using a smartphone, PC, or HMD. For example, it records distance traveled and exercise time using GPS and accelerometer sensors.

[2081] Output: Collected behavioral data.

[2082] Step 2:

[2083] Input: Collected behavioral data.

[2084] Operation: The device encrypts collected behavioral data and sends it to the server using TLS / SSL. For example, by sending data when connected to Wi-Fi at night, battery consumption can be minimized.

[2085] Output: Behavioral data sent to the server.

[2086] Step 3:

[2087] Input: User-defined role model information (e.g., goal of aiming for safe driving).

[2088] Operation: The user uses a smartphone, PC, or HMD to spe...

Claims

1. Means for users to input or automatically record their actions, Means for transmitting the aforementioned behavioral data to a server, A means of setting a role model that the user aspires to, Means for transmitting the aforementioned role model information to a server, A means for generating the next action to take using a generated AI based on the aforementioned behavioral data and role model information, A means of notifying the user of the generated action suggestions, A means of recording the results of user actions again as logs and sending them to the server, A means for generating new suggestions based on the results of the aforementioned actions and sending feedback to the user, A system that includes this.

2. The system according to claim 1, comprising means for using GPS and sensors when collecting user behavior data.

3. The system according to claim 1, which includes means for notifying a user of action suggestions as a time-based reminder.

Images