system

Abstract

We provide the system. [Solution] A means for receiving user voice information and converting it from voice to text data, A method for performing natural language processing on converted text data to extract planned information, A means of generating and sending suggestions based on extracted schedule information, taking into account the user's behavioral history and environmental information, A means of receiving user responses to generated proposals and updating the schedule database, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Currently, many users only use communication apps as communication tools. In such usage, functions such as schedule management, task management, and schedule adjustment that users need in daily life are lacking, making it difficult to perform efficient life management. Also, to perform these functions individually, it is necessary to use a variety of applications, which is a problem of being cumbersome for users.

Means for Solving the Problems

[0005] This invention provides a means for receiving user input data and extracting schedule information through natural language processing. This allows for the automatic identification of schedules from user conversations. Furthermore, it includes means for generating personalized suggestions based on the extracted schedule information and sending them to the user's terminal. If the user responds to a suggestion, the schedule data is automatically updated based on that information. This enables users to intuitively manage their schedules and tasks on a single platform. In addition, by analyzing past activity history, personalized suggestions tailored to each user can be provided. Furthermore, this invention includes a function to automate schedule adjustments among multiple users and appropriately notify each user of the results. This enables efficient and convenient life management.

[0006] A "user" is an entity that uses a communication application to exchange information.

[0007] "Input data" refers to the entirety of information, including messages and instructions, that a user sends through a communication platform.

[0008] "Natural language processing" is the process of analyzing human language on a computer to understand its meaning and structure.

[0009] "Scheduled information" refers to information related to future actions and events, extracted from user conversations and instructions.

[0010] A "suggestion" is a recommended action or option that the system generates based on the user's situation and past history.

[0011] "Schedule data" refers to a collection of data that organizes information about a user's appointments and tasks and manages them in chronological order.

[0012] Personalization is the process of tailoring specific information and services to individual users.

[0013] "Activity history" refers to a record of actions and choices a user has made in the past.

[0014] "Schedule management" is the process of properly managing multiple appointments and tasks to achieve the optimal time allocation for the user. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] To implement this invention, the system is configured to function primarily through three elements: a server, a terminal, and a user. The server receives input data from the user and processes this information through natural language processing. It is equipped with an AI model for natural language processing and has the ability to efficiently extract schedule information from the user's conversation. This approach enables complex schedule management from the user's intuitive input.

[0037] The terminal is a device used directly by the user and plays the role of receiving notifications and suggestions from the server and presenting them to the user. Schedule data and task information are updated in response to user actions (e.g., approving or editing suggestions).

[0038] As a concrete example, consider a scenario where a user says in a LINE chat with a friend, "Let's go on a picnic on the first Saturday of next month." When the server receives this message, it uses natural language processing to extract information about the plan, such as "the first Saturday of next month" and "picnic." Based on this extracted information, the server generates personalized suggestions for places to visit, such as parks the user frequently visits or weather forecast services, and notifies the user's device. When the user confirms and accepts these suggestions, the device updates its schedule data and adds the event to the calendar.

[0039] Furthermore, the server refers to the user's past activity history to provide a means of further personalizing the suggestions. By suggesting more appropriate options to the user, it helps them manage their schedule quickly and accurately.

[0040] This invention makes it possible to transform ordinary communication apps into advanced planning and management tools, enriching users' lives.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] Users use the LINE app for everyday conversations. During this process, they send messages related to their schedules and tasks.

[0044] Step 2:

[0045] The terminal receives the user's message and prepares to forward it to the server.

[0046] Step 3:

[0047] The server retrieves the received message and analyzes it using a natural language processing engine. This extracts keywords related to the schedule and tasks.

[0048] Step 4:

[0049] The server generates suggestions or reminders based on the analysis results. This includes calculating potential schedules and task priorities.

[0050] Step 5:

[0051] The server sends the generated suggestions to the terminal and notifies the user. This notification includes an interface for review and editing.

[0052] Step 6:

[0053] Users can review proposals through their devices and approve or edit them as needed.

[0054] Step 7:

[0055] The terminal sends the user's actions back to the server.

[0056] Step 8:

[0057] The server updates the schedule data based on the user's response. This ensures that the user's schedule remains up-to-date.

[0058] Step 9:

[0059] If necessary, the server synchronizes with related services to ensure consistency with external calendar apps and task management tools.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In users' daily lives, intuitively managing schedules is difficult. Furthermore, traditional methods struggled to efficiently process user input data and provide appropriate suggestions. Moreover, providing personalized suggestions based on each user's individual activity history was not easy.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for receiving user input data, analyzing the information using natural language processing technology to extract schedule information, generating suggestions to the user based on the extracted schedule information by referring to related information, and sending those suggestions, and means for receiving the user's response to the generated suggestions, updating the information based on the response, and managing the schedule data. This enables users to manage their schedules easily and efficiently. Furthermore, it enables the provision of personalized suggestions based on the user's activity records, allowing for more accurate schedule adjustments.

[0065] A "user" is a person or entity that uses the system and provides input data.

[0066] "Input data" refers to information that users provide to the system, including schedule information.

[0067] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0068] "Schedule information" refers to data that describes the user's actions and detailed schedule.

[0069] A "proposal" is a suggestion sent from the system to the user regarding schedule management and scheduling adjustments.

[0070] "Response" refers to feedback that users provide to a proposal, such as approval or modification.

[0071] "Schedule data" refers to information recorded in a user's calendar or schedule.

[0072] "History of past actions" refers to a record of activities and choices the user has made in the past.

[0073] "Personalization" refers to individually optimizing content based on each user's preferences and history.

[0074] In its embodiment, this system primarily consists of three elements: a server, a terminal, and a user. The server is responsible for performing the main processing, the user utilizes its services, and the terminal serves as the interface connecting the user and the server.

[0075] The server functions as a network-connected computer system to receive input data from users. Specifically, the server uses natural language processing technology to analyze user input data and extract planned information. For this purpose, the server is equipped with a generative AI model. The analyzed data is then used to generate suggestions, taking into account the user's past behavioral history. For example, based on a user's statement, "I want to have a picnic on the first Saturday of next month," the server will make suggestions that take into account frequently visited parks and the weather information for that day.

[0076] A terminal is a device that receives user input and notifications and suggestions from the server. Users use this terminal to review, approve, or modify suggestions. This ensures that the schedule data is updated in real time. Examples of terminals include personal digital assistants (PDAs), computers, and smartphones.

[0077] Users can manage their own schedules and events through comments and input. The user interface is intuitive, making the system easy to use even without specialized knowledge.

[0078] For example, if a user sends a LINE message saying, "I want to have a picnic on the first Saturday of next month," the server analyzes the message and sends a personalized suggestion to the user's device. If the user accepts the suggestion, the plan is added to the device's calendar. In this process, the prompt "I want to have a picnic on the first Saturday of next month. Please tell me some good places to go on that day" is used as input to the AI ​​model.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The server receives user input data. This input is a natural language message sent from the user's device. The user sends a message like "Let's go on a picnic on the first Saturday of next month" via LINE or other communication apps. The server receives this message as data.

[0082] Step 2:

[0083] The server performs natural language processing on the received input data. Using a generative AI model, it analyzes the user's intent and related schedule information, extracting keywords such as "the first Saturday of next month" and "picnic." This analysis structures the input data as schedule information.

[0084] Step 3:

[0085] Based on the analyzed schedule information, the server generates suggestions by referencing the user's past activity history and current situation. For example, it collects information on parks the user has visited in the past and the weather on the day, and creates suggestions such as, "How about the park on the corner?" In this process, extracted keywords are matched with historical information to derive the most suitable suggestions.

[0086] Step 4:

[0087] The server sends the generated suggestions to the user's device. The device displays the suggestions to the user and prompts for confirmation. The suggestions include potential places to visit and their advantages. The user can easily review this on the screen.

[0088] Step 5:

[0089] Users review proposals via their devices and approve or modify them. If they accept a proposal, they click the accept button. If modifications are needed, the user sends new instructions. This action allows the server to receive the user's response.

[0090] Step 6:

[0091] The device updates the schedule data based on the user's response. If approved, the new appointment is added to the calendar and the update is complete. If modifications are made, instructions are sent back to the server, which then returns to the process of further analysis.

[0092] (Application Example 1)

[0093] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] Traditional schedule management systems required users to manually input and manage their schedules, which was time-consuming and laborious. Furthermore, it was difficult to provide personalized suggestions that fully utilized user behavior history and environmental information.

[0095] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0096] In this invention, the server includes means for receiving user voice information and converting it from voice to text data, means for performing natural language analysis on the converted text data to extract schedule information, and means for generating and transmitting suggestions based on the extracted schedule information, taking into account the user's behavioral history and environmental information. As a result, the user can manage their schedule with intuitive voice operations and receive personalized and optimized suggestions.

[0097] "Voice information" refers to data used to recognize the words spoken by a user as digital signals.

[0098] "Text data" refers to data in sentence form obtained by analyzing audio information.

[0099] "Natural language processing" is a technology that understands the meaning of input text data and extracts necessary information.

[0100] "Schedule information" refers to data about the user's future plans and schedules.

[0101] "Activity history" refers to a record of activities a user has performed in the past.

[0102] "Environmental information" refers to data related to the user's current situation, such as weather and traffic conditions.

[0103] A "suggestion" is a proposal or recommendation generated by the system based on the user's individual conditions.

[0104] "Means of conversion" refers to methods or devices for converting audio information into text data.

[0105] "Means of transmission" refers to functions for delivering suggestions and notifications to the user's device.

[0106] This invention is a system that manages schedules based on user voice information. The system mainly consists of three elements: a server, a terminal, and the user.

[0107] The server first converts the received audio information into text data using the Google® Cloud Speech-to-Text API. It then analyzes this converted text data using OpenAI® natural language processing models to extract schedule information. Based on this extracted schedule information, it analyzes the user's past activity history and externally acquired environmental information (such as weather and traffic information). Based on this, it generates personalized suggestions for the user and provides them to the user through their device.

[0108] The terminal acts as an interface with the user, receiving and displaying suggestions and notifications from the server. When the user accepts a suggestion or enters changes or supplementary information, the terminal resends this information to the server and updates the schedule data.

[0109] Users can utilize the system naturally within their homes, for example, by speaking to a robot equipped with a smart speaker. For instance, if they give a command such as, "Add a library appointment for next Wednesday," that information is processed throughout the system, and an appropriate schedule is set.

[0110] Examples of prompt statements are as follows:

[0111] User: Please write down that I plan to go to the library next Wednesday.

[0112] AI Model: Finds keywords in text, extracts the date, time, and destination, and adds them to the calendar.

[0113] This allows users to easily manage their schedules through a natural voice-based interface.

[0114] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0115] Step 1:

[0116] The user inputs voice information into the device. When the user speaks, the device receives it and sends it to the server as digital voice data. This voice data becomes the system's input.

[0117] Step 2:

[0118] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. During the conversion, the audio data is analyzed to extract the spoken content as accurate text. This text data becomes the input for the next process.

[0119] Step 3:

[0120] The server utilizes OpenAI's natural language processing model to extract schedule information from the converted text data. Specifically, the model identifies keywords and date / time information within the text and organizes them as the user's schedule data. This schedule information then serves as input for the next step.

[0121] Step 4:

[0122] The server references the user's past behavioral history data and environmental information obtained from external sources (such as weather and traffic information) to generate optimal suggestions based on the extracted schedule information. A generation AI model is used to analyze and optimize the information necessary for the suggestions. These generated suggestions are then output to the terminal.

[0123] Step 5:

[0124] The terminal notifies the user of the proposal sent from the server. The user reviews the proposal and, if they wish to accept it or make modifications, inputs them through the terminal. This user response becomes the input for the next process.

[0125] Step 6:

[0126] The server receives the user's response and updates the schedule database based on its content. After confirming the response, it saves the updated schedule information and notifies the user again if necessary. This process ensures that the user has the most up-to-date schedule.

[0127] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0128] This invention provides a system that incorporates an emotion engine that recognizes emotions from user communication. This system utilizes natural language input data provided by users within a communication application, and performs natural language analysis and emotion analysis on that data to enable personalized schedule management and suggestions for users.

[0129] The server receives messages sent by users and extracts schedule information using a natural language processing engine. It also uses an emotion engine to recognize the user's emotions from the messages. As a result, it generates suggestions and reminders based on the user's emotional state and notifies the device.

[0130] The device can receive notifications from the server and display customized messages tailored to the user's mood. The user reviews these suggestions and updates the schedule data by approving or editing them as appropriate.

[0131] As a concrete example, consider a scenario where a user says in a chat with a friend, "I'm tired, so I'm not going to do anything this weekend." The server receives this message and recognizes "fatigue" using its emotion engine. Based on this emotion information, the server generates suggestions for relaxing activities and presents a schedule that allows the user to refresh themselves without overexerting themselves.

[0132] Furthermore, in conversations involving multiple users, the system can comprehensively understand the emotional state of each user and adjust schedules while considering the overall emotional balance. This configuration enables flexible schedule management that takes into account the individual state of each user, resulting in more human-like interactions.

[0133] This invention employs a personalized approach that incorporates emotion recognition to improve the efficiency and comfort of user schedule management. This allows users to receive support optimized to their emotions and state at any given time.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] Users initiate conversations with friends and colleagues using the LINE app. During these conversations, they send messages that may contain information about their feelings or plans.

[0137] Step 2:

[0138] The terminal receives messages sent by the user and forwards those messages to the server.

[0139] Step 3:

[0140] The server passes the received message to a natural language processing engine, which extracts schedule information from the message. At the same time, it also passes the message to an emotion engine to recognize the user's emotions.

[0141] Step 4:

[0142] Based on the extracted schedule information, the server generates suggestions and reminders that take the user's emotions into consideration. For example, if the emotion of "tired" is detected, it will provide suggestions that include relaxing activities or ways to reduce tasks.

[0143] Step 5:

[0144] The server sends the generated suggestions to the user's device and notifies the user. The notification content is presented in an expression adapted to the user's emotions.

[0145] Step 6:

[0146] Users can check notifications on their devices and approve or change the content of suggestions and reminders. They can also request alternative, sentiment-based suggestions if needed.

[0147] Step 7:

[0148] The terminal sends the user's response to the server and updates the schedule data based on the user's approval.

[0149] Step 8:

[0150] The server synchronizes updated schedules with other related apps and services as needed and stores them in a central database, ensuring that the latest information is always available.

[0151] This process allows users to seamlessly manage their schedules and adjust tasks while receiving personalized support tailored to their emotional state.

[0152] (Example 2)

[0153] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0154] Traditional scheduling management systems manage schedules based on user input, but they have the problem of not being able to provide optimal suggestions to users because they do not take into account users' emotions or past activity history. Furthermore, there is a problem in scheduling adjustments that do not take into account the emotional state of individual users in communication among multiple users.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0156] In this invention, the server includes means for receiving user input data and extracting schedule information by performing natural language processing; means for generating and sending a user-optimized suggestion based on the extracted schedule information and the user's emotional data; means for receiving the user's response to the generated suggestion and updating the schedule data; and means for analyzing the user's emotional data and performing schedule adjustments that take into account the overall emotional balance among multiple users. This enables personalized schedule suggestions that respond to the user's emotions and allows for adjustments that take emotional balance into account in communication among multiple users.

[0157] "User input data" refers to text data that users send through a communication platform.

[0158] "Natural language processing" is a technology that allows computers to understand human language and extract structured information from text.

[0159] "Schedule information" refers to information about schedule-related elements and dates / times extracted from user input data.

[0160] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their input data.

[0161] A "suggestion" is a recommended action or schedule generated based on the user's schedule information and sentiment data.

[0162] "Schedule data" refers to digital time management information that compiles a user's schedule information.

[0163] "Emotional balance" is a concept that refers to the harmony or equilibrium of emotional states among multiple users.

[0164] Personalization is the process of making adjustments and optimizations according to the individual user's characteristics and circumstances.

[0165] This invention provides a system that utilizes an emotion engine and a natural language processing engine to streamline user schedule management and provide personalized suggestions based on emotions.

[0166] The server receives messages sent by users through communication apps. A natural language processing engine is used to perform natural language processing on the received data. In this process, a natural language API is typically used as a text analysis tool to extract scheduled information from the messages. Additionally, an emotion engine analyzes the user's emotional data to identify emotions such as "joy," "sadness," and "fatigue."

[0167] The server generates personalized suggestions based on extracted schedule and sentiment data. To achieve this, it uses a suggestion generation engine to customize templates and create messages that resonate with the user's emotions. For example, if a user messages, "I'm tired, so I'm not doing anything this weekend," the server recognizes the "fatigue" and suggests relaxing activities.

[0168] The generated suggestions are notified to the device and displayed to the user. The user reviews the suggestions and updates their schedule data by approving or editing them. In this scenario, a schedule management application is installed on the device, and the schedule data is synchronized with a cloud system.

[0169] Furthermore, when multiple users communicate, the server adjusts the schedule considering the overall emotional balance and notifies each user of the adjustment results. This allows users to manage their activities in an emotionally optimized environment.

[0170] An example of a prompt for a generative AI model is: "When a user says, 'I'm tired, so I'm not going to do anything this weekend,' suggest relaxing activities and provide a schedule that addresses their feelings."

[0171] In this way, it is possible to build a system that enables flexible schedule management and proposal provision tailored to the individual needs and emotions of users.

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] The server receives messages sent by users through communication apps. The input for this operation is message data from the user, and processing begins when the server receives this data. The received data is then prepared for analysis in the next step.

[0175] Step 2:

[0176] The server uses a natural language processing engine to parse the received message. The input here is the message data received in the previous step. This process extracts schedule information and other relevant information from the message. This operation includes grammatical analysis and key phrase extraction, and the output is a set of extracted schedule information.

[0177] Step 3:

[0178] The server then uses an emotion engine to extract emotion data from the messages. The input for this process is the message data obtained in step 2. This engine analyzes the emotions in natural language and generates emotion tags such as "joy" and "sadness." The output is a dataset containing the emotion tags.

[0179] Step 4:

[0180] The server generates user-optimized suggestions based on extracted schedule information and sentiment data. The inputs for this stage are the schedule information from step 2 and the sentiment data from step 3. The server uses a template-based generation method to create user-appropriate suggestions. The output is a customized suggestion message.

[0181] Step 5:

[0182] The server notifies the user's terminal of the generated suggestion. The input for this step is the suggestion message generated in step 4. The notification from the server is pushed to the terminal, and the output is the notification message displayed on the terminal.

[0183] Step 6:

[0184] The terminal receives suggestion notifications from the server and displays them to the user. The input for this operation is the suggestion message delivered from the server, which is then displayed on the terminal. The user can then review the displayed suggestion.

[0185] Step 7:

[0186] The user reviews the proposal and approves or edits it as needed. The input for this operation is the proposal displayed on the terminal. The schedule data is updated based on the user's instructions, and this update is synchronized with the cloud system. The output is the updated schedule data.

[0187] Step 8:

[0188] The server adjusts schedules while considering the overall emotional balance among multiple users. Inputs at this stage include emotional data and schedules from individual users. The server integrates the emotional data to create an optimal schedule for all users. The output is the adjusted schedule information, which is then notified to each user.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] In modern society, schedule management is a crucial daily task, and there is a particular demand for personalized schedule suggestions that take into account the user's emotional state. However, conventional systems struggle to adequately reflect the user's emotional state in schedule management, limiting their ability to improve the user's quality of life. Furthermore, when coordinating schedules among multiple users, uniform adjustments are made without regard for each user's feelings, resulting in adjustments that are not optimal for the user. These challenges need to be addressed.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes means for receiving user input data and performing natural language analysis to extract schedule information and sentiment information; means for generating and sending suggestions to the user based on the extracted schedule information and sentiment information; and means for receiving the user's response to the generated suggestions and updating schedule data based on the sentiment state. This enables personalized schedule management that takes the user's emotions into consideration.

[0194] "Means for receiving user input data and performing natural language analysis to extract schedule information and emotional information" refers to a system that analyzes and identifies content related to specific schedules and the user's emotional state from the natural language data entered by the user.

[0195] "A means of generating and sending suggestions to users based on extracted schedule and sentiment information" refers to a function that utilizes the schedule and sentiment information obtained through analysis to create optimized suggestions for users and send them to them.

[0196] "A means of receiving user responses to generated suggestions and updating schedule data based on emotional state" refers to a function that, after receiving feedback from the user, updates the schedule information in light of that feedback and the user's emotional state.

[0197] "Means for analyzing a user's past activity and emotional history to personalize suggestions" refers to a function that evaluates a user's past behavioral records and emotional changes, and then individualizes and makes more appropriate suggestions based on that evaluation.

[0198] "A means of coordinating schedules among multiple users and notifying them of the results while taking into account each user's emotional state" refers to a function that aligns the schedules of all participating users while communicating the adjustment results while considering each user's emotional needs.

[0199] The system that implements this application consists of a server that receives and processes user input data in natural language, and a terminal that displays the results and enables interaction with the user.

[0200] The server is built using programming languages ​​such as Python and employs libraries like NLTK and Transformers for natural language processing. This allows it to extract schedule and sentiment information from user messages. For sentiment analysis, it uses pre-trained models such as BERT to identify the user's emotional state.

[0201] The terminal displays suggestions sent from the server to the user, receives the user's response, and sends it back to the server. The system's adaptability and efficiency are improved when the user approves or edits the suggestions. The terminal could be a smartphone, tablet, or even a consumer robot.

[0202] For example, if a user says in a conversation with a friend, "I've been so busy lately, I'm not feeling very enthusiastic," the server receives this and analyzes it using its emotion engine. Then, taking into account past activity history and emotional tendencies, it suggests relaxing activities or changes to plans. For instance, it might display a personalized message on the device such as, "How about taking some time this weekend to enjoy a relaxing hobby?"

[0203] An example of a prompt for a generative AI model is: "Generate a sentence to suggest when a user says they're not feeling up to it. Example: 'You seem a little tired today....'" This prompt is designed to guide the AI ​​to determine the most appropriate suggestion for the user's emotional state.

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The server receives the user's input message in natural language. It captures the input message as text data and prepares it for the next processing step.

[0207] Step 2:

[0208] The server analyzes the received text data using a natural language processing engine. Specifically, it performs morphological analysis using the Python NLTK library to extract schedule information and sentiment information from the message. The input here is the user's text data, and the output is the analyzed schedule information and sentiment information.

[0209] Step 3:

[0210] The server uses an emotion engine to extract emotional information, which is then analyzed by a pre-trained model such as BERT to identify the user's emotional state. In this step, the input is the emotional information obtained in the previous step, and the output is a label representing the user's emotional state (e.g., joy, sadness, fatigue).

[0211] Step 4:

[0212] The server generates personalized suggestions based on analyzed schedule information and emotional state. Past activity and emotional history are also considered in the suggestion generation process. Using a generative AI model, the prompt "Generate a suggestion for when the user says they're not in the mood" is given to create the suggestion content. The input here is emotional state and past history, and the output is a personalized suggestion message.

[0213] Step 5:

[0214] The server sends the generated suggestion message to the terminal. The terminal notifies the user of this suggestion message and waits for user confirmation. The input is the suggestion message from the server, and the output is a visual or audio notification to the user.

[0215] Step 6:

[0216] The user receives suggestions from their terminal and approves or edits them. The decisions made by the user are sent back to the server, and the schedule data is updated. The input is user feedback, and the output is the updated schedule data.

[0217] Step 7:

[0218] When scheduling adjustments are needed for multiple users, the server takes each user's emotional state into consideration while making the adjustments. It then notifies each user of the results and obtains their consent. The input consists of each user's emotional data and schedule information, and the output is the adjusted schedule notification.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] To implement this invention, the system is configured to function primarily through three elements: a server, a terminal, and a user. The server receives input data from the user and processes this information through natural language processing. It is equipped with an AI model for natural language processing and has the ability to efficiently extract schedule information from the user's conversation. This approach enables complex schedule management from the user's intuitive input.

[0236] The terminal is a device used directly by the user and plays the role of receiving notifications and suggestions from the server and presenting them to the user. Schedule data and task information are updated in response to user actions (e.g., approving or editing suggestions).

[0237] As a concrete example, consider a scenario where a user says in a LINE chat with a friend, "Let's go on a picnic on the first Saturday of next month." When the server receives this message, it uses natural language processing to extract information about the plan, such as "the first Saturday of next month" and "picnic." Based on this extracted information, the server generates personalized suggestions for places to visit, such as parks the user frequently visits or weather forecast services, and notifies the user's device. When the user confirms and accepts these suggestions, the device updates its schedule data and adds the event to the calendar.

[0238] Furthermore, the server refers to the user's past activity history to provide a means of further personalizing the suggestions. By suggesting more appropriate options to the user, it helps them manage their schedule quickly and accurately.

[0239] This invention makes it possible to transform ordinary communication apps into advanced planning and management tools, enriching users' lives.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] Users use the LINE app for everyday conversations. During this process, they send messages related to their schedules and tasks.

[0243] Step 2:

[0244] The terminal receives the user's message and prepares to forward it to the server.

[0245] Step 3:

[0246] The server retrieves the received message and analyzes it using a natural language processing engine. This extracts keywords related to the schedule and tasks.

[0247] Step 4:

[0248] The server generates suggestions or reminders based on the analysis results. This includes calculating potential schedules and task priorities.

[0249] Step 5:

[0250] The server sends the generated suggestions to the terminal and notifies the user. This notification includes an interface for review and editing.

[0251] Step 6:

[0252] Users can review proposals through their devices and approve or edit them as needed.

[0253] Step 7:

[0254] The terminal sends the user's actions back to the server.

[0255] Step 8:

[0256] The server updates the schedule data based on the user's response. This ensures that the user's schedule remains up-to-date.

[0257] Step 9:

[0258] If necessary, the server synchronizes with related services to ensure consistency with external calendar apps and task management tools.

[0259] (Example 1)

[0260] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0261] In users' daily lives, intuitively managing schedules is difficult. Furthermore, traditional methods struggled to efficiently process user input data and provide appropriate suggestions. Moreover, providing personalized suggestions based on each user's individual activity history was not easy.

[0262] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0263] In this invention, the server includes means for receiving user input data, analyzing the information using natural language processing technology to extract schedule information, generating suggestions to the user based on the extracted schedule information by referring to related information, and sending those suggestions, and means for receiving the user's response to the generated suggestions, updating the information based on the response, and managing the schedule data. This enables users to manage their schedules easily and efficiently. Furthermore, it enables the provision of personalized suggestions based on the user's activity records, allowing for more accurate schedule adjustments.

[0264] A "user" is a person or entity that uses the system and provides input data.

[0265] "Input data" refers to information that users provide to the system, including schedule information.

[0266] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0267] "Schedule information" refers to data that describes the user's actions and detailed schedule.

[0268] A "proposal" is a suggestion sent from the system to the user regarding schedule management and scheduling adjustments.

[0269] "Response" refers to feedback that users provide to a proposal, such as approval or modification.

[0270] "Schedule data" refers to information recorded in a user's calendar or schedule.

[0271] "History of past actions" refers to a record of activities and choices the user has made in the past.

[0272] "Personalization" refers to individually optimizing content based on each user's preferences and history.

[0273] In its embodiment, this system primarily consists of three elements: a server, a terminal, and a user. The server is responsible for performing the main processing, the user utilizes its services, and the terminal serves as the interface connecting the user and the server.

[0274] The server functions as a network-connected computer system to receive input data from users. Specifically, the server uses natural language processing technology to analyze user input data and extract planned information. For this purpose, the server is equipped with a generative AI model. The analyzed data is then used to generate suggestions, taking into account the user's past behavioral history. For example, based on a user's statement, "I want to have a picnic on the first Saturday of next month," the server will make suggestions that take into account frequently visited parks and the weather information for that day.

[0275] A terminal is a device that receives user input and notifications and suggestions from the server. Users use this terminal to review, approve, or modify suggestions. This ensures that the schedule data is updated in real time. Examples of terminals include personal digital assistants (PDAs), computers, and smartphones.

[0276] Users can manage their own schedules and events through comments and input. The user interface is intuitive, making the system easy to use even without specialized knowledge.

[0277] For example, if a user sends a LINE message saying, "I want to have a picnic on the first Saturday of next month," the server analyzes the message and sends a personalized suggestion to the user's device. If the user accepts the suggestion, the plan is added to the device's calendar. In this process, the prompt "I want to have a picnic on the first Saturday of next month. Please tell me some good places to go on that day" is used as input to the AI ​​model.

[0278] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0279] Step 1:

[0280] The server receives user input data. This input is a natural language message sent from the user's device. The user sends a message like "Let's go on a picnic on the first Saturday of next month" via LINE or other communication apps. The server receives this message as data.

[0281] Step 2:

[0282] The server performs natural language processing on the received input data. Using a generative AI model, it analyzes the user's intent and related schedule information, extracting keywords such as "the first Saturday of next month" and "picnic." This analysis structures the input data as schedule information.

[0283] Step 3:

[0284] Based on the analyzed schedule information, the server generates proposals by referring to the user's past behavior history and current situation. For example, it collects information about parks the user has visited in the past and the weather information for the day, and creates proposals such as "How about the park at the corner?" In this process, the extracted keywords are matched with the historical information to derive the optimal proposal.

[0285] Step 4:

[0286] The server sends the generated proposals to the user's terminal. The terminal displays the proposals to the user and requests confirmation. The proposals include the candidate visit locations and their advantages. The user can easily confirm this on the screen.

[0287] Step 5:

[0288] The user checks the proposals through the terminal and approves or modifies them. When accepting the proposal, the user clicks the acceptance button. If modification is required, the user sends new instructions. Through this operation, the server receives the user's response.

[0289] Step 6:

[0290] The terminal updates the schedule data based on the user's response. When approved, the new schedule is added to the calendar and the update is completed. When a modification is made, an instruction is sent to the server again, and the server returns to the procedure of performing further analysis.

[0291] (Application Example 1)

[0292] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0293] Traditional schedule management systems required users to manually input and manage their schedules, which was time-consuming and laborious. Furthermore, it was difficult to provide personalized suggestions that fully utilized user behavior history and environmental information.

[0294] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0295] In this invention, the server includes means for receiving user voice information and converting it from voice to text data, means for performing natural language analysis on the converted text data to extract schedule information, and means for generating and transmitting suggestions based on the extracted schedule information, taking into account the user's behavioral history and environmental information. As a result, the user can manage their schedule with intuitive voice operations and receive personalized and optimized suggestions.

[0296] "Voice information" refers to data used to recognize the words spoken by a user as digital signals.

[0297] "Text data" refers to data in sentence form obtained by analyzing audio information.

[0298] "Natural language processing" is a technology that understands the meaning of input text data and extracts necessary information.

[0299] "Schedule information" refers to data about the user's future plans and schedules.

[0300] "Activity history" refers to a record of activities a user has performed in the past.

[0301] "Environmental information" refers to data related to the user's current situation, such as weather and traffic conditions.

[0302] A "suggestion" is a proposal or recommendation generated by the system based on the user's individual conditions.

[0303] The "converting means" is a method or device for converting voice information into text data.

[0304] The "transmitting means" is a function for transmitting proposals or notifications to the user's device.

[0305] This invention is a system for schedule management based on the user's voice information. The system is mainly composed of three elements: a server, a terminal, and a user.

[0306] The server first converts the received voice information into text data by utilizing the Google Cloud Speech-to-Text API. For this converted text data, it performs analysis using OpenAI's natural language processing model and is responsible for extracting schedule information. Based on the extracted schedule information, it analyzes in combination with the user's past behavior history and environmental information (such as weather information and traffic information, etc.) obtained from the outside. Based on this, it generates personalized proposals for the user and provides them to the user through the terminal.

[0307] The terminal functions as an interface with the user, receives and displays proposals and notifications from the server. When the user accepts a proposal or inputs modification or supplementary information, the terminal resends this information to the server to update the schedule data.

[0308] The user can use the system in a natural form, for example, by speaking to a robot equipped with a smart speaker within the home. For example, when giving an instruction such as "Add a library appointment for next Wednesday," the information is processed throughout the system and an appropriate schedule is set.

[0309] Examples of prompt sentences are as follows:

[0310] User: Write down the plan to go to the library next Wednesday.

[0311] AI Model: Finds keywords in text, extracts the date, time, and destination, and adds them to the calendar.

[0312] This allows users to easily manage their schedules through a natural voice-based interface.

[0313] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0314] Step 1:

[0315] The user inputs voice information into the device. When the user speaks, the device receives it and sends it to the server as digital voice data. This voice data becomes the system's input.

[0316] Step 2:

[0317] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. During the conversion, the audio data is analyzed to extract the spoken content as accurate text. This text data becomes the input for the next process.

[0318] Step 3:

[0319] The server utilizes OpenAI's natural language processing model to extract schedule information from the converted text data. Specifically, the model identifies keywords and date / time information within the text and organizes them as the user's schedule data. This schedule information then serves as input for the next step.

[0320] Step 4:

[0321] The server references the user's past behavioral history data and environmental information obtained from external sources (such as weather and traffic information) to generate optimal suggestions based on the extracted schedule information. A generation AI model is used to analyze and optimize the information necessary for the suggestions. These generated suggestions are then output to the terminal.

[0322] Step 5:

[0323] The terminal notifies the user of the proposal sent from the server. The user reviews the proposal and, if they wish to accept it or make modifications, inputs them through the terminal. This user response becomes the input for the next process.

[0324] Step 6:

[0325] The server receives the user's response and updates the schedule database based on its content. After confirming the response, it saves the updated schedule information and notifies the user again if necessary. This process ensures that the user has the most up-to-date schedule.

[0326] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0327] This invention provides a system that incorporates an emotion engine that recognizes emotions from user communication. This system utilizes natural language input data provided by users within a communication application, and performs natural language analysis and emotion analysis on that data to enable personalized schedule management and suggestions for users.

[0328] The server receives messages sent by users and extracts schedule information using a natural language processing engine. It also uses an emotion engine to recognize the user's emotions from the messages. As a result, it generates suggestions and reminders based on the user's emotional state and notifies the device.

[0329] The device can receive notifications from the server and display customized messages tailored to the user's mood. The user reviews these suggestions and updates the schedule data by approving or editing them as appropriate.

[0330] As a concrete example, consider a scenario where a user says in a chat with a friend, "I'm tired, so I'm not going to do anything this weekend." The server receives this message and recognizes "fatigue" using its emotion engine. Based on this emotion information, the server generates suggestions for relaxing activities and presents a schedule that allows the user to refresh themselves without overexerting themselves.

[0331] Furthermore, in conversations involving multiple users, the system can comprehensively understand the emotional state of each user and adjust schedules while considering the overall emotional balance. This configuration enables flexible schedule management that takes into account the individual state of each user, resulting in more human-like interactions.

[0332] This invention employs a personalized approach that incorporates emotion recognition to improve the efficiency and comfort of user schedule management. This allows users to receive support optimized to their emotions and state at any given time.

[0333] The following describes the processing flow.

[0334] Step 1:

[0335] Users initiate conversations with friends and colleagues using the LINE app. During these conversations, they send messages that may contain information about their feelings or plans.

[0336] Step 2:

[0337] The terminal receives messages sent by the user and forwards those messages to the server.

[0338] Step 3:

[0339] The server passes the received message to a natural language processing engine, which extracts schedule information from the message. At the same time, it also passes the message to an emotion engine to recognize the user's emotions.

[0340] Step 4:

[0341] Based on the extracted schedule information, the server generates suggestions and reminders that take the user's emotions into consideration. For example, if the emotion of "tired" is detected, it will provide suggestions that include relaxing activities or ways to reduce tasks.

[0342] Step 5:

[0343] The server sends the generated suggestions to the user's device and notifies the user. The notification content is presented in an expression adapted to the user's emotions.

[0344] Step 6:

[0345] Users can check notifications on their devices and approve or change the content of suggestions and reminders. They can also request alternative, sentiment-based suggestions if needed.

[0346] Step 7:

[0347] The terminal sends the user's response to the server and updates the schedule data based on the user's approval.

[0348] Step 8:

[0349] The server synchronizes updated schedules with other related apps and services as needed and stores them in a central database, ensuring that the latest information is always available.

[0350] This process allows users to seamlessly manage their schedules and adjust tasks while receiving personalized support tailored to their emotional state.

[0351] (Example 2)

[0352] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0353] Traditional scheduling management systems manage schedules based on user input, but they have the problem of not being able to provide optimal suggestions to users because they do not take into account users' emotions or past activity history. Furthermore, there is a problem in scheduling adjustments that do not take into account the emotional state of individual users in communication among multiple users.

[0354] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0355] In this invention, the server includes means for receiving user input data and extracting schedule information by performing natural language processing; means for generating and sending a user-optimized suggestion based on the extracted schedule information and the user's emotional data; means for receiving the user's response to the generated suggestion and updating the schedule data; and means for analyzing the user's emotional data and performing schedule adjustments that take into account the overall emotional balance among multiple users. This enables personalized schedule suggestions that respond to the user's emotions and allows for adjustments that take emotional balance into account in communication among multiple users.

[0356] "User input data" refers to text data that users send through a communication platform.

[0357] "Natural language processing" is a technology that allows computers to understand human language and extract structured information from text.

[0358] "Schedule information" refers to information about schedule-related elements and dates / times extracted from user input data.

[0359] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their input data.

[0360] A "suggestion" is a recommended action or schedule generated based on the user's schedule information and sentiment data.

[0361] "Schedule data" refers to digital time management information that compiles a user's schedule information.

[0362] "Emotional balance" is a concept that refers to the harmony or equilibrium of emotional states among multiple users.

[0363] Personalization is the process of making adjustments and optimizations according to the individual user's characteristics and circumstances.

[0364] This invention provides a system that utilizes an emotion engine and a natural language processing engine to streamline user schedule management and provide personalized suggestions based on emotions.

[0365] The server receives messages sent by users through communication apps. A natural language processing engine is used to perform natural language processing on the received data. In this process, a natural language API is typically used as a text analysis tool to extract scheduled information from the messages. Additionally, an emotion engine analyzes the user's emotional data to identify emotions such as "joy," "sadness," and "fatigue."

[0366] The server generates personalized suggestions based on extracted schedule and sentiment data. To achieve this, it uses a suggestion generation engine to customize templates and create messages that resonate with the user's emotions. For example, if a user messages, "I'm tired, so I'm not doing anything this weekend," the server recognizes the "fatigue" and suggests relaxing activities.

[0367] The generated suggestions are notified to the device and displayed to the user. The user reviews the suggestions and updates their schedule data by approving or editing them. In this scenario, a schedule management application is installed on the device, and the schedule data is synchronized with a cloud system.

[0368] Furthermore, when multiple users communicate, the server adjusts the schedule considering the overall emotional balance and notifies each user of the adjustment results. This allows users to manage their activities in an emotionally optimized environment.

[0369] An example of a prompt for a generative AI model is: "When a user says, 'I'm tired, so I'm not going to do anything this weekend,' suggest relaxing activities and provide a schedule that addresses their feelings."

[0370] In this way, it is possible to build a system that enables flexible schedule management and proposal provision tailored to the individual needs and emotions of users.

[0371] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0372] Step 1:

[0373] The server receives messages sent by users through communication apps. The input for this operation is message data from the user, and processing begins when the server receives this data. The received data is then prepared for analysis in the next step.

[0374] Step 2:

[0375] The server uses a natural language processing engine to parse the received message. The input here is the message data received in the previous step. This process extracts schedule information and other relevant information from the message. This operation includes grammatical analysis and key phrase extraction, and the output is a set of extracted schedule information.

[0376] Step 3:

[0377] The server then uses an emotion engine to extract emotion data from the messages. The input for this process is the message data obtained in step 2. This engine analyzes the emotions in natural language and generates emotion tags such as "joy" and "sadness." The output is a dataset containing the emotion tags.

[0378] Step 4:

[0379] The server generates user-optimized suggestions based on extracted schedule information and sentiment data. The inputs for this stage are the schedule information from step 2 and the sentiment data from step 3. The server uses a template-based generation method to create user-appropriate suggestions. The output is a customized suggestion message.

[0380] Step 5:

[0381] The server notifies the user's terminal of the generated suggestion. The input for this step is the suggestion message generated in step 4. The notification from the server is pushed to the terminal, and the output is the notification message displayed on the terminal.

[0382] Step 6:

[0383] The terminal receives suggestion notifications from the server and displays them to the user. The input for this operation is the suggestion message delivered from the server, which is then displayed on the terminal. The user can then review the displayed suggestion.

[0384] Step 7:

[0385] The user reviews the proposal and approves or edits it as needed. The input for this operation is the proposal displayed on the terminal. The schedule data is updated based on the user's instructions, and this update is synchronized with the cloud system. The output is the updated schedule data.

[0386] Step 8:

[0387] The server adjusts schedules while considering the overall emotional balance among multiple users. Inputs at this stage include emotional data and schedules from individual users. The server integrates the emotional data to create an optimal schedule for all users. The output is the adjusted schedule information, which is then notified to each user.

[0388] (Application Example 2)

[0389] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0390] In modern society, schedule management is a crucial daily task, and there is a particular demand for personalized schedule suggestions that take into account the user's emotional state. However, conventional systems struggle to adequately reflect the user's emotional state in schedule management, limiting their ability to improve the user's quality of life. Furthermore, when coordinating schedules among multiple users, uniform adjustments are made without regard for each user's feelings, resulting in adjustments that are not optimal for the user. These challenges need to be addressed.

[0391] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0392] In this invention, the server includes means for receiving user input data and performing natural language analysis to extract schedule information and sentiment information; means for generating and sending suggestions to the user based on the extracted schedule information and sentiment information; and means for receiving the user's response to the generated suggestions and updating schedule data based on the sentiment state. This enables personalized schedule management that takes the user's emotions into consideration.

[0393] "Means for receiving user input data and performing natural language analysis to extract schedule information and emotional information" refers to a system that analyzes and identifies content related to specific schedules and the user's emotional state from the natural language data entered by the user.

[0394] "A means of generating and sending suggestions to users based on extracted schedule and sentiment information" refers to a function that utilizes the schedule and sentiment information obtained through analysis to create optimized suggestions for users and send them to them.

[0395] "A means of receiving user responses to generated suggestions and updating schedule data based on emotional state" refers to a function that, after receiving feedback from the user, updates the schedule information in light of that feedback and the user's emotional state.

[0396] "Means for analyzing a user's past activity and emotional history to personalize suggestions" refers to a function that evaluates a user's past behavioral records and emotional changes, and then individualizes and makes more appropriate suggestions based on that evaluation.

[0397] "A means of coordinating schedules among multiple users and notifying them of the results while taking into account each user's emotional state" refers to a function that aligns the schedules of all participating users while communicating the adjustment results while considering each user's emotional needs.

[0398] The system that implements this application consists of a server that receives and processes user input data in natural language, and a terminal that displays the results and enables interaction with the user.

[0399] The server is built using programming languages ​​such as Python and employs libraries like NLTK and Transformers for natural language processing. This allows it to extract schedule and sentiment information from user messages. For sentiment analysis, it uses pre-trained models such as BERT to identify the user's emotional state.

[0400] The terminal displays suggestions sent from the server to the user, receives the user's response, and sends it back to the server. The system's adaptability and efficiency are improved when the user approves or edits the suggestions. The terminal could be a smartphone, tablet, or even a consumer robot.

[0401] For example, if a user says in a conversation with a friend, "I've been so busy lately, I'm not feeling very enthusiastic," the server receives this and analyzes it using its emotion engine. Then, taking into account past activity history and emotional tendencies, it suggests relaxing activities or changes to plans. For instance, it might display a personalized message on the device such as, "How about taking some time this weekend to enjoy a relaxing hobby?"

[0402] An example of a prompt for a generative AI model is: "Generate a sentence to suggest when a user says they're not feeling up to it. Example: 'You seem a little tired today....'" This prompt is designed to guide the AI ​​to determine the most appropriate suggestion for the user's emotional state.

[0403] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0404] Step 1:

[0405] The server receives the user's input message in natural language. It captures the input message as text data and prepares it for the next processing step.

[0406] Step 2:

[0407] The server analyzes the received text data using a natural language processing engine. Specifically, it performs morphological analysis using the Python NLTK library to extract schedule information and sentiment information from the message. The input here is the user's text data, and the output is the analyzed schedule information and sentiment information.

[0408] Step 3:

[0409] The server uses an emotion engine to extract emotional information, which is then analyzed by a pre-trained model such as BERT to identify the user's emotional state. In this step, the input is the emotional information obtained in the previous step, and the output is a label representing the user's emotional state (e.g., joy, sadness, fatigue).

[0410] Step 4:

[0411] The server generates personalized suggestions based on analyzed schedule information and emotional state. Past activity and emotional history are also considered in the suggestion generation process. Using a generative AI model, the prompt "Generate a suggestion for when the user says they're not in the mood" is given to create the suggestion content. The input here is emotional state and past history, and the output is a personalized suggestion message.

[0412] Step 5:

[0413] The server sends the generated suggestion message to the terminal. The terminal notifies the user of this suggestion message and waits for user confirmation. The input is the suggestion message from the server, and the output is a visual or audio notification to the user.

[0414] Step 6:

[0415] The user receives suggestions from their terminal and approves or edits them. The decisions made by the user are sent back to the server, and the schedule data is updated. The input is user feedback, and the output is the updated schedule data.

[0416] Step 7:

[0417] When scheduling adjustments are needed for multiple users, the server takes each user's emotional state into consideration while making the adjustments. It then notifies each user of the results and obtains their consent. The input consists of each user's emotional data and schedule information, and the output is the adjusted schedule notification.

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0419] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0421] [Third Embodiment]

[0422] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0423] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0424] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0426] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0428] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0429] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0430] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0433] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0434] To implement this invention, the system is configured to function primarily through three elements: a server, a terminal, and a user. The server receives input data from the user and processes this information through natural language processing. It is equipped with an AI model for natural language processing and has the ability to efficiently extract schedule information from the user's conversation. This approach enables complex schedule management from the user's intuitive input.

[0435] The terminal is a device used directly by the user and plays the role of receiving notifications and suggestions from the server and presenting them to the user. Schedule data and task information are updated in response to user actions (e.g., approving or editing suggestions).

[0436] As a concrete example, consider a scenario where a user says in a LINE chat with a friend, "Let's go on a picnic on the first Saturday of next month." When the server receives this message, it uses natural language processing to extract information about the plan, such as "the first Saturday of next month" and "picnic." Based on this extracted information, the server generates personalized suggestions for places to visit, such as parks the user frequently visits or weather forecast services, and notifies the user's device. When the user confirms and accepts these suggestions, the device updates its schedule data and adds the event to the calendar.

[0437] Furthermore, the server refers to the user's past activity history to provide a means of further personalizing the suggestions. By suggesting more appropriate options to the user, it helps them manage their schedule quickly and accurately.

[0438] This invention makes it possible to transform ordinary communication apps into advanced planning and management tools, enriching users' lives.

[0439] The following describes the processing flow.

[0440] Step 1:

[0441] Users use the LINE app for everyday conversations. During this process, they send messages related to their schedules and tasks.

[0442] Step 2:

[0443] The terminal receives the user's message and prepares to forward it to the server.

[0444] Step 3:

[0445] The server retrieves the received message and analyzes it using a natural language processing engine. This extracts keywords related to the schedule and tasks.

[0446] Step 4:

[0447] The server generates suggestions or reminders based on the analysis results. This includes calculating potential schedules and task priorities.

[0448] Step 5:

[0449] The server sends the generated suggestions to the terminal and notifies the user. This notification includes an interface for review and editing.

[0450] Step 6:

[0451] Users can review proposals through their devices and approve or edit them as needed.

[0452] Step 7:

[0453] The terminal sends the user's actions back to the server.

[0454] Step 8:

[0455] The server updates the schedule data based on the user's response. This ensures that the user's schedule remains up-to-date.

[0456] Step 9:

[0457] If necessary, the server synchronizes with related services to ensure consistency with external calendar apps and task management tools.

[0458] (Example 1)

[0459] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0460] In users' daily lives, intuitively managing schedules is difficult. Furthermore, traditional methods struggled to efficiently process user input data and provide appropriate suggestions. Moreover, providing personalized suggestions based on each user's individual activity history was not easy.

[0461] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0462] In this invention, the server includes means for receiving user input data, analyzing the information using natural language processing technology to extract schedule information, generating suggestions to the user based on the extracted schedule information by referring to related information, and sending those suggestions, and means for receiving the user's response to the generated suggestions, updating the information based on the response, and managing the schedule data. This enables users to manage their schedules easily and efficiently. Furthermore, it enables the provision of personalized suggestions based on the user's activity records, allowing for more accurate schedule adjustments.

[0463] A "user" is a person or entity that uses the system and provides input data.

[0464] "Input data" refers to information that users provide to the system, including schedule information.

[0465] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0466] "Schedule information" refers to data that describes the user's actions and detailed schedule.

[0467] A "proposal" is a suggestion sent from the system to the user regarding schedule management and scheduling adjustments.

[0468] "Response" refers to feedback that users provide to a proposal, such as approval or modification.

[0469] "Schedule data" refers to information recorded in a user's calendar or schedule.

[0470] "History of past actions" refers to a record of activities and choices the user has made in the past.

[0471] "Personalization" refers to individually optimizing content based on each user's preferences and history.

[0472] In its embodiment, this system primarily consists of three elements: a server, a terminal, and a user. The server is responsible for performing the main processing, the user utilizes its services, and the terminal serves as the interface connecting the user and the server.

[0473] The server functions as a network-connected computer system to receive input data from users. Specifically, the server uses natural language processing technology to analyze user input data and extract planned information. For this purpose, the server is equipped with a generative AI model. The analyzed data is then used to generate suggestions, taking into account the user's past behavioral history. For example, based on a user's statement, "I want to have a picnic on the first Saturday of next month," the server will make suggestions that take into account frequently visited parks and the weather information for that day.

[0474] A terminal is a device that receives user input and notifications and suggestions from the server. Users use this terminal to review, approve, or modify suggestions. This ensures that the schedule data is updated in real time. Examples of terminals include personal digital assistants (PDAs), computers, and smartphones.

[0475] Users can manage their own schedules and events through comments and input. The user interface is intuitive, making the system easy to use even without specialized knowledge.

[0476] For example, if a user sends a LINE message saying, "I want to have a picnic on the first Saturday of next month," the server analyzes the message and sends a personalized suggestion to the user's device. If the user accepts the suggestion, the plan is added to the device's calendar. In this process, the prompt "I want to have a picnic on the first Saturday of next month. Please tell me some good places to go on that day" is used as input to the AI ​​model.

[0477] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0478] Step 1:

[0479] The server receives user input data. This input is a natural language message sent from the user's device. The user sends a message like "Let's go on a picnic on the first Saturday of next month" via LINE or other communication apps. The server receives this message as data.

[0480] Step 2:

[0481] The server performs natural language processing on the received input data. Using a generative AI model, it analyzes the user's intent and related schedule information, extracting keywords such as "the first Saturday of next month" and "picnic." This analysis structures the input data as schedule information.

[0482] Step 3:

[0483] Based on the analyzed schedule information, the server generates suggestions by referencing the user's past activity history and current situation. For example, it collects information on parks the user has visited in the past and the weather on the day, and creates suggestions such as, "How about the park on the corner?" In this process, extracted keywords are matched with historical information to derive the most suitable suggestions.

[0484] Step 4:

[0485] The server sends the generated suggestions to the user's device. The device displays the suggestions to the user and prompts for confirmation. The suggestions include potential places to visit and their advantages. The user can easily review this on the screen.

[0486] Step 5:

[0487] Users review proposals via their devices and approve or modify them. If they accept a proposal, they click the accept button. If modifications are needed, the user sends new instructions. This action allows the server to receive the user's response.

[0488] Step 6:

[0489] The device updates the schedule data based on the user's response. If approved, the new appointment is added to the calendar and the update is complete. If modifications are made, instructions are sent back to the server, which then returns to the process of further analysis.

[0490] (Application Example 1)

[0491] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0492] Traditional schedule management systems required users to manually input and manage their schedules, which was time-consuming and laborious. Furthermore, it was difficult to provide personalized suggestions that fully utilized user behavior history and environmental information.

[0493] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0494] In this invention, the server includes means for receiving user voice information and converting it from voice to text data, means for performing natural language analysis on the converted text data to extract schedule information, and means for generating and transmitting suggestions based on the extracted schedule information, taking into account the user's behavioral history and environmental information. As a result, the user can manage their schedule with intuitive voice operations and receive personalized and optimized suggestions.

[0495] "Voice information" refers to data used to recognize the words spoken by a user as digital signals.

[0496] "Text data" refers to data in sentence form obtained by analyzing audio information.

[0497] "Natural language processing" is a technology that understands the meaning of input text data and extracts necessary information.

[0498] "Schedule information" refers to data about the user's future plans and schedules.

[0499] "Activity history" refers to a record of activities a user has performed in the past.

[0500] "Environmental information" refers to data related to the user's current situation, such as weather and traffic conditions.

[0501] A "suggestion" is a proposal or recommendation generated by the system based on the user's individual conditions.

[0502] "Means of conversion" refers to methods or devices for converting audio information into text data.

[0503] "Means of transmission" refers to functions for delivering suggestions and notifications to the user's device.

[0504] This invention is a system that manages schedules based on user voice information. The system mainly consists of three elements: a server, a terminal, and the user.

[0505] The server first converts the received audio information into text data using the Google Cloud Speech-to-Text API. It then analyzes this converted text data using OpenAI's natural language processing model to extract schedule information. Based on this extracted schedule information, it analyzes the user's past activity history and externally acquired environmental information (such as weather and traffic information). Based on this, it generates personalized suggestions for the user and provides them to the user through their device.

[0506] The terminal acts as an interface with the user, receiving and displaying suggestions and notifications from the server. When the user accepts a suggestion or enters changes or supplementary information, the terminal resends this information to the server and updates the schedule data.

[0507] Users can utilize the system naturally within their homes, for example, by speaking to a robot equipped with a smart speaker. For instance, if they give a command such as, "Add a library appointment for next Wednesday," that information is processed throughout the system, and an appropriate schedule is set.

[0508] Examples of prompt statements are as follows:

[0509] User: Please write down that I plan to go to the library next Wednesday.

[0510] AI Model: Finds keywords in text, extracts the date, time, and destination, and adds them to the calendar.

[0511] This allows users to easily manage their schedules through a natural voice-based interface.

[0512] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0513] Step 1:

[0514] The user inputs voice information into the device. When the user speaks, the device receives it and sends it to the server as digital voice data. This voice data becomes the system's input.

[0515] Step 2:

[0516] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. During the conversion, the audio data is analyzed to extract the spoken content as accurate text. This text data becomes the input for the next process.

[0517] Step 3:

[0518] The server utilizes OpenAI's natural language processing model to extract schedule information from the converted text data. Specifically, the model identifies keywords and date / time information within the text and organizes them as the user's schedule data. This schedule information then serves as input for the next step.

[0519] Step 4:

[0520] The server references the user's past behavioral history data and environmental information obtained from external sources (such as weather and traffic information) to generate optimal suggestions based on the extracted schedule information. A generation AI model is used to analyze and optimize the information necessary for the suggestions. These generated suggestions are then output to the terminal.

[0521] Step 5:

[0522] The terminal notifies the user of the proposal sent from the server. The user reviews the proposal and, if they wish to accept it or make modifications, inputs them through the terminal. This user response becomes the input for the next process.

[0523] Step 6:

[0524] The server receives the user's response and updates the schedule database based on its content. After confirming the response, it saves the updated schedule information and notifies the user again if necessary. This process ensures that the user has the most up-to-date schedule.

[0525] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0526] This invention provides a system that incorporates an emotion engine that recognizes emotions from user communication. This system utilizes natural language input data provided by users within a communication application, and performs natural language analysis and emotion analysis on that data to enable personalized schedule management and suggestions for users.

[0527] The server receives messages sent by users and extracts schedule information using a natural language processing engine. It also uses an emotion engine to recognize the user's emotions from the messages. As a result, it generates suggestions and reminders based on the user's emotional state and notifies the device.

[0528] The device can receive notifications from the server and display customized messages tailored to the user's mood. The user reviews these suggestions and updates the schedule data by approving or editing them as appropriate.

[0529] As a concrete example, consider a scenario where a user says in a chat with a friend, "I'm tired, so I'm not going to do anything this weekend." The server receives this message and recognizes "fatigue" using its emotion engine. Based on this emotion information, the server generates suggestions for relaxing activities and presents a schedule that allows the user to refresh themselves without overexerting themselves.

[0530] Furthermore, in conversations involving multiple users, the system can comprehensively understand the emotional state of each user and adjust schedules while considering the overall emotional balance. This configuration enables flexible schedule management that takes into account the individual state of each user, resulting in more human-like interactions.

[0531] This invention employs a personalized approach that incorporates emotion recognition to improve the efficiency and comfort of user schedule management. This allows users to receive support optimized to their emotions and state at any given time.

[0532] The following describes the processing flow.

[0533] Step 1:

[0534] Users initiate conversations with friends and colleagues using the LINE app. During these conversations, they send messages that may contain information about their feelings or plans.

[0535] Step 2:

[0536] The terminal receives messages sent by the user and forwards those messages to the server.

[0537] Step 3:

[0538] The server passes the received message to a natural language processing engine, which extracts schedule information from the message. At the same time, it also passes the message to an emotion engine to recognize the user's emotions.

[0539] Step 4:

[0540] Based on the extracted schedule information, the server generates suggestions and reminders that take the user's emotions into consideration. For example, if the emotion of "tired" is detected, it will provide suggestions that include relaxing activities or ways to reduce tasks.

[0541] Step 5:

[0542] The server sends the generated suggestions to the user's device and notifies the user. The notification content is presented in an expression adapted to the user's emotions.

[0543] Step 6:

[0544] Users can check notifications on their devices and approve or change the content of suggestions and reminders. They can also request alternative, sentiment-based suggestions if needed.

[0545] Step 7:

[0546] The terminal sends the user's response to the server and updates the schedule data based on the user's approval.

[0547] Step 8:

[0548] The server synchronizes updated schedules with other related apps and services as needed and stores them in a central database, ensuring that the latest information is always available.

[0549] This process allows users to seamlessly manage their schedules and adjust tasks while receiving personalized support tailored to their emotional state.

[0550] (Example 2)

[0551] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0552] Traditional scheduling management systems manage schedules based on user input, but they have the problem of not being able to provide optimal suggestions to users because they do not take into account users' emotions or past activity history. Furthermore, there is a problem in scheduling adjustments that do not take into account the emotional state of individual users in communication among multiple users.

[0553] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0554] In this invention, the server includes means for receiving user input data and extracting schedule information by performing natural language processing; means for generating and sending a user-optimized suggestion based on the extracted schedule information and the user's emotional data; means for receiving the user's response to the generated suggestion and updating the schedule data; and means for analyzing the user's emotional data and performing schedule adjustments that take into account the overall emotional balance among multiple users. This enables personalized schedule suggestions that respond to the user's emotions and allows for adjustments that take emotional balance into account in communication among multiple users.

[0555] "User input data" refers to text data that users send through a communication platform.

[0556] "Natural language processing" is a technology that allows computers to understand human language and extract structured information from text.

[0557] "Schedule information" refers to information about schedule-related elements and dates / times extracted from user input data.

[0558] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their input data.

[0559] A "suggestion" is a recommended action or schedule generated based on the user's schedule information and sentiment data.

[0560] "Schedule data" refers to digital time management information that compiles a user's schedule information.

[0561] "Emotional balance" is a concept that refers to the harmony or equilibrium of emotional states among multiple users.

[0562] Personalization is the process of making adjustments and optimizations according to the individual user's characteristics and circumstances.

[0563] This invention provides a system that utilizes an emotion engine and a natural language processing engine to streamline user schedule management and provide personalized suggestions based on emotions.

[0564] The server receives messages sent by users through communication apps. A natural language processing engine is used to perform natural language processing on the received data. In this process, a natural language API is typically used as a text analysis tool to extract scheduled information from the messages. Additionally, an emotion engine analyzes the user's emotional data to identify emotions such as "joy," "sadness," and "fatigue."

[0565] The server generates personalized suggestions based on extracted schedule and sentiment data. To achieve this, it uses a suggestion generation engine to customize templates and create messages that resonate with the user's emotions. For example, if a user messages, "I'm tired, so I'm not doing anything this weekend," the server recognizes the "fatigue" and suggests relaxing activities.

[0566] The generated suggestions are notified to the device and displayed to the user. The user reviews the suggestions and updates their schedule data by approving or editing them. In this scenario, a schedule management application is installed on the device, and the schedule data is synchronized with a cloud system.

[0567] Furthermore, when multiple users communicate, the server adjusts the schedule considering the overall emotional balance and notifies each user of the adjustment results. This allows users to manage their activities in an emotionally optimized environment.

[0568] An example of a prompt for a generative AI model is: "When a user says, 'I'm tired, so I'm not going to do anything this weekend,' suggest relaxing activities and provide a schedule that addresses their feelings."

[0569] In this way, it is possible to build a system that enables flexible schedule management and proposal provision tailored to the individual needs and emotions of users.

[0570] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0571] Step 1:

[0572] The server receives messages sent by users through communication apps. The input for this operation is message data from the user, and processing begins when the server receives this data. The received data is then prepared for analysis in the next step.

[0573] Step 2:

[0574] The server uses a natural language processing engine to parse the received message. The input here is the message data received in the previous step. This process extracts schedule information and other relevant information from the message. This operation includes grammatical analysis and key phrase extraction, and the output is a set of extracted schedule information.

[0575] Step 3:

[0576] The server then uses an emotion engine to extract emotion data from the messages. The input for this process is the message data obtained in step 2. This engine analyzes the emotions in natural language and generates emotion tags such as "joy" and "sadness." The output is a dataset containing the emotion tags.

[0577] Step 4:

[0578] The server generates user-optimized suggestions based on extracted schedule information and sentiment data. The inputs for this stage are the schedule information from step 2 and the sentiment data from step 3. The server uses a template-based generation method to create user-appropriate suggestions. The output is a customized suggestion message.

[0579] Step 5:

[0580] The server notifies the user's terminal of the generated suggestion. The input for this step is the suggestion message generated in step 4. The notification from the server is pushed to the terminal, and the output is the notification message displayed on the terminal.

[0581] Step 6:

[0582] The terminal receives suggestion notifications from the server and displays them to the user. The input for this operation is the suggestion message delivered from the server, which is then displayed on the terminal. The user can then review the displayed suggestion.

[0583] Step 7:

[0584] The user reviews the proposal and approves or edits it as needed. The input for this operation is the proposal displayed on the terminal. The schedule data is updated based on the user's instructions, and this update is synchronized with the cloud system. The output is the updated schedule data.

[0585] Step 8:

[0586] The server adjusts schedules while considering the overall emotional balance among multiple users. Inputs at this stage include emotional data and schedules from individual users. The server integrates the emotional data to create an optimal schedule for all users. The output is the adjusted schedule information, which is then notified to each user.

[0587] (Application Example 2)

[0588] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0589] In modern society, schedule management is a crucial daily task, and there is a particular demand for personalized schedule suggestions that take into account the user's emotional state. However, conventional systems struggle to adequately reflect the user's emotional state in schedule management, limiting their ability to improve the user's quality of life. Furthermore, when coordinating schedules among multiple users, uniform adjustments are made without regard for each user's feelings, resulting in adjustments that are not optimal for the user. These challenges need to be addressed.

[0590] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0591] In this invention, the server includes means for receiving user input data and performing natural language analysis to extract schedule information and sentiment information; means for generating and sending suggestions to the user based on the extracted schedule information and sentiment information; and means for receiving the user's response to the generated suggestions and updating schedule data based on the sentiment state. This enables personalized schedule management that takes the user's emotions into consideration.

[0592] "Means for receiving user input data and performing natural language analysis to extract schedule information and emotional information" refers to a system that analyzes and identifies content related to specific schedules and the user's emotional state from the natural language data entered by the user.

[0593] "A means of generating and sending suggestions to users based on extracted schedule and sentiment information" refers to a function that utilizes the schedule and sentiment information obtained through analysis to create optimized suggestions for users and send them to them.

[0594] "A means of receiving user responses to generated suggestions and updating schedule data based on emotional state" refers to a function that, after receiving feedback from the user, updates the schedule information in light of that feedback and the user's emotional state.

[0595] "Means for analyzing a user's past activity and emotional history to personalize suggestions" refers to a function that evaluates a user's past behavioral records and emotional changes, and then individualizes and makes more appropriate suggestions based on that evaluation.

[0596] "A means of coordinating schedules among multiple users and notifying them of the results while taking into account each user's emotional state" refers to a function that aligns the schedules of all participating users while communicating the adjustment results while considering each user's emotional needs.

[0597] The system that implements this application consists of a server that receives and processes user input data in natural language, and a terminal that displays the results and enables interaction with the user.

[0598] The server is built using programming languages ​​such as Python and employs libraries like NLTK and Transformers for natural language processing. This allows it to extract schedule and sentiment information from user messages. For sentiment analysis, it uses pre-trained models such as BERT to identify the user's emotional state.

[0599] The terminal displays suggestions sent from the server to the user, receives the user's response, and sends it back to the server. The system's adaptability and efficiency are improved when the user approves or edits the suggestions. The terminal could be a smartphone, tablet, or even a consumer robot.

[0600] For example, if a user says in a conversation with a friend, "I've been so busy lately, I'm not feeling very enthusiastic," the server receives this and analyzes it using its emotion engine. Then, taking into account past activity history and emotional tendencies, it suggests relaxing activities or changes to plans. For instance, it might display a personalized message on the device such as, "How about taking some time this weekend to enjoy a relaxing hobby?"

[0601] An example of a prompt for a generative AI model is: "Generate a sentence to suggest when a user says they're not feeling up to it. Example: 'You seem a little tired today....'" This prompt is designed to guide the AI ​​to determine the most appropriate suggestion for the user's emotional state.

[0602] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0603] Step 1:

[0604] The server receives the user's input message in natural language. It captures the input message as text data and prepares it for the next processing step.

[0605] Step 2:

[0606] The server analyzes the received text data using a natural language processing engine. Specifically, it performs morphological analysis using the Python NLTK library to extract schedule information and sentiment information from the message. The input here is the user's text data, and the output is the analyzed schedule information and sentiment information.

[0607] Step 3:

[0608] The server uses an emotion engine to extract emotional information, which is then analyzed by a pre-trained model such as BERT to identify the user's emotional state. In this step, the input is the emotional information obtained in the previous step, and the output is a label representing the user's emotional state (e.g., joy, sadness, fatigue).

[0609] Step 4:

[0610] The server generates personalized suggestions based on analyzed schedule information and emotional state. Past activity and emotional history are also considered in the suggestion generation process. Using a generative AI model, the prompt "Generate a suggestion for when the user says they're not in the mood" is given to create the suggestion content. The input here is emotional state and past history, and the output is a personalized suggestion message.

[0611] Step 5:

[0612] The server sends the generated suggestion message to the terminal. The terminal notifies the user of this suggestion message and waits for user confirmation. The input is the suggestion message from the server, and the output is a visual or audio notification to the user.

[0613] Step 6:

[0614] The user receives suggestions from their terminal and approves or edits them. The decisions made by the user are sent back to the server, and the schedule data is updated. The input is user feedback, and the output is the updated schedule data.

[0615] Step 7:

[0616] When scheduling adjustments are needed for multiple users, the server takes each user's emotional state into consideration while making the adjustments. It then notifies each user of the results and obtains their consent. The input consists of each user's emotional data and schedule information, and the output is the adjusted schedule notification.

[0617] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0618] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0620] [Fourth Embodiment]

[0621] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0622] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0623] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0625] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0627] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0628] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0629] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0630] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0631] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0632] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0633] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] To implement this invention, the system is configured to function primarily through three elements: a server, a terminal, and a user. The server receives input data from the user and processes this information through natural language processing. It is equipped with an AI model for natural language processing and has the ability to efficiently extract schedule information from the user's conversation. This approach enables complex schedule management from the user's intuitive input.

[0635] The terminal is a device used directly by the user and plays the role of receiving notifications and suggestions from the server and presenting them to the user. Schedule data and task information are updated in response to user actions (e.g., approving or editing suggestions).

[0636] As a concrete example, consider a scenario where a user says in a LINE chat with a friend, "Let's go on a picnic on the first Saturday of next month." When the server receives this message, it uses natural language processing to extract information about the plan, such as "the first Saturday of next month" and "picnic." Based on this extracted information, the server generates personalized suggestions for places to visit, such as parks the user frequently visits or weather forecast services, and notifies the user's device. When the user confirms and accepts these suggestions, the device updates its schedule data and adds the event to the calendar.

[0637] Furthermore, the server refers to the user's past activity history to provide a means of further personalizing the suggestions. By suggesting more appropriate options to the user, it helps them manage their schedule quickly and accurately.

[0638] This invention makes it possible to transform ordinary communication apps into advanced planning and management tools, enriching users' lives.

[0639] The following describes the processing flow.

[0640] Step 1:

[0641] Users use the LINE app for everyday conversations. During this process, they send messages related to their schedules and tasks.

[0642] Step 2:

[0643] The terminal receives the user's message and prepares to forward it to the server.

[0644] Step 3:

[0645] The server retrieves the received message and analyzes it using a natural language processing engine. This extracts keywords related to the schedule and tasks.

[0646] Step 4:

[0647] The server generates suggestions or reminders based on the analysis results. This includes calculating potential schedules and task priorities.

[0648] Step 5:

[0649] The server sends the generated suggestions to the terminal and notifies the user. This notification includes an interface for review and editing.

[0650] Step 6:

[0651] Users can review proposals through their devices and approve or edit them as needed.

[0652] Step 7:

[0653] The terminal sends the user's actions back to the server.

[0654] Step 8:

[0655] The server updates the schedule data based on the user's response. This ensures that the user's schedule remains up-to-date.

[0656] Step 9:

[0657] If necessary, the server synchronizes with related services to ensure consistency with external calendar apps and task management tools.

[0658] (Example 1)

[0659] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0660] In users' daily lives, intuitively managing schedules is difficult. Furthermore, traditional methods struggled to efficiently process user input data and provide appropriate suggestions. Moreover, providing personalized suggestions based on each user's individual activity history was not easy.

[0661] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0662] In this invention, the server includes means for receiving user input data, analyzing the information using natural language processing technology to extract schedule information, generating suggestions to the user based on the extracted schedule information by referring to related information, and sending those suggestions, and means for receiving the user's response to the generated suggestions, updating the information based on the response, and managing the schedule data. This enables users to manage their schedules easily and efficiently. Furthermore, it enables the provision of personalized suggestions based on the user's activity records, allowing for more accurate schedule adjustments.

[0663] A "user" is a person or entity that uses the system and provides input data.

[0664] "Input data" refers to information that users provide to the system, including schedule information.

[0665] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0666] "Schedule information" refers to data that describes the user's actions and detailed schedule.

[0667] A "proposal" is a suggestion sent from the system to the user regarding schedule management and scheduling adjustments.

[0668] "Response" refers to feedback that users provide to a proposal, such as approval or modification.

[0669] "Schedule data" refers to information recorded in a user's calendar or schedule.

[0670] "History of past actions" refers to a record of activities and choices the user has made in the past.

[0671] "Personalization" refers to individually optimizing content based on each user's preferences and history.

[0672] In its embodiment, this system primarily consists of three elements: a server, a terminal, and a user. The server is responsible for performing the main processing, the user utilizes its services, and the terminal serves as the interface connecting the user and the server.

[0673] The server functions as a network-connected computer system to receive input data from users. Specifically, the server uses natural language processing technology to analyze user input data and extract planned information. For this purpose, the server is equipped with a generative AI model. The analyzed data is then used to generate suggestions, taking into account the user's past behavioral history. For example, based on a user's statement, "I want to have a picnic on the first Saturday of next month," the server will make suggestions that take into account frequently visited parks and the weather information for that day.

[0674] A terminal is a device that receives user input and notifications and suggestions from the server. Users use this terminal to review, approve, or modify suggestions. This ensures that the schedule data is updated in real time. Examples of terminals include personal digital assistants (PDAs), computers, and smartphones.

[0675] Users can manage their own schedules and events through comments and input. The user interface is intuitive, making the system easy to use even without specialized knowledge.

[0676] For example, if a user sends a LINE message saying, "I want to have a picnic on the first Saturday of next month," the server analyzes the message and sends a personalized suggestion to the user's device. If the user accepts the suggestion, the plan is added to the device's calendar. In this process, the prompt "I want to have a picnic on the first Saturday of next month. Please tell me some good places to go on that day" is used as input to the AI ​​model.

[0677] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0678] Step 1:

[0679] The server receives user input data. This input is a natural language message sent from the user's device. The user sends a message like "Let's go on a picnic on the first Saturday of next month" via LINE or other communication apps. The server receives this message as data.

[0680] Step 2:

[0681] The server performs natural language processing on the received input data. Using a generative AI model, it analyzes the user's intent and related schedule information, extracting keywords such as "the first Saturday of next month" and "picnic." This analysis structures the input data as schedule information.

[0682] Step 3:

[0683] Based on the analyzed schedule information, the server generates suggestions by referencing the user's past activity history and current situation. For example, it collects information on parks the user has visited in the past and the weather on the day, and creates suggestions such as, "How about the park on the corner?" In this process, extracted keywords are matched with historical information to derive the most suitable suggestions.

[0684] Step 4:

[0685] The server sends the generated suggestions to the user's device. The device displays the suggestions to the user and prompts for confirmation. The suggestions include potential places to visit and their advantages. The user can easily review this on the screen.

[0686] Step 5:

[0687] Users review proposals via their devices and approve or modify them. If they accept a proposal, they click the accept button. If modifications are needed, the user sends new instructions. This action allows the server to receive the user's response.

[0688] Step 6:

[0689] The device updates the schedule data based on the user's response. If approved, the new appointment is added to the calendar and the update is complete. If modifications are made, instructions are sent back to the server, which then returns to the process of further analysis.

[0690] (Application Example 1)

[0691] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0692] Traditional schedule management systems required users to manually input and manage their schedules, which was time-consuming and laborious. Furthermore, it was difficult to provide personalized suggestions that fully utilized user behavior history and environmental information.

[0693] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0694] In this invention, the server includes means for receiving user voice information and converting it from voice to text data, means for performing natural language analysis on the converted text data to extract schedule information, and means for generating and transmitting suggestions based on the extracted schedule information, taking into account the user's behavioral history and environmental information. As a result, the user can manage their schedule with intuitive voice operations and receive personalized and optimized suggestions.

[0695] "Voice information" refers to data used to recognize the words spoken by a user as digital signals.

[0696] "Text data" refers to data in sentence form obtained by analyzing audio information.

[0697] "Natural language processing" is a technology that understands the meaning of input text data and extracts necessary information.

[0698] "Schedule information" refers to data about the user's future plans and schedules.

[0699] "Activity history" refers to a record of activities a user has performed in the past.

[0700] "Environmental information" refers to data related to the user's current situation, such as weather and traffic conditions.

[0701] A "suggestion" is a proposal or recommendation generated by the system based on the user's individual conditions.

[0702] "Means of conversion" refers to methods or devices for converting audio information into text data.

[0703] "Means of transmission" refers to functions for delivering suggestions and notifications to the user's device.

[0704] This invention is a system that manages schedules based on user voice information. The system mainly consists of three elements: a server, a terminal, and the user.

[0705] The server first converts the received audio information into text data using the Google Cloud Speech-to-Text API. It then analyzes this converted text data using OpenAI's natural language processing model to extract schedule information. Based on this extracted schedule information, it analyzes the user's past activity history and externally acquired environmental information (such as weather and traffic information). Based on this, it generates personalized suggestions for the user and provides them to the user through their device.

[0706] The terminal acts as an interface with the user, receiving and displaying suggestions and notifications from the server. When the user accepts a suggestion or enters changes or supplementary information, the terminal resends this information to the server and updates the schedule data.

[0707] Users can utilize the system naturally within their homes, for example, by speaking to a robot equipped with a smart speaker. For instance, if they give a command such as, "Add a library appointment for next Wednesday," that information is processed throughout the system, and an appropriate schedule is set.

[0708] Examples of prompt statements are as follows:

[0709] User: Please write down that I plan to go to the library next Wednesday.

[0710] AI Model: Finds keywords in text, extracts the date, time, and destination, and adds them to the calendar.

[0711] This allows users to easily manage their schedules through a natural voice-based interface.

[0712] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0713] Step 1:

[0714] The user inputs voice information into the device. When the user speaks, the device receives it and sends it to the server as digital voice data. This voice data becomes the system's input.

[0715] Step 2:

[0716] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. During the conversion, the audio data is analyzed to extract the spoken content as accurate text. This text data becomes the input for the next process.

[0717] Step 3:

[0718] The server utilizes OpenAI's natural language processing model to extract schedule information from the converted text data. Specifically, the model identifies keywords and date / time information within the text and organizes them as the user's schedule data. This schedule information then serves as input for the next step.

[0719] Step 4:

[0720] The server references the user's past behavioral history data and environmental information obtained from external sources (such as weather and traffic information) to generate optimal suggestions based on the extracted schedule information. A generation AI model is used to analyze and optimize the information necessary for the suggestions. These generated suggestions are then output to the terminal.

[0721] Step 5:

[0722] The terminal notifies the user of the proposal sent from the server. The user reviews the proposal and, if they wish to accept it or make modifications, inputs them through the terminal. This user response becomes the input for the next process.

[0723] Step 6:

[0724] The server receives the user's response and updates the schedule database based on its content. After confirming the response, it saves the updated schedule information and notifies the user again if necessary. This process ensures that the user has the most up-to-date schedule.

[0725] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0726] This invention provides a system that incorporates an emotion engine that recognizes emotions from user communication. This system utilizes natural language input data provided by users within a communication application, and performs natural language analysis and emotion analysis on that data to enable personalized schedule management and suggestions for users.

[0727] The server receives messages sent by users and extracts schedule information using a natural language processing engine. It also uses an emotion engine to recognize the user's emotions from the messages. As a result, it generates suggestions and reminders based on the user's emotional state and notifies the device.

[0728] The device can receive notifications from the server and display customized messages tailored to the user's mood. The user reviews these suggestions and updates the schedule data by approving or editing them as appropriate.

[0729] As a concrete example, consider a scenario where a user says in a chat with a friend, "I'm tired, so I'm not going to do anything this weekend." The server receives this message and recognizes "fatigue" using its emotion engine. Based on this emotion information, the server generates suggestions for relaxing activities and presents a schedule that allows the user to refresh themselves without overexerting themselves.

[0730] Furthermore, in conversations involving multiple users, the system can comprehensively understand the emotional state of each user and adjust schedules while considering the overall emotional balance. This configuration enables flexible schedule management that takes into account the individual state of each user, resulting in more human-like interactions.

[0731] This invention employs a personalized approach that incorporates emotion recognition to improve the efficiency and comfort of user schedule management. This allows users to receive support optimized to their emotions and state at any given time.

[0732] The following describes the processing flow.

[0733] Step 1:

[0734] Users initiate conversations with friends and colleagues using the LINE app. During these conversations, they send messages that may contain information about their feelings or plans.

[0735] Step 2:

[0736] The terminal receives messages sent by the user and forwards those messages to the server.

[0737] Step 3:

[0738] The server passes the received message to a natural language processing engine, which extracts schedule information from the message. At the same time, it also passes the message to an emotion engine to recognize the user's emotions.

[0739] Step 4:

[0740] Based on the extracted schedule information, the server generates suggestions and reminders that take the user's emotions into consideration. For example, if the emotion of "tired" is detected, it will provide suggestions that include relaxing activities or ways to reduce tasks.

[0741] Step 5:

[0742] The server sends the generated suggestions to the user's device and notifies the user. The notification content is presented in an expression adapted to the user's emotions.

[0743] Step 6:

[0744] Users can check notifications on their devices and approve or change the content of suggestions and reminders. They can also request alternative, sentiment-based suggestions if needed.

[0745] Step 7:

[0746] The terminal sends the user's response to the server and updates the schedule data based on the user's approval.

[0747] Step 8:

[0748] The server synchronizes updated schedules with other related apps and services as needed and stores them in a central database, ensuring that the latest information is always available.

[0749] This process allows users to seamlessly manage their schedules and adjust tasks while receiving personalized support tailored to their emotional state.

[0750] (Example 2)

[0751] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0752] Traditional scheduling management systems manage schedules based on user input, but they have the problem of not being able to provide optimal suggestions to users because they do not take into account users' emotions or past activity history. Furthermore, there is a problem in scheduling adjustments that do not take into account the emotional state of individual users in communication among multiple users.

[0753] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0754] In this invention, the server includes means for receiving user input data and extracting schedule information by performing natural language processing; means for generating and sending a user-optimized suggestion based on the extracted schedule information and the user's emotional data; means for receiving the user's response to the generated suggestion and updating the schedule data; and means for analyzing the user's emotional data and performing schedule adjustments that take into account the overall emotional balance among multiple users. This enables personalized schedule suggestions that respond to the user's emotions and allows for adjustments that take emotional balance into account in communication among multiple users.

[0755] "User input data" refers to text data that users send through a communication platform.

[0756] "Natural language processing" is a technology that allows computers to understand human language and extract structured information from text.

[0757] "Schedule information" refers to information about schedule-related elements and dates / times extracted from user input data.

[0758] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their input data.

[0759] A "suggestion" is a recommended action or schedule generated based on the user's schedule information and sentiment data.

[0760] "Schedule data" refers to digital time management information that compiles a user's schedule information.

[0761] "Emotional balance" is a concept that refers to the harmony or equilibrium of emotional states among multiple users.

[0762] Personalization is the process of making adjustments and optimizations according to the individual user's characteristics and circumstances.

[0763] This invention provides a system that utilizes an emotion engine and a natural language processing engine to streamline user schedule management and provide personalized suggestions based on emotions.

[0764] The server receives messages sent by users through communication apps. A natural language processing engine is used to perform natural language processing on the received data. In this process, a natural language API is typically used as a text analysis tool to extract scheduled information from the messages. Additionally, an emotion engine analyzes the user's emotional data to identify emotions such as "joy," "sadness," and "fatigue."

[0765] The server generates personalized suggestions based on extracted schedule and sentiment data. To achieve this, it uses a suggestion generation engine to customize templates and create messages that resonate with the user's emotions. For example, if a user messages, "I'm tired, so I'm not doing anything this weekend," the server recognizes the "fatigue" and suggests relaxing activities.

[0766] The generated suggestions are notified to the device and displayed to the user. The user reviews the suggestions and updates their schedule data by approving or editing them. In this scenario, a schedule management application is installed on the device, and the schedule data is synchronized with a cloud system.

[0767] Furthermore, when multiple users communicate, the server adjusts the schedule considering the overall emotional balance and notifies each user of the adjustment results. This allows users to manage their activities in an emotionally optimized environment.

[0768] An example of a prompt for a generative AI model is: "When a user says, 'I'm tired, so I'm not going to do anything this weekend,' suggest relaxing activities and provide a schedule that addresses their feelings."

[0769] In this way, it is possible to build a system that enables flexible schedule management and proposal provision tailored to the individual needs and emotions of users.

[0770] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0771] Step 1:

[0772] The server receives messages sent by users through communication apps. The input for this operation is message data from the user, and processing begins when the server receives this data. The received data is then prepared for analysis in the next step.

[0773] Step 2:

[0774] The server uses a natural language processing engine to parse the received message. The input here is the message data received in the previous step. This process extracts schedule information and other relevant information from the message. This operation includes grammatical analysis and key phrase extraction, and the output is a set of extracted schedule information.

[0775] Step 3:

[0776] The server then uses an emotion engine to extract emotion data from the messages. The input for this process is the message data obtained in step 2. This engine analyzes the emotions in natural language and generates emotion tags such as "joy" and "sadness." The output is a dataset containing the emotion tags.

[0777] Step 4:

[0778] The server generates user-optimized suggestions based on extracted schedule information and sentiment data. The inputs for this stage are the schedule information from step 2 and the sentiment data from step 3. The server uses a template-based generation method to create user-appropriate suggestions. The output is a customized suggestion message.

[0779] Step 5:

[0780] The server notifies the user's terminal of the generated suggestion. The input for this step is the suggestion message generated in step 4. The notification from the server is pushed to the terminal, and the output is the notification message displayed on the terminal.

[0781] Step 6:

[0782] The terminal receives suggestion notifications from the server and displays them to the user. The input for this operation is the suggestion message delivered from the server, which is then displayed on the terminal. The user can then review the displayed suggestion.

[0783] Step 7:

[0784] The user reviews the proposal and approves or edits it as needed. The input for this operation is the proposal displayed on the terminal. The schedule data is updated based on the user's instructions, and this update is synchronized with the cloud system. The output is the updated schedule data.

[0785] Step 8:

[0786] The server adjusts schedules while considering the overall emotional balance among multiple users. Inputs at this stage include emotional data and schedules from individual users. The server integrates the emotional data to create an optimal schedule for all users. The output is the adjusted schedule information, which is then notified to each user.

[0787] (Application Example 2)

[0788] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0789] In modern society, schedule management is a crucial daily task, and there is a particular demand for personalized schedule suggestions that take into account the user's emotional state. However, conventional systems struggle to adequately reflect the user's emotional state in schedule management, limiting their ability to improve the user's quality of life. Furthermore, when coordinating schedules among multiple users, uniform adjustments are made without regard for each user's feelings, resulting in adjustments that are not optimal for the user. These challenges need to be addressed.

[0790] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0791] In this invention, the server includes means for receiving user input data and performing natural language analysis to extract schedule information and sentiment information; means for generating and sending suggestions to the user based on the extracted schedule information and sentiment information; and means for receiving the user's response to the generated suggestions and updating schedule data based on the sentiment state. This enables personalized schedule management that takes the user's emotions into consideration.

[0792] "Means for receiving user input data and performing natural language analysis to extract schedule information and emotional information" refers to a system that analyzes and identifies content related to specific schedules and the user's emotional state from the natural language data entered by the user.

[0793] "A means of generating and sending suggestions to users based on extracted schedule and sentiment information" refers to a function that utilizes the schedule and sentiment information obtained through analysis to create optimized suggestions for users and send them to them.

[0794] "A means of receiving user responses to generated suggestions and updating schedule data based on emotional state" refers to a function that, after receiving feedback from the user, updates the schedule information in light of that feedback and the user's emotional state.

[0795] "Means for analyzing a user's past activity and emotional history to personalize suggestions" refers to a function that evaluates a user's past behavioral records and emotional changes, and then individualizes and makes more appropriate suggestions based on that evaluation.

[0796] "A means of coordinating schedules among multiple users and notifying them of the results while taking into account each user's emotional state" refers to a function that aligns the schedules of all participating users while communicating the adjustment results while considering each user's emotional needs.

[0797] The system that implements this application consists of a server that receives and processes user input data in natural language, and a terminal that displays the results and enables interaction with the user.

[0798] The server is built using programming languages ​​such as Python and employs libraries like NLTK and Transformers for natural language processing. This allows it to extract schedule and sentiment information from user messages. For sentiment analysis, it uses pre-trained models such as BERT to identify the user's emotional state.

[0799] The terminal displays suggestions sent from the server to the user, receives the user's response, and sends it back to the server. The system's adaptability and efficiency are improved when the user approves or edits the suggestions. The terminal could be a smartphone, tablet, or even a consumer robot.

[0800] For example, if a user says in a conversation with a friend, "I've been so busy lately, I'm not feeling very enthusiastic," the server receives this and analyzes it using its emotion engine. Then, taking into account past activity history and emotional tendencies, it suggests relaxing activities or changes to plans. For instance, it might display a personalized message on the device such as, "How about taking some time this weekend to enjoy a relaxing hobby?"

[0801] An example of a prompt for a generative AI model is: "Generate a sentence to suggest when a user says they're not feeling up to it. Example: 'You seem a little tired today....'" This prompt is designed to guide the AI ​​to determine the most appropriate suggestion for the user's emotional state.

[0802] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0803] Step 1:

[0804] The server receives the user's input message in natural language. It captures the input message as text data and prepares it for the next processing step.

[0805] Step 2:

[0806] The server analyzes the received text data using a natural language processing engine. Specifically, it performs morphological analysis using the Python NLTK library to extract schedule information and sentiment information from the message. The input here is the user's text data, and the output is the analyzed schedule information and sentiment information.

[0807] Step 3:

[0808] The server uses an emotion engine to extract emotional information, which is then analyzed by a pre-trained model such as BERT to identify the user's emotional state. In this step, the input is the emotional information obtained in the previous step, and the output is a label representing the user's emotional state (e.g., joy, sadness, fatigue).

[0809] Step 4:

[0810] The server generates personalized suggestions based on analyzed schedule information and emotional state. Past activity and emotional history are also considered in the suggestion generation process. Using a generative AI model, the prompt "Generate a suggestion for when the user says they're not in the mood" is given to create the suggestion content. The input here is emotional state and past history, and the output is a personalized suggestion message.

[0811] Step 5:

[0812] The server sends the generated suggestion message to the terminal. The terminal notifies the user of this suggestion message and waits for user confirmation. The input is the suggestion message from the server, and the output is a visual or audio notification to the user.

[0813] Step 6:

[0814] The user receives suggestions from their terminal and approves or edits them. The decisions made by the user are sent back to the server, and the schedule data is updated. The input is user feedback, and the output is the updated schedule data.

[0815] Step 7:

[0816] When scheduling adjustments are needed for multiple users, the server takes each user's emotional state into consideration while making the adjustments. It then notifies each user of the results and obtains their consent. The input consists of each user's emotional data and schedule information, and the output is the adjusted schedule notification.

[0817] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0818] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0819] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0820] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0821] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0822] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0823] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0824] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0825] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0826] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0827] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0828] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0829] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0830] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0831] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0832] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0833] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0834] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0835] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0836] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0837] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0838] The following is further disclosed regarding the embodiments described above.

[0839] (Claim 1)

[0840] A means of receiving user input data and extracting schedule information by performing natural language processing,

[0841] A means of generating and sending proposals to users based on extracted schedule information,

[0842] A means for receiving user responses to generated proposals and updating schedule data,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, comprising means for analyzing a user's past activity history and personalizing suggestions.

[0846] (Claim 3)

[0847] The system according to claim 1, comprising means for coordinating schedules among multiple users and notifying each user of the results.

[0848] "Example 1"

[0849] (Claim 1)

[0850] A means for receiving user input data, analyzing the information using natural language processing technology, and extracting planned information,

[0851] A means for generating a proposal to the user based on extracted schedule information, referring to related information, and sending that proposal,

[0852] A means for receiving user responses to generated proposals, updating information based on those responses, and managing schedule data,

[0853] A means of individually optimizing suggestions by referring to the history of actions taken by the user in the past,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, comprising means for improving the accuracy of suggestions based on user activity records.

[0857] (Claim 3)

[0858] The system according to claim 1, comprising means for coordinating the schedule of activities among multiple users and notifying each user of the details of the coordination.

[0859] "Application Example 1"

[0860] (Claim 1)

[0861] A means for receiving user voice information and converting it from voice to text data,

[0862] A method for performing natural language processing on converted text data to extract planned information,

[0863] A means of generating and sending suggestions based on extracted schedule information, taking into account the user's behavioral history and environmental information,

[0864] A means of receiving user responses to generated proposals and updating the schedule database,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, which has means for referencing weather information and traffic conditions when generating suggestions and for suggesting the optimal route and time for a visit.

[0868] (Claim 3)

[0869] The system according to claim 1, which includes means for extracting information from speech using an external data analysis API when performing speech recognition and natural language processing.

[0870] "Example 2 of combining an emotion engine"

[0871] (Claim 1)

[0872] A means of receiving user input data and extracting schedule information by performing natural language processing,

[0873] A means for generating and sending user-optimized suggestions based on extracted schedule information and user sentiment data,

[0874] A means for receiving user responses to generated proposals and updating schedule data,

[0875] A means of analyzing user emotional data and adjusting schedules while considering the overall emotional balance among multiple users,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, comprising means for analyzing the user's past activity history and sentiment data to further personalize suggestions.

[0879] (Claim 3)

[0880] The system according to claim 1, comprising means for scheduling among multiple users and sending the results as a notification based on each user's emotional state.

[0881] "Application example 2 when combining with an emotional engine"

[0882] (Claim 1)

[0883] A means for receiving user input data and performing natural language analysis to extract planned information and sentiment information,

[0884] A means for generating and sending suggestions to the user based on extracted schedule information and sentiment information,

[0885] A means for receiving user responses to generated suggestions and updating schedule data based on emotional state,

[0886] A system that includes this.

[0887] (Claim 2)

[0888] The system according to claim 1, comprising means for analyzing a user's past activity history and sentiment history to personalize suggestions.

[0889] (Claim 3)

[0890] The system according to claim 1, comprising means for scheduling among multiple users and notifying each user of the results while taking into consideration their emotional state. [Explanation of Symbols]

[0891] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving user voice information and converting it from voice to text data, A method for performing natural language processing on converted text data to extract planned information, A means of generating and sending suggestions based on extracted schedule information, taking into account the user's behavioral history and environmental information, A means of receiving user responses to generated proposals and updating the schedule database, A system that includes this.

2. The system according to claim 1, which has means for referencing weather information and traffic conditions when generating suggestions and for suggesting the optimal route and time for a visit.

3. The system according to claim 1, which includes means for extracting information from speech using an external data analysis API when performing speech recognition and natural language processing.

Images