System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving data input by a user; means for performing authentication of the user; means for generating a response to the input of the user by using generative artificial intelligence; means for providing a response result to the user; means for managing schedule information of the user and performing remind notification; and means for collecting and analyzing health data of the user and performing notification at the time of abnormality.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a need for a unified solution to the multiple challenges people face in their daily lives and health management. Specifically, there is a lack of systems that efficiently and effectively manage schedules, track health data, detect anomalies, and support daily communication. To solve these challenges, a comprehensive system that uses generative artificial intelligence to realize natural dialogue with users and can respond to their diverse needs is needed. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. A system is constructed that includes a means for receiving data entered by a user, a means for authenticating the user, a means for generating a response to the user's input using generative artificial intelligence, a means for providing the response results to the user, a means for managing the user's schedule information and sending reminder notifications, and a means for collecting and analyzing the user's health data and sending notifications in the event of an abnormality. Furthermore, comprehensive functionality is provided by adding a means for using a machine learning algorithm to analyze the health data, receiving the user's input as voice data and text data, and converting the voice data into text data using voice recognition technology. This system allows users to centrally manage their health while receiving the support they need in their daily lives.

[0006] "User" refers to any individual or end user who uses the System.

[0007] "Data" includes information that users input into the system, such as text, voice, schedule information, health data, etc.

[0008] "Authentication" is the process of verifying a user's identity and confirming that they are a legitimate user.

[0009] "Generative artificial intelligence" refers to artificial intelligence technology, specifically generative AI models, that enable natural interactions in response to user input.

[0010] "Response" refers to the answer or reaction provided by the system in response to user input.

[0011] "Reminder notification" refers to the sending of a reminder message based on the user's schedule information.

[0012] "Health data" includes information about the user's health condition, specifically the contents of the user's medicine record and walking records.

[0013] "Analysis" is the process of analyzing collected data and extracting significant information and patterns.

[0014] "Anomaly detection" refers to monitoring collected data to identify abnormal or out-of-control conditions.

[0015] "Notification" refers to the transmission of warnings and information to inform relevant parties of abnormality detection and other necessary information.

[0016] A "machine learning algorithm" is a type of algorithm that learns patterns from data and makes predictions and classifications.

[0017] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text data. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system of the present invention is constructed to allow users to perform various tasks related to daily life and health management. The functions of each component and the processing flow will be specifically explained below.

[0040] Receiving data entered by the user

[0041] User

[0042] Users use devices such as smartphones and tablets to input data into the app by text or voice, including schedules, health data, and everyday conversations.

[0043] Terminal

[0044] The device receives user input and sends it to the server as text data. If voice data is input, it is converted into text using voice recognition technology.

[0045] User authentication

[0046] User

[0047] When users start using the app, they enter their username and password.

[0048] Terminal

[0049] The terminal transmits these authentication data to the server.

[0050] server

[0051] The server accesses the database and performs authentication based on the user information. If authentication is successful, a session is started and an authentication token is sent to the terminal. If authentication fails, an error message is returned.

[0052] Response generation by generative artificial intelligence

[0053] server

[0054] The server uses a generative AI (e.g., GPT-4) to generate a response based on the user's input. The generated response is temporarily stored on the server.

[0055] Terminal

[0056] The terminal receives the generated response from the server and provides it to the user in text and audio.

[0057] Schedule management and reminder notifications

[0058] User

[0059] Users input their future plans into the app.

[0060] Terminal

[0061] This input is sent to the server.

[0062] server

[0063] The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, the server generates the corresponding notification.

[0064] Terminal

[0065] The device receives the reminder notification and notifies the user, for example, "A reminder to take your medicine has been set for 10:00 AM."

[0066] Health data collection and anomaly detection

[0067] User

[0068] Users enter their medication record and walking records into the app.

[0069] Terminal

[0070] The entered health data is sent to the server.

[0071] server

[0072] The server stores this data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, it notifies the specified contacts (e.g., family members, medical institutions).

[0073] Terminal

[0074] Reports detected anomalies to users and provides instructions on what to do.

[0075] Specific examples

[0076] 1. Set schedule reminders

[0077] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[0078] Device: Converts speech to text and sends it to the server.

[0079] Server: Stores reminders in a database and generates notifications at the scheduled times.

[0080] Device: Receive reminder notifications and notify the user.

[0081] 2. Detecting Anomalies in Health Data

[0082] User: Updates walking record and enters it into the app.

[0083] Terminal: Sends data to the server.

[0084] Server: Analyzes data, detects abnormalities, and sends relevant notifications to family members and medical institutions.

[0085] Device: Notify the user that an anomaly has been detected.

[0086] The system of the present invention makes it possible to streamline the user's lifestyle and health management and provide comprehensive daily support.

[0087] The processing flow will be explained below.

[0088] Step 1:

[0089] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[0090] Step 2:

[0091] Terminal: Sends the entered username and password to the server.

[0092] Step 3:

[0093] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[0094] Step 4:

[0095] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[0096] Step 5:

[0097] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[0098] Step 6:

[0099] Terminal: Sends user input to the server.

[0100] Step 7:

[0101] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[0102] Step 8:

[0103] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[0104] Step 9:

[0105] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[0106] Step 10:

[0107] Server: Generates a notification based on the reminder content.

[0108] Step 11:

[0109] Device: Receive reminder notifications and notify users via text and voice.

[0110] Step 12:

[0111] User: Enter new data from your medication record or walking log into the app.

[0112] Step 13:

[0113] Terminal: Sends the entered health data to the server.

[0114] Step 14:

[0115] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[0116] Step 15:

[0117] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[0118] Step 16:

[0119] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[0120] Step 17:

[0121] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[0122] Step 18:

[0123] User: Type everyday conversations and questions into the app or speak into the microphone.

[0124] Step 19:

[0125] Terminal: Sends user input or voice data to the server.

[0126] Step 20:

[0127] Server: When voice input is received, it converts it into text using speech recognition technology, and uses generative AI to generate appropriate responses to user questions and conversations.

[0128] Step 21:

[0129] Terminal: Receives the generated response from the server and provides it to the user in text and voice.

[0130] Example 1

[0131] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] In modern society, it is important for individual users to efficiently manage their daily lives and health, and there is a growing demand for systems that support this. However, conventional systems do not integrate various functions such as receiving user data, authenticating users, generating responses, managing schedules, and analyzing health data, which makes them difficult to use and makes it difficult to manage efficiently. Furthermore, there are issues with the recognition accuracy and response speed when using voice input.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0134] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative artificial intelligence model, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying in the event of an abnormality, and means for converting voice input into text data using voice recognition technology. This enables the user to efficiently manage their daily life and health using a single system, and also improves the recognition accuracy and response speed for voice input.

[0135] The "means for receiving data entered by the user" is a function that receives information entered by the user via text or voice to the server via the terminal.

[0136] A "means of authenticating a user" is the process of identifying a user using authentication information, such as a username and password, and initiating a session on a server.

[0137] "Means for generating a response to user input using a generative artificial intelligence model" refers to a technology that analyzes input data from a user and generates an appropriate response using an artificial intelligence model.

[0138] The "means for providing a response result to a user" is a function that sends a response generated by a server to a terminal and allows the user to receive the response in text or voice.

[0139] "Means for managing the user's schedule information and sending reminder notifications" refers to a system that saves the schedule entered by the user and generates reminder notifications at pre-set times to notify the user.

[0140] "Means of collecting and analyzing user health data and notifying users in the event of abnormalities" refers to a function that stores the health information entered by users in a database, detects abnormalities using a machine learning algorithm, and notifies users of the results.

[0141] "Means for converting voice input into text data using voice recognition technology" refers to technology that converts data input by voice by a user into text data using voice recognition technology and transmits it to a server.

[0142] A "machine learning algorithm" is a mathematical model that analyzes large amounts of data to learn patterns and trends and make future predictions and decisions.

[0143] A "generative artificial intelligence model" refers to an algorithm that understands human natural language and generates appropriate responses based on that language, and is a fundamental technology for natural language processing.

[0144] A "database" is a storage system that centrally manages various data such as user information, schedule information, and health data, and allows efficient access.

[0145] The system of the present invention is designed to enable a user to efficiently perform tasks related to daily life and health management. Specific embodiments for carrying out the present invention will be described in detail below.

[0146] System Configuration

[0147] The system consists of three main components: users, devices, and servers. Users access the system using devices such as smartphones and tablets. The devices send input data from the users to the server, which processes the data, generates a response, and returns the result to the device.

[0148] Hardware and software used

[0149] Device: An input device such as a smartphone, tablet, or PC.

[0150] Server: Cloud server or on-premise server.

[0151] Database: A database (e.g., MySQL) for managing user information, schedule information, and health data.

[0152] Speech recognition technology: APIs for converting voice input into text (e.g., Google Speech-to-Text).

[0153] Text generation AI model: A generative artificial intelligence model (e.g., GPT-4).

[0154] Speech synthesis technology: APIs for converting text to speech (e.g., Google Text-to-Speech).

[0155] Machine learning algorithms: Algorithms for analyzing health data (e.g., TensorFlow).

[0156] Processing flow

[0157] 1. User authentication:

[0158] User: Enter your username and password on the login screen.

[0159] Terminal: Sends the entered authentication information to the server.

[0160] Server: Accesses the database and checks the authentication information. If authentication is successful, generates a session ID and authentication token and sends them to the device.

[0161] 2. Data entry and submission:

[0162] User: Enters data into the app via text or voice. For example, "Add a meeting for 3 PM."

[0163] Terminal: Converts speech into text and sends the data to the server.

[0164] 3. Response generation using generative artificial intelligence models:

[0165] Server: Sends user input data to the generative artificial intelligence model and generates a response, e.g., "A meeting has been scheduled for 3 PM."

[0166] 4. Response provision:

[0167] Server: Sends the generated response to the terminal.

[0168] Terminal: Provides a response to the user in text or voice.

[0169] 5. Schedule Management and Reminders:

[0170] User: Enters future events into the app. Example: "Set a reminder to take my medicine tomorrow at 10 AM."

[0171] Terminal: Sends input data to the server.

[0172] Server: Saves schedule data to a database and generates notifications at the reminder time.

[0173] Device: Receives notifications and notifies the user.

[0174] 6. Health data collection and anomaly detection:

[0175] User: Enter walking records and health data. Example: "Today's steps are 5000."

[0176] Terminal: Sends data to the server.

[0177] Server: Stores data in a database, analyzes it using machine learning algorithms, and sends notifications to configured contacts if an anomaly is detected.

[0178] Specific examples

[0179] Set schedule reminders:

[0180] User: Says, "Set a reminder to take my medicine tomorrow at 10 AM."

[0181] Device: Converts speech to text and sends it to the server.

[0182] Server: Stores reminders in a database and generates notifications at the scheduled times.

[0183] Device: Receive reminder notifications and notify the user.

[0184] Anomaly detection in health data:

[0185] User: Enters "5000 steps today."

[0186] Terminal: Sends data to the server.

[0187] Server: Analyzes the data and if an abnormality is detected, sends a notification to family members or medical institutions.

[0188] Device: Notify the user that an anomaly has been detected.

[0189] The system of the present invention allows users to efficiently manage their daily lives and health management on a single platform. In addition, the use of generative artificial intelligence models enables appropriate and prompt responses to user input.

[0190] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0191] Step 1: User authentication

[0192] ---

[0193] User:

[0194] Enter your username and password on the login screen. Input data: username (text), password (text).

[0195] Device:

[0196] Receives the entered authentication information and sends it to the server. Output data: Authentication information (JSON format).

[0197] server:

[0198] Accesses the database and checks the authentication information. If successful, generates a session ID and authentication token and sends them to the device. If unsuccessful, returns an error message.

[0199] Input data: Authentication information (JSON format)

[0200] Data calculation: User information matching and token generation

[0201] Output data: Session ID, authentication token (if successful) or error message (if unsuccessful)

[0202] Step 2: Enter and submit data

[0203] ---

[0204] User:

[0205] Enter data into the app by text or voice, for example, "Add a meeting for 3 PM."

[0206] Input data: text or audio data

[0207] Device:

[0208] For voice input, speech recognition technology is used to convert speech into text.

[0209] Input data: Audio data

[0210] Data Computing: Speech-to-Text Conversion

[0211] Output data: Text data

[0212] Device:

[0213] The converted text data or the directly entered text is sent to the server.

[0214] Output data: Input data (JSON format)

[0215] Step 3: Response generation using a generative artificial intelligence model

[0216] ---

[0217] server:

[0218] The input data is analyzed and applied to a generative artificial intelligence model. For example, the prompt sentence "Add a meeting at 3 PM" is input to the generative AI.

[0219] Input data: Text data (prompt)

[0220] Data Computation: Response Generation with Generative AI Models

[0221] Output data: Response text

[0222] server:

[0223] The generated response is temporarily stored and sent to the terminal.

[0224] Output data: Response data (JSON format)

[0225] Step 4: Provide a response

[0226] ---

[0227] Device:

[0228] The response data received from the server is displayed to the user. If audio is desired, a technology is used to convert text data into audio.

[0229] Input data: Response data (JSON format)

[0230] Data Computing: Text-to-Speech

[0231] Output data: Audio data

[0232] Device:

[0233] The response is displayed or played aloud to the user.

[0234] Output data: User presented

[0235] Step 5: Scheduling and Reminders

[0236] ---

[0237] User:

[0238] Enter future events into the app, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[0239] Input data: Schedule data (text)

[0240] Device:

[0241] Sends input data to the server.

[0242] Output data: Schedule data (JSON format)

[0243] server:

[0244] Save schedule data to a database and generate notifications at reminder times.

[0245] Input data: Schedule data

[0246] Data calculation: Schedule saving and reminder setting

[0247] Output data: Reminder notification data

[0248] Device:

[0249] Receive reminder notifications and notify users.

[0250] Input data: Reminder notification data

[0251] Output data: User presented

[0252] Step 6: Health data collection and anomaly detection

[0253] ---

[0254] User:

[0255] Enter your health data into the app. For example, enter "Today's steps are 5,000."

[0256] Input data: Health data (text)

[0257] Device:

[0258] Send the data to the server.

[0259] Output data: Health data (JSON format)

[0260] server:

[0261] Health data is stored in a database and analyzed using machine learning algorithms.

[0262] Input data: Health data

[0263] Data Computing: Data Analysis and Anomaly Detection

[0264] Output data: Analysis results (notification data in case of abnormality)

[0265] server:

[0266] If an abnormality is detected, a notification will be sent to the specified contacts (e.g., family members, medical institutions).

[0267] Output data: Error notification data

[0268] Device:

[0269] Notify the user that an abnormality has been detected and provide instructions on what action to take.

[0270] Input data: Error notification data

[0271] Output data: User presented

[0272] (Application example 1)

[0273] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] Modern factories require systems to manage worker health and improve the safety of the work environment. Conventional systems struggle to monitor worker health data and work environment information in real time, and respond immediately when an abnormality is detected. Furthermore, they do not effectively manage individual work schedules or send reminder notifications. This can lead to reduced work efficiency and increased safety risks for workers. A system is needed to resolve these issues and improve work efficiency and safety within factories.

[0275] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0276] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and sending a notification when an abnormality occurs, means for monitoring the worker's health data in real time and issuing a warning when an abnormality is detected, means for managing the work schedule and sending reminder notifications, and means for monitoring the temperature, hazardous gas concentration, etc. of the work environment and sending a notification when an abnormal value is detected. This makes it possible to monitor the worker's health condition in real time and take prompt action when an abnormality is detected, ensuring the safety of the work environment and improving work efficiency.

[0277] "Means for receiving data entered by a user" refers to means for receiving and processing text or voice data entered by a user using a device such as a smartphone or tablet.

[0278] "Means for authenticating users" refers to the means for authenticating that a user is a legitimate user using a user name and password.

[0279] "Means for generating responses to user input using generative artificial intelligence" means means for using a generative AI model (e.g., GPT-4) to generate appropriate responses based on input from a user.

[0280] The "means for providing a response result to a user" refers to a means for transmitting a response generated on a server to a user's terminal and presenting it in text or audio format.

[0281] "Means for managing user schedule information and sending reminder notifications" refers to means for saving and managing the plans and schedule information set by the user and sending reminder notifications at designated times.

[0282] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to the means for storing and analyzing health data entered by users on a server, and notifying users in the event of an abnormality.

[0283] "Means for monitoring workers' health data in real time and issuing warnings when abnormalities are detected" refers to means for collecting health data in real time from workers' smartwatches and healthcare devices and issuing warnings when abnormalities are detected.

[0284] The "means for managing work schedules and issuing reminder notifications" is a means for managing the work schedules of workers and issuing reminder notifications based on the set schedules.

[0285] "Means for monitoring the temperature, hazardous gas concentration, etc. of the working environment and issuing a notification if an abnormal value is detected" refers to a means for monitoring the temperature and hazardous gas concentration within a factory using sensors and issuing an immediate notification if an abnormal value is detected.

[0286] The system of this invention includes functions for receiving data entered by users, authenticating users, generating responses using artificial intelligence, providing response results, managing schedules and sending reminders, collecting and analyzing health data, and notifying users when an abnormality occurs. Furthermore, to adapt to a factory environment, the system adds functions for monitoring workers' health data in real time, issuing warnings when an abnormality is detected, and monitoring the work environment.

[0287] System Configuration

[0288] 1. User Device

[0289] Using a smartphone or tablet, the user inputs data by voice or text. The voice data is converted into text using voice recognition technology. The received data is sent to the server.

[0290] 2. Server

[0291] User authentication: The server accesses the database and performs authentication based on the username and password. If authentication is successful, a session is initiated and an authentication token is sent to the user's device. If authentication fails, an error message is returned.

[0292] Response generation using generative AI: The server uses a generative AI model (e.g., GPT-4) to generate a response based on user input. The generated response is temporarily stored on the server and then sent to the user's device.

[0293] Schedule management and reminder notifications: The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, it generates a corresponding notification and sends it to the user's device.

[0294] Health data collection and analysis: Health data is collected in real time from workers' smartwatches and healthcare devices, and analyzed using machine learning algorithms. If an abnormality is detected, notification will be sent to designated contacts.

[0295] Work environment monitoring: Sensors are used to monitor the temperature and hazardous gas concentrations in the work environment, and if abnormal values ​​are detected, workers are immediately notified.

[0296] Work Schedule Management: Manage the work schedules of workers and send reminder notifications based on the set schedule.

[0297] Hardware and software used

[0298] Hardware: smartphones, tablets, smartwatches, sensors

[0299] Software: speech recognition technology, generative AI models (e.g., GPT-4), machine learning algorithms

[0300] Specific examples

[0301] 1. Set schedule reminders

[0302] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[0303] Device: Converts speech to text and sends it to the server.

[0304] Server: Stores reminders in a database and generates notifications at the scheduled times.

[0305] On your device: Receive reminder notifications and notify the user.

[0306] 2. Detecting Anomalies in Health Data

[0307] User: Updates walking log and enters it into the app.

[0308] Terminal: Sends data to the server.

[0309] Server: Analyzes data, detects anomalies, and sends relevant notifications to family members and medical institutions.

[0310] Device: Notify the user that an anomaly has been detected.

[0311] Prompt Sentence Examples

[0312] Implement a program that collects health data for worker ID: 123456 and notifies if any abnormalities are found. The collected data is heart rate and number of steps, and a notification will be sent if the heart rate exceeds 100 or the number of steps is less than 500. A notification will also be sent if the temperature in the work environment exceeds 35 degrees or the concentration of harmful gases exceeds 50. The APIs to be used are as follows.

[0313] Health data collection: https: / / api.smartwatch.com / users / {user_id} / health

[0314] Speech recognition: https: / / api.speech-to-text.com / convert

[0315] Work schedule management: https: / / api.factory-robot.com / schedules

[0316] Environmental monitoring: https: / / api.factory-sensors.com / environment

[0317] Send notification: https: / / api.notification.com / send

[0318] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0319] Step 1:

[0320] Receiving Data Input

[0321] User: Opens the app on their smartphone and enters voice or text data. For example, they might say, "Set a reminder to take my medicine tomorrow at 10 AM."

[0322] Input: Voice or text data

[0323] Terminal: When voice data is input, it is converted into text using voice recognition technology (e.g., API: https: / / api.speech-to-text.com / convert). The converted text data is sent to the server.

[0324] Output: User input as text data

[0325] Step 2:

[0326] User authentication

[0327] Terminal: Provides an interface for entering a username and password, where the user enters their credentials.

[0328] Input: Username and Password

[0329] Device: Sends authentication information to the server.

[0330] Server: Accesses the database and verifies the entered authentication information. If authentication is successful, starts a session, generates an authentication token, and sends it to the device. If authentication fails, generates an error message.

[0331] Output: Authentication token or error message

[0332] Step 3:

[0333] Response Generation

[0334] Server: Based on user input, generates appropriate responses using a generative AI model (e.g., GPT-4).

[0335] Input: User-entered text data

[0336] Server: Temporarily stores the generated response and sends it to the user's device.

[0337] Output: The text data of the generated response

[0338] Step 4:

[0339] Providing a response

[0340] Device: Displays or speaks to the user the generated response received from the server, for example, "A reminder to take your medicine has been set for tomorrow at 10 AM."

[0341] Input: The text data of the generated response

[0342] Terminal: Displaying text data in a user interface or playing it aloud.

[0343] Output: Providing a visual or audio response

[0344] Step 5:

[0345] Schedule management and reminder notifications

[0346] User: Enters an appointment. For example, "Take medicine at 10:00 AM tomorrow."

[0347] Input: User schedule input

[0348] Terminal: Sends schedule data to the server.

[0349] Server: Stores schedule data in a database and sets reminder notification timing.

[0350] Output: Schedule data stored in the database and reminder times

[0351] Step 6:

[0352] Real-time monitoring of health data

[0353] User: Wears a smartwatch or healthcare device to collect health data.

[0354] Input: Health data collected from smartwatches and healthcare devices

[0355] Device: Sends health data to the server.

[0356] Server: Analyzes health data using machine learning algorithms and generates alerts and notifies configured contacts if anomalies are detected.

[0357] Output: Health data analysis results and warning notifications in case of abnormalities

[0358] Step 7:

[0359] Work environment monitoring

[0360] Server: Use sensors to collect environmental data such as temperature and harmful gas concentrations in the factory (e.g., API: https: / / api.factory-sensors.com / environment).

[0361] Input: Environmental data from sensors

[0362] Server: Analyzes environmental data and immediately notifies workers if any abnormal values ​​are detected.

[0363] Output: Environmental data analysis results and notifications in case of abnormalities

[0364] Step 8:

[0365] Work schedule management and reminder notifications

[0366] User: Sets the work schedule, for example, "Start the next process at 10 o'clock."

[0367] Input: User work schedule input

[0368] Server: Stores work schedule data in a database and sets reminder notification timing. When the reminder time arrives, it generates a corresponding notification and sends it to the device.

[0369] Output: Work schedule data and reminder notifications

[0370] Through the above processing steps, this system can realize safety management of workers at the factory site and improvement of work efficiency.

[0371] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0372] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, integrating functions such as user input, authentication, AI response, reminder notifications, health data analysis, emotion recognition, etc. The following describes in detail the function of each component and its processing flow.

[0373] Receiving data entered by the user

[0374] User

[0375] Users use their smartphone or tablet to launch the app and enter data via text or voice, including schedules, health data, daily communication, and emotional state.

[0376] Terminal

[0377] The device receives user input and transmits it to the server as text and voice data, and uses voice recognition technology to convert the voice data into text.

[0378] User authentication

[0379] User

[0380] When users start using the app, they enter their username and password.

[0381] Terminal

[0382] The terminal transmits these authentication data to the server.

[0383] server

[0384] The server accesses the database and performs authentication based on the user information. If authentication is successful, it generates a user ID and starts a session. It sends an authentication token to the terminal, and if authentication fails, it returns an error message.

[0385] Response generation by generative artificial intelligence

[0386] server

[0387] The server uses generative AI to analyze user input and generate appropriate responses, and an emotion engine to recognize the user's emotional state and reflect it in the response.

[0388] Terminal

[0389] The terminal receives the generated response from the server and provides it to the user in text and audio.

[0390] Emotion recognition by emotion engine

[0391] server

[0392] The emotion engine analyzes user input data (text and voice) and recognizes the user's emotional state, for example, identifying whether the user's input contains emotions such as joy, sadness, or anger.

[0393] server

[0394] Based on the recognized emotions, the generative AI adjusts its response and provides appropriate feedback to the user.

[0395] Schedule management and reminder notifications

[0396] User

[0397] Users input their future plans into the app.

[0398] Terminal

[0399] The entered schedule information is sent to the server.

[0400] server

[0401] The server stores the schedule information in a database and sets the timing of reminder notifications. When the time comes, the server generates the reminder content.

[0402] Terminal

[0403] The device will receive reminder notifications and notify the user via text and voice.

[0404] Health data collection and anomaly detection

[0405] User

[0406] Users enter their medication record and walking records into the app.

[0407] Terminal

[0408] The entered health data is sent to the server.

[0409] server

[0410] The server stores health data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, a notification is sent to the specified contacts (e.g., family members, medical institutions).

[0411] Terminal

[0412] Notify users when data is updated and alert them if anomalies are detected.

[0413] Specific examples

[0414] 1. Set schedule reminders

[0415] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[0416] Terminal: Sends input to the server.

[0417] Server: The generating AI understands the reminders and stores them in a database.

[0418] Device: Notify that a reminder has been set.

[0419] 2. Use of Emotion Recognition

[0420] User: In everyday conversation, type "I'm very happy today."

[0421] Terminal: Sends input to the server.

[0422] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[0423] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[0424] Terminal: Provides generated responses to the user in text and audio.

[0425] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[0426] The processing flow will be explained below.

[0427] Step 1:

[0428] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[0429] Step 2:

[0430] Terminal: Sends the entered username and password to the server.

[0431] Step 3:

[0432] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[0433] Step 4:

[0434] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[0435] Step 5:

[0436] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[0437] Step 6:

[0438] Terminal: Sends user input to the server.

[0439] Step 7:

[0440] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[0441] Step 8:

[0442] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[0443] Step 9:

[0444] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[0445] Step 10:

[0446] Server: Generates a notification based on the reminder content.

[0447] Step 11:

[0448] Device: Receive reminder notifications and notify users via text and voice.

[0449] Step 12:

[0450] User: Enter new data from your medication record or walking log into the app.

[0451] Step 13:

[0452] Terminal: Sends the entered health data to the server.

[0453] Step 14:

[0454] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[0455] Step 15:

[0456] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[0457] Step 16:

[0458] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[0459] Step 17:

[0460] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[0461] Processing flow incorporating emotion recognition

[0462] Step 18:

[0463] User: Type everyday conversations and questions into the app or speak into the microphone.

[0464] Step 19:

[0465] Terminal: Sends user input or voice data to the server.

[0466] Step 20:

[0467] Server: When voice input is received, the voice data is converted into text data using voice recognition technology.

[0468] Step 21:

[0469] Server: Uses an emotion engine to recognize emotions from the user's text or voice data. For example, the emotion engine identifies emotions such as "happy," "sad," and "angry."

[0470] Step 22:

[0471] Server: Based on the recognized emotion, the generative AI adjusts and generates an appropriate response. For example, if the user inputs "I'm happy," the generative AI will generate a response such as "That's great!"

[0472] Step 23:

[0473] Terminal: Receives the generated response and provides it to the user in text and voice.

[0474] Specific examples

[0475] 1. Set schedule reminders

[0476] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[0477] Terminal: Sends input to the server.

[0478] Server: The generating AI understands the reminders and stores them in a database.

[0479] Device: Notify that a reminder has been set.

[0480] 2. Use of Emotion Recognition

[0481] User: In everyday conversation, type "I'm very happy today."

[0482] Terminal: Sends input to the server.

[0483] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[0484] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[0485] Terminal: Provides generated responses to the user in text and audio.

[0486] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[0487] Example 2

[0488] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0489] In today's multifunctional digital lifestyle, systems that integrate multiple functions are required to support users' efficient lifestyle and health management. However, conventional systems often provide separate functions for user input data processing, authentication, generative AI responses, emotion recognition, schedule management, and health data analysis, making integrated management difficult. Furthermore, it is difficult to seamlessly integrate these functions, which can result in a poor user experience. Furthermore, systems are unable to properly recognize the user's emotional state and reflect it in responses, resulting in unnatural communication.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0491] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative AI model, means for recognizing the user's emotional state using an emotion engine and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, and means for collecting and analyzing the user's health data and notifying in the event of an abnormality. This enables efficient lifestyle and health management for the user and realizes natural communication through emotion recognition.

[0492] "Means for receiving data entered by a user" refers to a device or program that allows a user to enter data in text or voice format via a smartphone or tablet and receive it.

[0493] A "means for authenticating a user" is a device or program that verifies a user's identity and grants appropriate access rights based on authentication information such as a user name and password.

[0494] A "means for generating a response to a user's input using generative artificial intelligence" is a device or program that uses a generative AI model to create an appropriate response based on the user's input data.

[0495] "Means for recognizing a user's emotional state using an emotion engine and reflecting that in responses" refers to a device or program for analyzing a user's input data and identifying emotions, and a mechanism for adjusting responses based on those emotions.

[0496] The "means for providing a response result to a user" refers to a device or program for displaying or audibly providing the generated response to a user.

[0497] The "means for managing the user's schedule information and sending reminder notifications" refers to a device or program that records the schedule information entered by the user and sends reminder notifications at appropriate times.

[0498] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to a device or program that collects health-related data entered by users, analyzes it using a machine learning algorithm, and notifies users in the event of an abnormality.

[0499] A "machine learning algorithm" is an algorithm that uses mathematical and statistical techniques to predict future data patterns or classify data based on past data.

[0500] "Speech recognition technology" is a technology for analyzing voice data and converting it into corresponding text data.

[0501] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, and integrates functions such as user input, authentication, AI response, reminder notification, health data analysis, emotion recognition, etc. A specific embodiment of this system will be described.

[0502] Receiving data entered by the user

[0503] Users use a smartphone or tablet to launch the app and input data via text or voice. The input content includes schedules, health data, daily communication, and emotional state. The device receives this input data and sends it to the server as text and voice data. Voice recognition technology (e.g., a voice recognition API) is used to convert the voice data into text.

[0504] User authentication

[0505] When a user starts using an app, they enter their username and password. The device sends this authentication data to the server. The server accesses a database (for example, an SQL database) and performs authentication based on the user information. If authentication is successful, a token is generated and sent to the device. This allows the user to use the system securely.

[0506] Response generation by generative artificial intelligence

[0507] The server uses a generative AI model to analyze the user's input and generate an appropriate response. It uses an emotion engine to recognize the user's emotional state and reflect it in the response. In this case, the generative AI model uses natural language processing technology, for example. The server sends the generated response to the device, which then provides it to the user in text and voice.

[0508] Emotion recognition by emotion engine

[0509] The server uses an emotion engine to analyze the user's input data (text and voice) to recognize their emotional state. Specifically, it identifies emotions such as joy, sadness, and anger based on the input data. As a result, the generative AI model adjusts the response content based on the recognized emotion and provides feedback to the user.

[0510] Schedule management and reminder notifications

[0511] The user enters future plans into the app. The device sends the entered schedule information to the server. The server saves the schedule information in a database and sets the timing for reminder notifications. When the reminder time arrives, the server generates the reminder content. The device notifies the user of the reminder via text and voice.

[0512] Health data collection and anomaly detection

[0513] Users enter health data such as their medication record and walking records into the app. The device then sends this health data to the server. The server then analyzes the health data stored in the database and uses machine learning algorithms to detect abnormalities. If an abnormality is detected, a notification is sent to set contacts (e.g., family members, medical institutions) and the user is also notified via the device.

[0514] Specific examples

[0515] 1. Set schedule reminders

[0516] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[0517] Terminal: Sends input to the server.

[0518] Server: The generative AI model understands the reminders and stores them in a database.

[0519] Device: Notify that a reminder has been set.

[0520] 2. Use of Emotion Recognition

[0521] User: In everyday conversation, type "I'm very happy today."

[0522] Terminal: Sends input to the server.

[0523] Server: The generative AI model analyzes the input, and the emotion engine recognizes the emotion "happy."

[0524] Server: The generative AI model generates an appropriate response based on the user's emotions (e.g., "That's great!").

[0525] Terminal: Provides generated responses to the user in text and audio.

[0526] The system of the present invention streamlines users' daily lives and health management, and provides more natural and effective communication with users by recognizing emotions and responding appropriately. Furthermore, by utilizing natural language processing technology and machine learning algorithms, the system is capable of advanced data analysis and response generation, flexibly responding to diverse user needs.

[0527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0528] Program processing flow

[0529] Step 1: User enters data

[0530] A user launches the app using a smartphone or tablet, then enters data via text or voice, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[0531] Input: User text or voice data.

[0532] Output: Text data or audio file.

[0533] Specific operation: In the case of voice input, the voice data is converted into text using a speech recognition API. The text data generated is "Set a reminder to take my medicine tomorrow at 10 AM."

[0534] Step 2: User authentication

[0535] The user enters their username and password into the app's login screen.

[0536] The terminal transmits these authentication data to the server.

[0537] The server compares the received authentication data with a database and authenticates the user.

[0538] Input: Username and Password.

[0539] Output: An authentication token or an error message.

[0540] Specific operation: The server accesses a database such as MySQL and verifies whether the username "user123" and password "password123" are correct. If authentication is successful, it generates a JWT token and sends it to the device. If authentication fails, it returns an error message.

[0541] Step 3: Generative AI generates a response

[0542] The server receives the user's text input and sends prompt sentences to a generative AI model (e.g., a natural language processing model) for analysis.

[0543] Input: User input data (e.g., "I'm very happy today").

[0544] Output: The generated response text (e.g., "That's great!").

[0545] Specific operation: Send a prompt to the generative AI model saying, "The user feels happy. Please generate an appropriate response." Receive the generated text response.

[0546] Step 4: Emotion Recognition with the Emotion Engine

[0547] The server sends the user's input data to an emotion engine (e.g., an emotion analysis API) to identify the emotional state.

[0548] Input: User input data (e.g., "I'm very happy today").

[0549] Output: Emotional state (e.g., "happy").

[0550] Specific operation: The server uses the emotion analysis API to recognize the emotion from the user's input. If the analysis result is "happy," that emotion is reflected in the generation AI's response.

[0551] Step 5: Providing response results

[0552] The server sends the generated response to the terminal.

[0553] The device provides the received response to the user, either displaying it as text or converting it to speech and reading it aloud.

[0554] Input: Response text from the generation AI.

[0555] Output: The response (text or audio) provided to the user.

[0556] Specific behavior: The device receives the response "That's great!" and displays it as text or reads it aloud using the speech synthesis API.

[0557] Step 6: Scheduling and Reminders

[0558] Users input their future plans into the app.

[0559] The terminal transmits this schedule information to the server.

[0560] The server stores the schedule information in a database and sets the timing of reminder notifications.

[0561] Input: Schedule information (e.g., "Take your medicine tomorrow at 10 AM").

[0562] Output: Remind notification.

[0563] Specific operation: The server saves the schedule information in a database and manages the timing of reminders using a Cron job. When the reminder time arrives, it sends a notification to the device, informing the user that it is time to take their medicine.

[0564] Step 7: Health data collection and anomaly detection

[0565] Users enter their medication records and walking records into the app.

[0566] The terminal transmits this health data to the server.

[0567] The server stores the health data in a database and analyzes it using machine learning algorithms.

[0568] Input: Health data (e.g. walking records, blood pressure values).

[0569] Output: Anomaly detection notification.

[0570] How it works: The server analyzes health data using machine learning algorithms such as Scikit-learn. If an abnormality is detected, it sends a notification to the specified contacts (e.g., family members, medical institutions), and notifies the user via their device that an abnormality has been detected.

[0571] keyword

[0572] Generative AI model, prompt sentence

[0573] (Application example 2)

[0574] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0575] Conventional systems mainly provide simple responses and notifications to user inputs, but lack the ability to respond or suggest products that take into account the user's emotional and health states. Therefore, there is a need for more personalized and effective systems that contribute to users' lifestyles and health management. Furthermore, to increase customer satisfaction in virtual stores, a personalized shopping assistant that reflects the customer's current emotional and health states is necessary.

[0576] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0577] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for recognizing the user's emotional state and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying the user in the event of an abnormality, and means for suggesting products based on the user's health data and emotional state. This allows the user to receive personalized responses and product suggestions that take into account their emotional state and health condition at any given time, significantly improving the shopping experience in virtual stores.

[0578] "Means for receiving data" is a function for receiving text or voice data entered by a user and sending it to a server.

[0579] "Means for authenticating users" refers to a function that verifies the authentication information (user name and password) entered by the user and confirms the user's legitimacy.

[0580] "Generative artificial intelligence" is a technique that allows computers to generate appropriate responses to specific inputs, often using machine learning or deep learning models (e.g., GPT-3).

[0581] "Means for recognizing emotional states and reflecting them in responses" refers to technology that analyzes the user's input, recognizes the emotions (joy, sadness, anger, etc.) at the time, and reflects that emotional state in responses.

[0582] The "means for providing the response result to the user" is a function for conveying the generated response to the user in text or voice.

[0583] "Means for managing schedule information and sending reminder notifications" is a function that manages the schedules set by the user and sends notifications when the scheduled time approaches.

[0584] "Means for collecting and analyzing health data and notifying in the event of an abnormality" refers to a function that collects and analyzes health data entered by the user and notifies in the event of an abnormality.

[0585] The "means for suggesting products" is a function for suggesting optimal products based on the user's health data and emotional state.

[0586] To implement this invention, the following system must be constructed. First, the device used by the user is assumed to be a smartphone, smart glasses, or a head-mounted display (HMD). The user inputs data through this device, and the data is sent to a server.

[0587] Data Receipt and Authentication

[0588] The user launches the app using a smartphone or HMD and inputs data via voice or text. The user is authenticated by entering a username and password. The authentication information is sent to the server and authenticated. If authentication is successful, the user can use the system.

[0589] User Input and Response Generation

[0590] The server uses a generative AI model (e.g., GPT-3) to analyze the user's input data and generate an appropriate response. An emotion recognition engine is used to analyze the emotional state contained in the user's input and reflect it in the response. For example, if a user inputs "I'm feeling great today," the generative AI model will recognize that emotion and generate a response such as "That's great!"

[0591] Schedule management and reminder notifications

[0592] Users enter their schedule information into the app, and the server stores that information in a database. When it's time for a reminder, the server generates the reminder content and sends it to the user's device. For example, if you enter "Set a reminder to take my medicine tomorrow at 10 a.m.", a reminder will be sent to the user at the specified time.

[0593] Health data analysis and product proposals

[0594] The server collects the user's health data and analyzes it using machine learning algorithms. If an abnormality is detected, the server notifies the user and their designated contacts. The server also makes personalized product suggestions based on the user's health and emotional state. For example, if a user inputs "I feel a little tired today," the server will suggest "relaxation products" based on the user's low level of exercise.

[0595] Hardware and Software

[0596] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[0597] Software: Generative AI models (e.g., GPT-3), emotion recognition engines, health data analysis algorithms, product recommendation engines

[0598] Examples and prompts

[0599] For example, the following process occurs:

[0600] 1. The user puts on the HMD and speaks, "What products do you have available today?"

[0601] 2. The emotion recognition engine recognizes the emotion "happy."

[0602] 3. The generative AI generates a response: "We're having a special sale today!"

[0603] 4. The product suggestion engine suggests a "set of fresh vegetables."

[0604] 5. The HMD presents the generated responses and suggestions to the user visually and audibly.

[0605] An example prompt is:

[0606] User sentiment: Happy

[0607] Health data: Blood pressure: normal, Heart rate: normal, Exercise: low

[0608] User Input: What products do you have available today?

[0609] Response: We have a special sale today!

[0610] The system allows users to receive a personalized shopping experience that takes into account their emotional and health state at any given time.

[0611] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0612] Step 1:

[0613] The user starts up their smartphone or HMD and inputs input data (voice or text). For example, the user might say, "What products do you have available today?" The input data is converted into text data using voice recognition technology.

[0614] Input: User voice or text input

[0615] Data processing: Converting voice data into text using voice recognition technology

[0616] Output: Text data

[0617] Step 2:

[0618] The device sends the converted text data to the server, which authenticates the user if necessary. If authentication is successful, the server sends the input data to the generative AI model.

[0619] Input: Text data (user input)

[0620] Data Processing: User Credential Verification

[0621] Output: Authenticated text data, user ID

[0622] Step 3:

[0623] The server uses a generative AI model to analyze the text data and generate appropriate responses. It also uses an emotion recognition engine to recognize emotional states (e.g., happy, sad) from the input data and reflect them in the response.

[0624] Input: authenticated text data, user ID

[0625] Data processing: Response generation using a generative AI model, emotional state analysis using an emotion recognition engine

[0626] Output: The generated response text

[0627] Step 4:

[0628] The server stores the user's schedule information in a database and sets appropriate reminder times. For example, if a user inputs "Set a reminder to take my medicine at 10:00 AM tomorrow," the server generates the reminder content and sends it to the device when the time comes.

[0629] Input: Schedule information (e.g., time to take medicine)

[0630] Data processing: setting the timing of reminder notifications, generating reminder notification content

[0631] Output: Reminder notification

[0632] Step 5:

[0633] The server collects the user's health data and analyzes it using machine learning algorithms. For example, if the user's step count or blood pressure data is collected, the server analyzes the data to determine whether it is normal or not. If an abnormality is detected, the server sends a notification to the specified contacts (e.g., medical institutions).

[0634] Input: Health data (e.g., number of steps, blood pressure)

[0635] Data processing: Data analysis using machine learning algorithms

[0636] Output: Abnormality detection notification

[0637] Step 6:

[0638] The server then makes personalized product recommendations based on the user's emotional state and health data. For example, if a user inputs "I feel a little tired today" and health data indicates that they are not exercising much, the server will suggest "relaxation products."

[0639] Input: Emotional state, health data

[0640] Data processing: Analysis by product recommendation engine based on emotional state and health data

[0641] Output: Personalized product recommendations

[0642] Step 7:

[0643] The device provides the user with the responses, notifications, and product suggestions sent from the server visually and audibly, for example, by displaying the responses and product suggestions generated through the HMD and providing audio notifications.

[0644] Input: Generated response text, reminders, product suggestions

[0645] Data processing: Converting responses and notifications into visual and audio formats

[0646] Output: Visual and audio notification to the user

[0647] In this way, users can receive a shopping experience that is personalized to their emotional and health state at the time.

[0648] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0649] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0650] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0651] [Second embodiment]

[0652] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0653] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0654] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0655] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0656] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0657] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0658] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0659] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0660] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0661] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0662] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0663] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0664] The system of the present invention is constructed to allow users to perform various tasks related to daily life and health management. The functions of each component and the processing flow will be specifically explained below.

[0665] Receiving data entered by the user

[0666] User

[0667] Users use devices such as smartphones and tablets to input data into the app by text or voice, including schedules, health data, and everyday conversations.

[0668] Terminal

[0669] The device receives user input and sends it to the server as text data. If voice data is input, it is converted into text using voice recognition technology.

[0670] User authentication

[0671] User

[0672] When users start using the app, they enter their username and password.

[0673] Terminal

[0674] The terminal transmits these authentication data to the server.

[0675] server

[0676] The server accesses the database and performs authentication based on the user information. If authentication is successful, a session is started and an authentication token is sent to the terminal. If authentication fails, an error message is returned.

[0677] Response generation by generative artificial intelligence

[0678] server

[0679] The server uses a generative AI (e.g., GPT-4) to generate a response based on the user's input. The generated response is temporarily stored on the server.

[0680] Terminal

[0681] The terminal receives the generated response from the server and provides it to the user in text and audio.

[0682] Schedule management and reminder notifications

[0683] User

[0684] Users input their future plans into the app.

[0685] Terminal

[0686] This input is sent to the server.

[0687] server

[0688] The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, the server generates the corresponding notification.

[0689] Terminal

[0690] The device receives the reminder notification and notifies the user, for example, "A reminder to take your medicine has been set for 10:00 AM."

[0691] Health data collection and anomaly detection

[0692] User

[0693] Users enter their medication record and walking records into the app.

[0694] Terminal

[0695] The entered health data is sent to the server.

[0696] server

[0697] The server stores this data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, it notifies the specified contacts (e.g., family members, medical institutions).

[0698] Terminal

[0699] Reports detected anomalies to users and provides instructions on what to do.

[0700] Specific examples

[0701] 1. Set schedule reminders

[0702] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[0703] Device: Converts speech to text and sends it to the server.

[0704] Server: Stores reminders in a database and generates notifications at the scheduled times.

[0705] Device: Receive reminder notifications and notify the user.

[0706] 2. Detecting Anomalies in Health Data

[0707] User: Updates walking record and enters it into the app.

[0708] Terminal: Sends data to the server.

[0709] Server: Analyzes data, detects abnormalities, and sends relevant notifications to family members and medical institutions.

[0710] Device: Notify the user that an anomaly has been detected.

[0711] The system of the present invention makes it possible to streamline the user's lifestyle and health management and provide comprehensive daily support.

[0712] The processing flow will be explained below.

[0713] Step 1:

[0714] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[0715] Step 2:

[0716] Terminal: Sends the entered username and password to the server.

[0717] Step 3:

[0718] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[0719] Step 4:

[0720] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[0721] Step 5:

[0722] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[0723] Step 6:

[0724] Terminal: Sends user input to the server.

[0725] Step 7:

[0726] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[0727] Step 8:

[0728] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[0729] Step 9:

[0730] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[0731] Step 10:

[0732] Server: Generates a notification based on the reminder content.

[0733] Step 11:

[0734] Device: Receive reminder notifications and notify users via text and voice.

[0735] Step 12:

[0736] User: Enter new data from your medication record or walking log into the app.

[0737] Step 13:

[0738] Terminal: Sends the entered health data to the server.

[0739] Step 14:

[0740] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[0741] Step 15:

[0742] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[0743] Step 16:

[0744] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[0745] Step 17:

[0746] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[0747] Step 18:

[0748] User: Type everyday conversations and questions into the app or speak into the microphone.

[0749] Step 19:

[0750] Terminal: Sends user input or voice data to the server.

[0751] Step 20:

[0752] Server: When voice input is received, it converts it into text using speech recognition technology, and uses generative AI to generate appropriate responses to user questions and conversations.

[0753] Step 21:

[0754] Terminal: Receives the generated response from the server and provides it to the user in text and voice.

[0755] Example 1

[0756] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0757] In modern society, it is important for individual users to efficiently manage their daily lives and health, and there is a growing demand for systems that support this. However, conventional systems do not integrate various functions such as receiving user data, authenticating users, generating responses, managing schedules, and analyzing health data, which makes them difficult to use and makes it difficult to manage efficiently. Furthermore, there are issues with the recognition accuracy and response speed when using voice input.

[0758] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0759] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative artificial intelligence model, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying in the event of an abnormality, and means for converting voice input into text data using voice recognition technology. This enables the user to efficiently manage their daily life and health using a single system, and also improves the recognition accuracy and response speed for voice input.

[0760] The "means for receiving data entered by the user" is a function that receives information entered by the user via text or voice to the server via the terminal.

[0761] A "means of authenticating a user" is the process of identifying a user using authentication information, such as a username and password, and initiating a session on a server.

[0762] "Means for generating a response to user input using a generative artificial intelligence model" refers to a technology that analyzes input data from a user and generates an appropriate response using an artificial intelligence model.

[0763] The "means for providing a response result to a user" is a function that sends a response generated by a server to a terminal and allows the user to receive the response in text or voice.

[0764] "Means for managing the user's schedule information and sending reminder notifications" refers to a system that saves the schedule entered by the user and generates reminder notifications at pre-set times to notify the user.

[0765] "Means of collecting and analyzing user health data and notifying users in the event of abnormalities" refers to a function that stores the health information entered by users in a database, detects abnormalities using a machine learning algorithm, and notifies users of the results.

[0766] "Means for converting voice input into text data using voice recognition technology" refers to technology that converts data input by voice by a user into text data using voice recognition technology and transmits it to a server.

[0767] A "machine learning algorithm" is a mathematical model that analyzes large amounts of data to learn patterns and trends and make future predictions and decisions.

[0768] A "generative artificial intelligence model" refers to an algorithm that understands human natural language and generates appropriate responses based on that language, and is a fundamental technology for natural language processing.

[0769] A "database" is a storage system that centrally manages various data such as user information, schedule information, and health data, and allows efficient access.

[0770] The system of the present invention is designed to enable a user to efficiently perform tasks related to daily life and health management. Specific embodiments for carrying out the present invention will be described in detail below.

[0771] System Configuration

[0772] The system consists of three main components: users, devices, and servers. Users access the system using devices such as smartphones and tablets. The devices send input data from the users to the server, which processes the data, generates a response, and returns the result to the device.

[0773] Hardware and software used

[0774] Device: An input device such as a smartphone, tablet, or PC.

[0775] Server: Cloud server or on-premise server.

[0776] Database: A database (e.g., MySQL) for managing user information, schedule information, and health data.

[0777] Speech recognition technology: APIs for converting voice input into text (e.g., Google Speech-to-Text).

[0778] Text generation AI model: A generative artificial intelligence model (e.g., GPT-4).

[0779] Speech synthesis technology: APIs for converting text to speech (e.g., Google Text-to-Speech).

[0780] Machine learning algorithms: Algorithms for analyzing health data (e.g., TensorFlow).

[0781] Processing flow

[0782] 1. User authentication:

[0783] User: Enter your username and password on the login screen.

[0784] Terminal: Sends the entered authentication information to the server.

[0785] Server: Accesses the database and checks the authentication information. If authentication is successful, generates a session ID and authentication token and sends them to the device.

[0786] 2. Data entry and submission:

[0787] User: Enters data into the app via text or voice. For example, "Add a meeting for 3 PM."

[0788] Terminal: Converts speech into text and sends the data to the server.

[0789] 3. Response generation using generative artificial intelligence models:

[0790] Server: Sends user input data to the generative artificial intelligence model and generates a response, e.g., "A meeting has been scheduled for 3 PM."

[0791] 4. Response provision:

[0792] Server: Sends the generated response to the terminal.

[0793] Terminal: Provides a response to the user in text or voice.

[0794] 5. Schedule Management and Reminders:

[0795] User: Enters future events into the app. Example: "Set a reminder to take my medicine tomorrow at 10 AM."

[0796] Terminal: Sends input data to the server.

[0797] Server: Saves schedule data to a database and generates notifications at the reminder time.

[0798] Device: Receives notifications and notifies the user.

[0799] 6. Health data collection and anomaly detection:

[0800] User: Enter walking records and health data. Example: "Today's steps are 5000."

[0801] Terminal: Sends data to the server.

[0802] Server: Stores data in a database, analyzes it using machine learning algorithms, and sends notifications to configured contacts if an anomaly is detected.

[0803] Specific examples

[0804] Set schedule reminders:

[0805] User: Says, "Set a reminder to take my medicine tomorrow at 10 AM."

[0806] Device: Converts speech to text and sends it to the server.

[0807] Server: Stores reminders in a database and generates notifications at the scheduled times.

[0808] Device: Receive reminder notifications and notify the user.

[0809] Anomaly detection in health data:

[0810] User: Enters "5000 steps today."

[0811] Terminal: Sends data to the server.

[0812] Server: Analyzes the data and if an abnormality is detected, sends a notification to family members or medical institutions.

[0813] Device: Notify the user that an anomaly has been detected.

[0814] The system of the present invention allows users to efficiently manage their daily lives and health management on a single platform. In addition, the use of generative artificial intelligence models enables appropriate and prompt responses to user input.

[0815] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0816] Step 1: User authentication

[0817] ---

[0818] User:

[0819] Enter your username and password on the login screen. Input data: username (text), password (text).

[0820] Device:

[0821] Receives the entered authentication information and sends it to the server. Output data: Authentication information (JSON format).

[0822] server:

[0823] Accesses the database and checks the authentication information. If successful, generates a session ID and authentication token and sends them to the device. If unsuccessful, returns an error message.

[0824] Input data: Authentication information (JSON format)

[0825] Data calculation: User information matching and token generation

[0826] Output data: Session ID, authentication token (if successful) or error message (if unsuccessful)

[0827] Step 2: Enter and submit data

[0828] ---

[0829] User:

[0830] Enter data into the app by text or voice, for example, "Add a meeting for 3 PM."

[0831] Input data: text or audio data

[0832] Device:

[0833] For voice input, speech recognition technology is used to convert speech into text.

[0834] Input data: Audio data

[0835] Data Computing: Speech-to-Text Conversion

[0836] Output data: Text data

[0837] Device:

[0838] The converted text data or the directly entered text is sent to the server.

[0839] Output data: Input data (JSON format)

[0840] Step 3: Response generation using a generative artificial intelligence model

[0841] ---

[0842] server:

[0843] The input data is analyzed and applied to a generative artificial intelligence model. For example, the prompt sentence "Add a meeting at 3 PM" is input to the generative AI.

[0844] Input data: Text data (prompt)

[0845] Data Computation: Response Generation with Generative AI Models

[0846] Output data: Response text

[0847] server:

[0848] The generated response is temporarily stored and sent to the terminal.

[0849] Output data: Response data (JSON format)

[0850] Step 4: Provide a response

[0851] ---

[0852] Device:

[0853] The response data received from the server is displayed to the user. If audio is desired, a technology is used to convert text data into audio.

[0854] Input data: Response data (JSON format)

[0855] Data Computing: Text-to-Speech

[0856] Output data: Audio data

[0857] Device:

[0858] The response is displayed or played aloud to the user.

[0859] Output data: User presented

[0860] Step 5: Scheduling and Reminders

[0861] ---

[0862] User:

[0863] Enter future events into the app, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[0864] Input data: Schedule data (text)

[0865] Device:

[0866] Sends input data to the server.

[0867] Output data: Schedule data (JSON format)

[0868] server:

[0869] Save schedule data to a database and generate notifications at reminder times.

[0870] Input data: Schedule data

[0871] Data calculation: Schedule saving and reminder setting

[0872] Output data: Reminder notification data

[0873] Device:

[0874] Receive reminder notifications and notify users.

[0875] Input data: Reminder notification data

[0876] Output data: User presented

[0877] Step 6: Health data collection and anomaly detection

[0878] ---

[0879] User:

[0880] Enter your health data into the app. For example, enter "Today's steps are 5,000."

[0881] Input data: Health data (text)

[0882] Device:

[0883] Send the data to the server.

[0884] Output data: Health data (JSON format)

[0885] server:

[0886] Health data is stored in a database and analyzed using machine learning algorithms.

[0887] Input data: Health data

[0888] Data Computing: Data Analysis and Anomaly Detection

[0889] Output data: Analysis results (notification data in case of abnormality)

[0890] server:

[0891] If an abnormality is detected, a notification will be sent to the specified contacts (e.g., family members, medical institutions).

[0892] Output data: Error notification data

[0893] Device:

[0894] Notify the user that an abnormality has been detected and provide instructions on what action to take.

[0895] Input data: Error notification data

[0896] Output data: User presented

[0897] (Application example 1)

[0898] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0899] Modern factories require systems to manage worker health and improve the safety of the work environment. Conventional systems struggle to monitor worker health data and work environment information in real time, and respond immediately when an abnormality is detected. Furthermore, they do not effectively manage individual work schedules or send reminder notifications. This can lead to reduced work efficiency and increased safety risks for workers. A system is needed to resolve these issues and improve work efficiency and safety within factories.

[0900] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0901] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and sending a notification when an abnormality occurs, means for monitoring the worker's health data in real time and issuing a warning when an abnormality is detected, means for managing the work schedule and sending reminder notifications, and means for monitoring the temperature, hazardous gas concentration, etc. of the work environment and sending a notification when an abnormal value is detected. This makes it possible to monitor the worker's health condition in real time and take prompt action when an abnormality is detected, ensuring the safety of the work environment and improving work efficiency.

[0902] "Means for receiving data entered by a user" refers to means for receiving and processing text or voice data entered by a user using a device such as a smartphone or tablet.

[0903] "Means for authenticating users" refers to the means for authenticating that a user is a legitimate user using a user name and password.

[0904] "Means for generating responses to user input using generative artificial intelligence" means means for using a generative AI model (e.g., GPT-4) to generate appropriate responses based on input from a user.

[0905] The "means for providing a response result to a user" refers to a means for transmitting a response generated on a server to a user's terminal and presenting it in text or audio format.

[0906] "Means for managing user schedule information and sending reminder notifications" refers to means for saving and managing the plans and schedule information set by the user and sending reminder notifications at designated times.

[0907] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to the means for storing and analyzing health data entered by users on a server, and notifying users in the event of an abnormality.

[0908] "Means for monitoring workers' health data in real time and issuing warnings when abnormalities are detected" refers to means for collecting health data in real time from workers' smartwatches and healthcare devices and issuing warnings when abnormalities are detected.

[0909] The "means for managing work schedules and issuing reminder notifications" is a means for managing the work schedules of workers and issuing reminder notifications based on the set schedules.

[0910] "Means for monitoring the temperature, hazardous gas concentration, etc. of the working environment and issuing a notification if an abnormal value is detected" refers to a means for monitoring the temperature and hazardous gas concentration within a factory using sensors and issuing an immediate notification if an abnormal value is detected.

[0911] The system of this invention includes functions for receiving data entered by users, authenticating users, generating responses using artificial intelligence, providing response results, managing schedules and sending reminders, collecting and analyzing health data, and notifying users when an abnormality occurs. Furthermore, to adapt to a factory environment, the system adds functions for monitoring workers' health data in real time, issuing warnings when an abnormality is detected, and monitoring the work environment.

[0912] System Configuration

[0913] 1. User Device

[0914] Using a smartphone or tablet, the user inputs data by voice or text. The voice data is converted into text using voice recognition technology. The received data is sent to the server.

[0915] 2. Server

[0916] User authentication: The server accesses the database and performs authentication based on the username and password. If authentication is successful, a session is initiated and an authentication token is sent to the user's device. If authentication fails, an error message is returned.

[0917] Response generation using generative AI: The server uses a generative AI model (e.g., GPT-4) to generate a response based on user input. The generated response is temporarily stored on the server and then sent to the user's device.

[0918] Schedule management and reminder notifications: The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, it generates a corresponding notification and sends it to the user's device.

[0919] Health data collection and analysis: Health data is collected in real time from workers' smartwatches and healthcare devices, and analyzed using machine learning algorithms. If an abnormality is detected, notification will be sent to designated contacts.

[0920] Work environment monitoring: Sensors are used to monitor the temperature and hazardous gas concentrations in the work environment, and if abnormal values ​​are detected, workers are immediately notified.

[0921] Work Schedule Management: Manage the work schedules of workers and send reminder notifications based on the set schedule.

[0922] Hardware and software used

[0923] Hardware: smartphones, tablets, smartwatches, sensors

[0924] Software: speech recognition technology, generative AI models (e.g., GPT-4), machine learning algorithms

[0925] Specific examples

[0926] 1. Set schedule reminders

[0927] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[0928] Device: Converts speech to text and sends it to the server.

[0929] Server: Stores reminders in a database and generates notifications at the scheduled times.

[0930] On your device: Receive reminder notifications and notify the user.

[0931] 2. Detecting Anomalies in Health Data

[0932] User: Updates walking log and enters it into the app.

[0933] Terminal: Sends data to the server.

[0934] Server: Analyzes data, detects anomalies, and sends relevant notifications to family members and medical institutions.

[0935] Device: Notify the user that an anomaly has been detected.

[0936] Prompt Sentence Examples

[0937] Implement a program that collects health data for worker ID: 123456 and notifies if any abnormalities are found. The collected data is heart rate and number of steps, and a notification will be sent if the heart rate exceeds 100 or the number of steps is less than 500. A notification will also be sent if the temperature in the work environment exceeds 35 degrees or the concentration of harmful gases exceeds 50. The APIs to be used are as follows.

[0938] Health data collection: https: / / api.smartwatch.com / users / {user_id} / health

[0939] Speech recognition: https: / / api.speech-to-text.com / convert

[0940] Work schedule management: https: / / api.factory-robot.com / schedules

[0941] Environmental monitoring: https: / / api.factory-sensors.com / environment

[0942] Send notification: https: / / api.notification.com / send

[0943] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0944] Step 1:

[0945] Receiving Data Input

[0946] User: Opens the app on their smartphone and enters voice or text data. For example, they might say, "Set a reminder to take my medicine tomorrow at 10 AM."

[0947] Input: Voice or text data

[0948] Terminal: When voice data is input, it is converted into text using voice recognition technology (e.g., API: https: / / api.speech-to-text.com / convert). The converted text data is sent to the server.

[0949] Output: User input as text data

[0950] Step 2:

[0951] User authentication

[0952] Terminal: Provides an interface for entering a username and password, where the user enters their credentials.

[0953] Input: Username and Password

[0954] Device: Sends authentication information to the server.

[0955] Server: Accesses the database and verifies the entered authentication information. If authentication is successful, starts a session, generates an authentication token, and sends it to the device. If authentication fails, generates an error message.

[0956] Output: Authentication token or error message

[0957] Step 3:

[0958] Response Generation

[0959] Server: Based on user input, generates appropriate responses using a generative AI model (e.g., GPT-4).

[0960] Input: User-entered text data

[0961] Server: Temporarily stores the generated response and sends it to the user's device.

[0962] Output: The text data of the generated response

[0963] Step 4:

[0964] Providing a response

[0965] Device: Displays or speaks to the user the generated response received from the server, for example, "A reminder to take your medicine has been set for tomorrow at 10 AM."

[0966] Input: The text data of the generated response

[0967] Terminal: Displaying text data in a user interface or playing it aloud.

[0968] Output: Providing a visual or audio response

[0969] Step 5:

[0970] Schedule management and reminder notifications

[0971] User: Enters an appointment. For example, "Take medicine at 10:00 AM tomorrow."

[0972] Input: User schedule input

[0973] Terminal: Sends schedule data to the server.

[0974] Server: Stores schedule data in a database and sets reminder notification timing.

[0975] Output: Schedule data stored in the database and reminder times

[0976] Step 6:

[0977] Real-time monitoring of health data

[0978] User: Wears a smartwatch or healthcare device to collect health data.

[0979] Input: Health data collected from smartwatches and healthcare devices

[0980] Device: Sends health data to the server.

[0981] Server: Analyzes health data using machine learning algorithms and generates alerts and notifies configured contacts if anomalies are detected.

[0982] Output: Health data analysis results and warning notifications in case of abnormalities

[0983] Step 7:

[0984] Work environment monitoring

[0985] Server: Use sensors to collect environmental data such as temperature and harmful gas concentrations in the factory (e.g., API: https: / / api.factory-sensors.com / environment).

[0986] Input: Environmental data from sensors

[0987] Server: Analyzes environmental data and immediately notifies workers if any abnormal values ​​are detected.

[0988] Output: Environmental data analysis results and notifications in case of abnormalities

[0989] Step 8:

[0990] Work schedule management and reminder notifications

[0991] User: Sets the work schedule, for example, "Start the next process at 10 o'clock."

[0992] Input: User work schedule input

[0993] Server: Stores work schedule data in a database and sets reminder notification timing. When the reminder time arrives, it generates a corresponding notification and sends it to the device.

[0994] Output: Work schedule data and reminder notifications

[0995] Through the above processing steps, this system can realize safety management of workers at the factory site and improvement of work efficiency.

[0996] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0997] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, integrating functions such as user input, authentication, AI response, reminder notifications, health data analysis, emotion recognition, etc. The following describes in detail the function of each component and its processing flow.

[0998] Receiving data entered by the user

[0999] User

[1000] Users use their smartphone or tablet to launch the app and enter data via text or voice, including schedules, health data, daily communication, and emotional state.

[1001] Terminal

[1002] The device receives user input and transmits it to the server as text and voice data, and uses voice recognition technology to convert the voice data into text.

[1003] User authentication

[1004] User

[1005] When users start using the app, they enter their username and password.

[1006] Terminal

[1007] The terminal transmits these authentication data to the server.

[1008] server

[1009] The server accesses the database and performs authentication based on the user information. If authentication is successful, it generates a user ID and starts a session. It sends an authentication token to the terminal, and if authentication fails, it returns an error message.

[1010] Response generation by generative artificial intelligence

[1011] server

[1012] The server uses generative AI to analyze user input and generate appropriate responses, and an emotion engine to recognize the user's emotional state and reflect it in the response.

[1013] Terminal

[1014] The terminal receives the generated response from the server and provides it to the user in text and audio.

[1015] Emotion recognition by emotion engine

[1016] server

[1017] The emotion engine analyzes user input data (text and voice) and recognizes the user's emotional state, for example, identifying whether the user's input contains emotions such as joy, sadness, or anger.

[1018] server

[1019] Based on the recognized emotions, the generative AI adjusts its response and provides appropriate feedback to the user.

[1020] Schedule management and reminder notifications

[1021] User

[1022] Users input their future plans into the app.

[1023] Terminal

[1024] The entered schedule information is sent to the server.

[1025] server

[1026] The server stores the schedule information in a database and sets the timing of reminder notifications. When the time comes, the server generates the reminder content.

[1027] Terminal

[1028] The device will receive reminder notifications and notify the user via text and voice.

[1029] Health data collection and anomaly detection

[1030] User

[1031] Users enter their medication record and walking records into the app.

[1032] Terminal

[1033] The entered health data is sent to the server.

[1034] server

[1035] The server stores health data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, a notification is sent to the specified contacts (e.g., family members, medical institutions).

[1036] Terminal

[1037] Notify users when data is updated and alert them if anomalies are detected.

[1038] Specific examples

[1039] 1. Set schedule reminders

[1040] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1041] Terminal: Sends input to the server.

[1042] Server: The generating AI understands the reminders and stores them in a database.

[1043] Device: Notify that a reminder has been set.

[1044] 2. Use of Emotion Recognition

[1045] User: In everyday conversation, type "I'm very happy today."

[1046] Terminal: Sends input to the server.

[1047] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[1048] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[1049] Terminal: Provides generated responses to the user in text and audio.

[1050] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[1051] The processing flow will be explained below.

[1052] Step 1:

[1053] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[1054] Step 2:

[1055] Terminal: Sends the entered username and password to the server.

[1056] Step 3:

[1057] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[1058] Step 4:

[1059] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[1060] Step 5:

[1061] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1062] Step 6:

[1063] Terminal: Sends user input to the server.

[1064] Step 7:

[1065] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[1066] Step 8:

[1067] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[1068] Step 9:

[1069] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[1070] Step 10:

[1071] Server: Generates a notification based on the reminder content.

[1072] Step 11:

[1073] Device: Receive reminder notifications and notify users via text and voice.

[1074] Step 12:

[1075] User: Enter new data from your medication record or walking log into the app.

[1076] Step 13:

[1077] Terminal: Sends the entered health data to the server.

[1078] Step 14:

[1079] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[1080] Step 15:

[1081] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[1082] Step 16:

[1083] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[1084] Step 17:

[1085] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[1086] Processing flow incorporating emotion recognition

[1087] Step 18:

[1088] User: Type everyday conversations and questions into the app or speak into the microphone.

[1089] Step 19:

[1090] Terminal: Sends user input or voice data to the server.

[1091] Step 20:

[1092] Server: When voice input is received, the voice data is converted into text data using voice recognition technology.

[1093] Step 21:

[1094] Server: Uses an emotion engine to recognize emotions from the user's text or voice data. For example, the emotion engine identifies emotions such as "happy," "sad," and "angry."

[1095] Step 22:

[1096] Server: Based on the recognized emotion, the generative AI adjusts and generates an appropriate response. For example, if the user inputs "I'm happy," the generative AI will generate a response such as "That's great!"

[1097] Step 23:

[1098] Terminal: Receives the generated response and provides it to the user in text and voice.

[1099] Specific examples

[1100] 1. Set schedule reminders

[1101] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1102] Terminal: Sends input to the server.

[1103] Server: The generating AI understands the reminders and stores them in a database.

[1104] Device: Notify that a reminder has been set.

[1105] 2. Use of Emotion Recognition

[1106] User: In everyday conversation, type "I'm very happy today."

[1107] Terminal: Sends input to the server.

[1108] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[1109] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[1110] Terminal: Provides generated responses to the user in text and audio.

[1111] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[1112] Example 2

[1113] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1114] In today's multifunctional digital lifestyle, systems that integrate multiple functions are required to support users' efficient lifestyle and health management. However, conventional systems often provide separate functions for user input data processing, authentication, generative AI responses, emotion recognition, schedule management, and health data analysis, making integrated management difficult. Furthermore, it is difficult to seamlessly integrate these functions, which can result in a poor user experience. Furthermore, systems are unable to properly recognize the user's emotional state and reflect it in responses, resulting in unnatural communication.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1116] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative AI model, means for recognizing the user's emotional state using an emotion engine and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, and means for collecting and analyzing the user's health data and notifying in the event of an abnormality. This enables efficient lifestyle and health management for the user and realizes natural communication through emotion recognition.

[1117] "Means for receiving data entered by a user" refers to a device or program that allows a user to enter data in text or voice format via a smartphone or tablet and receive it.

[1118] A "means for authenticating a user" is a device or program that verifies a user's identity and grants appropriate access rights based on authentication information such as a user name and password.

[1119] A "means for generating a response to a user's input using generative artificial intelligence" is a device or program that uses a generative AI model to create an appropriate response based on the user's input data.

[1120] "Means for recognizing a user's emotional state using an emotion engine and reflecting that in responses" refers to a device or program for analyzing a user's input data and identifying emotions, and a mechanism for adjusting responses based on those emotions.

[1121] The "means for providing a response result to a user" refers to a device or program for displaying or audibly providing the generated response to a user.

[1122] The "means for managing the user's schedule information and sending reminder notifications" refers to a device or program that records the schedule information entered by the user and sends reminder notifications at appropriate times.

[1123] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to a device or program that collects health-related data entered by users, analyzes it using a machine learning algorithm, and notifies users in the event of an abnormality.

[1124] A "machine learning algorithm" is an algorithm that uses mathematical and statistical techniques to predict future data patterns or classify data based on past data.

[1125] "Speech recognition technology" is a technology for analyzing voice data and converting it into corresponding text data.

[1126] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, and integrates functions such as user input, authentication, AI response, reminder notification, health data analysis, emotion recognition, etc. A specific embodiment of this system will be described.

[1127] Receiving data entered by the user

[1128] Users use a smartphone or tablet to launch the app and input data via text or voice. The input content includes schedules, health data, daily communication, and emotional state. The device receives this input data and sends it to the server as text and voice data. Voice recognition technology (e.g., a voice recognition API) is used to convert the voice data into text.

[1129] User authentication

[1130] When a user starts using an app, they enter their username and password. The device sends this authentication data to the server. The server accesses a database (for example, an SQL database) and performs authentication based on the user information. If authentication is successful, a token is generated and sent to the device. This allows the user to use the system securely.

[1131] Response generation by generative artificial intelligence

[1132] The server uses a generative AI model to analyze the user's input and generate an appropriate response. It uses an emotion engine to recognize the user's emotional state and reflect it in the response. In this case, the generative AI model uses natural language processing technology, for example. The server sends the generated response to the device, which then provides it to the user in text and voice.

[1133] Emotion recognition by emotion engine

[1134] The server uses an emotion engine to analyze the user's input data (text and voice) to recognize their emotional state. Specifically, it identifies emotions such as joy, sadness, and anger based on the input data. As a result, the generative AI model adjusts the response content based on the recognized emotion and provides feedback to the user.

[1135] Schedule management and reminder notifications

[1136] The user enters future plans into the app. The device sends the entered schedule information to the server. The server saves the schedule information in a database and sets the timing for reminder notifications. When the reminder time arrives, the server generates the reminder content. The device notifies the user of the reminder via text and voice.

[1137] Health data collection and anomaly detection

[1138] Users enter health data such as their medication record and walking records into the app. The device then sends this health data to the server. The server then analyzes the health data stored in the database and uses machine learning algorithms to detect abnormalities. If an abnormality is detected, a notification is sent to set contacts (e.g., family members, medical institutions) and the user is also notified via the device.

[1139] Specific examples

[1140] 1. Set schedule reminders

[1141] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1142] Terminal: Sends input to the server.

[1143] Server: The generative AI model understands the reminders and stores them in a database.

[1144] Device: Notify that a reminder has been set.

[1145] 2. Use of Emotion Recognition

[1146] User: In everyday conversation, type "I'm very happy today."

[1147] Terminal: Sends input to the server.

[1148] Server: The generative AI model analyzes the input, and the emotion engine recognizes the emotion "happy."

[1149] Server: The generative AI model generates an appropriate response based on the user's emotions (e.g., "That's great!").

[1150] Terminal: Provides generated responses to the user in text and audio.

[1151] The system of the present invention streamlines users' daily lives and health management, and provides more natural and effective communication with users by recognizing emotions and responding appropriately. Furthermore, by utilizing natural language processing technology and machine learning algorithms, the system is capable of advanced data analysis and response generation, flexibly responding to diverse user needs.

[1152] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1153] Program processing flow

[1154] Step 1: User enters data

[1155] A user launches the app using a smartphone or tablet, then enters data via text or voice, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[1156] Input: User text or voice data.

[1157] Output: Text data or audio file.

[1158] Specific operation: In the case of voice input, the voice data is converted into text using a speech recognition API. The text data generated is "Set a reminder to take my medicine tomorrow at 10 AM."

[1159] Step 2: User authentication

[1160] The user enters their username and password into the app's login screen.

[1161] The terminal transmits these authentication data to the server.

[1162] The server compares the received authentication data with a database and authenticates the user.

[1163] Input: Username and Password.

[1164] Output: An authentication token or an error message.

[1165] Specific operation: The server accesses a database such as MySQL and verifies whether the username "user123" and password "password123" are correct. If authentication is successful, it generates a JWT token and sends it to the device. If authentication fails, it returns an error message.

[1166] Step 3: Generative AI generates a response

[1167] The server receives the user's text input and sends prompt sentences to a generative AI model (e.g., a natural language processing model) for analysis.

[1168] Input: User input data (e.g., "I'm very happy today").

[1169] Output: The generated response text (e.g., "That's great!").

[1170] Specific operation: Send a prompt to the generative AI model saying, "The user feels happy. Please generate an appropriate response." Receive the generated text response.

[1171] Step 4: Emotion Recognition with the Emotion Engine

[1172] The server sends the user's input data to an emotion engine (e.g., an emotion analysis API) to identify the emotional state.

[1173] Input: User input data (e.g., "I'm very happy today").

[1174] Output: Emotional state (e.g., "happy").

[1175] Specific operation: The server uses the emotion analysis API to recognize the emotion from the user's input. If the analysis result is "happy," that emotion is reflected in the generation AI's response.

[1176] Step 5: Providing response results

[1177] The server sends the generated response to the terminal.

[1178] The device provides the received response to the user, either displaying it as text or converting it to speech and reading it aloud.

[1179] Input: Response text from the generation AI.

[1180] Output: The response (text or audio) provided to the user.

[1181] Specific behavior: The device receives the response "That's great!" and displays it as text or reads it aloud using the speech synthesis API.

[1182] Step 6: Scheduling and Reminders

[1183] Users input their future plans into the app.

[1184] The terminal transmits this schedule information to the server.

[1185] The server stores the schedule information in a database and sets the timing of reminder notifications.

[1186] Input: Schedule information (e.g., "Take your medicine tomorrow at 10 AM").

[1187] Output: Remind notification.

[1188] Specific operation: The server saves the schedule information in a database and manages the timing of reminders using a Cron job. When the reminder time arrives, it sends a notification to the device, informing the user that it is time to take their medicine.

[1189] Step 7: Health data collection and anomaly detection

[1190] Users enter their medication records and walking records into the app.

[1191] The terminal transmits this health data to the server.

[1192] The server stores the health data in a database and analyzes it using machine learning algorithms.

[1193] Input: Health data (e.g. walking records, blood pressure values).

[1194] Output: Anomaly detection notification.

[1195] How it works: The server analyzes health data using machine learning algorithms such as Scikit-learn. If an abnormality is detected, it sends a notification to the specified contacts (e.g., family members, medical institutions), and notifies the user via their device that an abnormality has been detected.

[1196] keyword

[1197] Generative AI model, prompt sentence

[1198] (Application example 2)

[1199] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1200] Conventional systems mainly provide simple responses and notifications to user inputs, but lack the ability to respond or suggest products that take into account the user's emotional and health states. Therefore, there is a need for more personalized and effective systems that contribute to users' lifestyles and health management. Furthermore, to increase customer satisfaction in virtual stores, a personalized shopping assistant that reflects the customer's current emotional and health states is necessary.

[1201] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1202] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for recognizing the user's emotional state and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying the user in the event of an abnormality, and means for suggesting products based on the user's health data and emotional state. This allows the user to receive personalized responses and product suggestions that take into account their emotional state and health condition at any given time, significantly improving the shopping experience in virtual stores.

[1203] "Means for receiving data" is a function for receiving text or voice data entered by a user and sending it to a server.

[1204] "Means for authenticating users" refers to a function that verifies the authentication information (user name and password) entered by the user and confirms the user's legitimacy.

[1205] "Generative artificial intelligence" is a technique that allows computers to generate appropriate responses to specific inputs, often using machine learning or deep learning models (e.g., GPT-3).

[1206] "Means for recognizing emotional states and reflecting them in responses" refers to technology that analyzes the user's input, recognizes the emotions (joy, sadness, anger, etc.) at the time, and reflects that emotional state in responses.

[1207] The "means for providing the response result to the user" is a function for conveying the generated response to the user in text or voice.

[1208] "Means for managing schedule information and sending reminder notifications" is a function that manages the schedules set by the user and sends notifications when the scheduled time approaches.

[1209] "Means for collecting and analyzing health data and notifying in the event of an abnormality" refers to a function that collects and analyzes health data entered by the user and notifies in the event of an abnormality.

[1210] The "means for suggesting products" is a function for suggesting optimal products based on the user's health data and emotional state.

[1211] To implement this invention, the following system must be constructed. First, the device used by the user is assumed to be a smartphone, smart glasses, or a head-mounted display (HMD). The user inputs data through this device, and the data is sent to a server.

[1212] Data Receipt and Authentication

[1213] The user launches the app using a smartphone or HMD and inputs data via voice or text. The user is authenticated by entering a username and password. The authentication information is sent to the server and authenticated. If authentication is successful, the user can use the system.

[1214] User Input and Response Generation

[1215] The server uses a generative AI model (e.g., GPT-3) to analyze the user's input data and generate an appropriate response. An emotion recognition engine is used to analyze the emotional state contained in the user's input and reflect it in the response. For example, if a user inputs "I'm feeling great today," the generative AI model will recognize that emotion and generate a response such as "That's great!"

[1216] Schedule management and reminder notifications

[1217] Users enter their schedule information into the app, and the server stores that information in a database. When it's time for a reminder, the server generates the reminder content and sends it to the user's device. For example, if you enter "Set a reminder to take my medicine tomorrow at 10 a.m.", a reminder will be sent to the user at the specified time.

[1218] Health data analysis and product proposals

[1219] The server collects the user's health data and analyzes it using machine learning algorithms. If an abnormality is detected, the server notifies the user and their designated contacts. The server also makes personalized product suggestions based on the user's health and emotional state. For example, if a user inputs "I feel a little tired today," the server will suggest "relaxation products" based on the user's low level of exercise.

[1220] Hardware and Software

[1221] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[1222] Software: Generative AI models (e.g., GPT-3), emotion recognition engines, health data analysis algorithms, product recommendation engines

[1223] Examples and prompts

[1224] For example, the following process occurs:

[1225] 1. The user puts on the HMD and speaks, "What products do you have available today?"

[1226] 2. The emotion recognition engine recognizes the emotion "happy."

[1227] 3. The generative AI generates a response: "We're having a special sale today!"

[1228] 4. The product suggestion engine suggests a "set of fresh vegetables."

[1229] 5. The HMD presents the generated responses and suggestions to the user visually and audibly.

[1230] An example prompt is:

[1231] User sentiment: Happy

[1232] Health data: Blood pressure: normal, Heart rate: normal, Exercise: low

[1233] User Input: What products do you have available today?

[1234] Response: We have a special sale today!

[1235] The system allows users to receive a personalized shopping experience that takes into account their emotional and health state at any given time.

[1236] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1237] Step 1:

[1238] The user starts up their smartphone or HMD and inputs input data (voice or text). For example, the user might say, "What products do you have available today?" The input data is converted into text data using voice recognition technology.

[1239] Input: User voice or text input

[1240] Data processing: Converting voice data into text using voice recognition technology

[1241] Output: Text data

[1242] Step 2:

[1243] The device sends the converted text data to the server, which authenticates the user if necessary. If authentication is successful, the server sends the input data to the generative AI model.

[1244] Input: Text data (user input)

[1245] Data Processing: User Credential Verification

[1246] Output: Authenticated text data, user ID

[1247] Step 3:

[1248] The server uses a generative AI model to analyze the text data and generate appropriate responses. It also uses an emotion recognition engine to recognize emotional states (e.g., happy, sad) from the input data and reflect them in the response.

[1249] Input: authenticated text data, user ID

[1250] Data processing: Response generation using a generative AI model, emotional state analysis using an emotion recognition engine

[1251] Output: The generated response text

[1252] Step 4:

[1253] The server stores the user's schedule information in a database and sets appropriate reminder times. For example, if a user inputs "Set a reminder to take my medicine at 10:00 AM tomorrow," the server generates the reminder content and sends it to the device when the time comes.

[1254] Input: Schedule information (e.g., time to take medicine)

[1255] Data processing: setting the timing of reminder notifications, generating reminder notification content

[1256] Output: Reminder notification

[1257] Step 5:

[1258] The server collects the user's health data and analyzes it using machine learning algorithms. For example, if the user's step count or blood pressure data is collected, the server analyzes the data to determine whether it is normal or not. If an abnormality is detected, the server sends a notification to the specified contacts (e.g., medical institutions).

[1259] Input: Health data (e.g., number of steps, blood pressure)

[1260] Data processing: Data analysis using machine learning algorithms

[1261] Output: Abnormality detection notification

[1262] Step 6:

[1263] The server then makes personalized product recommendations based on the user's emotional state and health data. For example, if a user inputs "I feel a little tired today" and health data indicates that they are not exercising much, the server will suggest "relaxation products."

[1264] Input: Emotional state, health data

[1265] Data processing: Analysis by product recommendation engine based on emotional state and health data

[1266] Output: Personalized product recommendations

[1267] Step 7:

[1268] The device provides the user with the responses, notifications, and product suggestions sent from the server visually and audibly, for example, by displaying the responses and product suggestions generated through the HMD and providing audio notifications.

[1269] Input: Generated response text, reminders, product suggestions

[1270] Data processing: Converting responses and notifications into visual and audio formats

[1271] Output: Visual and audio notification to the user

[1272] In this way, users can receive a shopping experience that is personalized to their emotional and health state at the time.

[1273] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1274] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1275] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1276] [Third embodiment]

[1277] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1278] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1279] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1280] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1281] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1282] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1283] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1284] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1285] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1286] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1287] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1288] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1289] The system of the present invention is constructed to allow users to perform various tasks related to daily life and health management. The functions of each component and the processing flow will be specifically explained below.

[1290] Receiving data entered by the user

[1291] User

[1292] Users use devices such as smartphones and tablets to input data into the app by text or voice, including schedules, health data, and everyday conversations.

[1293] Terminal

[1294] The device receives user input and sends it to the server as text data. If voice data is input, it is converted into text using voice recognition technology.

[1295] User authentication

[1296] User

[1297] When users start using the app, they enter their username and password.

[1298] Terminal

[1299] The terminal transmits these authentication data to the server.

[1300] server

[1301] The server accesses the database and performs authentication based on the user information. If authentication is successful, a session is started and an authentication token is sent to the terminal. If authentication fails, an error message is returned.

[1302] Response generation by generative artificial intelligence

[1303] server

[1304] The server uses a generative AI (e.g., GPT-4) to generate a response based on the user's input. The generated response is temporarily stored on the server.

[1305] Terminal

[1306] The terminal receives the generated response from the server and provides it to the user in text and audio.

[1307] Schedule management and reminder notifications

[1308] User

[1309] Users input their future plans into the app.

[1310] Terminal

[1311] This input is sent to the server.

[1312] server

[1313] The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, the server generates the corresponding notification.

[1314] Terminal

[1315] The device receives the reminder notification and notifies the user, for example, "A reminder to take your medicine has been set for 10:00 AM."

[1316] Health data collection and anomaly detection

[1317] User

[1318] Users enter their medication record and walking records into the app.

[1319] Terminal

[1320] The entered health data is sent to the server.

[1321] server

[1322] The server stores this data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, it notifies the specified contacts (e.g., family members, medical institutions).

[1323] Terminal

[1324] Reports detected anomalies to users and provides instructions on what to do.

[1325] Specific examples

[1326] 1. Set schedule reminders

[1327] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[1328] Device: Converts speech to text and sends it to the server.

[1329] Server: Stores reminders in a database and generates notifications at the scheduled times.

[1330] Device: Receive reminder notifications and notify the user.

[1331] 2. Detecting Anomalies in Health Data

[1332] User: Updates walking record and enters it into the app.

[1333] Terminal: Sends data to the server.

[1334] Server: Analyzes data, detects abnormalities, and sends relevant notifications to family members and medical institutions.

[1335] Device: Notify the user that an anomaly has been detected.

[1336] The system of the present invention makes it possible to streamline the user's lifestyle and health management and provide comprehensive daily support.

[1337] The processing flow will be explained below.

[1338] Step 1:

[1339] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[1340] Step 2:

[1341] Terminal: Sends the entered username and password to the server.

[1342] Step 3:

[1343] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[1344] Step 4:

[1345] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[1346] Step 5:

[1347] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1348] Step 6:

[1349] Terminal: Sends user input to the server.

[1350] Step 7:

[1351] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[1352] Step 8:

[1353] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[1354] Step 9:

[1355] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[1356] Step 10:

[1357] Server: Generates a notification based on the reminder content.

[1358] Step 11:

[1359] Device: Receive reminder notifications and notify users via text and voice.

[1360] Step 12:

[1361] User: Enter new data from your medication record or walking log into the app.

[1362] Step 13:

[1363] Terminal: Sends the entered health data to the server.

[1364] Step 14:

[1365] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[1366] Step 15:

[1367] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[1368] Step 16:

[1369] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[1370] Step 17:

[1371] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[1372] Step 18:

[1373] User: Type everyday conversations and questions into the app or speak into the microphone.

[1374] Step 19:

[1375] Terminal: Sends user input or voice data to the server.

[1376] Step 20:

[1377] Server: When voice input is received, it converts it into text using speech recognition technology, and uses generative AI to generate appropriate responses to user questions and conversations.

[1378] Step 21:

[1379] Terminal: Receives the generated response from the server and provides it to the user in text and voice.

[1380] Example 1

[1381] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1382] In modern society, it is important for individual users to efficiently manage their daily lives and health, and there is a growing demand for systems that support this. However, conventional systems do not integrate various functions such as receiving user data, authenticating users, generating responses, managing schedules, and analyzing health data, which makes them difficult to use and makes it difficult to manage efficiently. Furthermore, there are issues with the recognition accuracy and response speed when using voice input.

[1383] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1384] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative artificial intelligence model, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying in the event of an abnormality, and means for converting voice input into text data using voice recognition technology. This enables the user to efficiently manage their daily life and health using a single system, and also improves the recognition accuracy and response speed for voice input.

[1385] The "means for receiving data entered by the user" is a function that receives information entered by the user via text or voice to the server via the terminal.

[1386] A "means of authenticating a user" is the process of identifying a user using authentication information, such as a username and password, and initiating a session on a server.

[1387] "Means for generating a response to user input using a generative artificial intelligence model" refers to a technology that analyzes input data from a user and generates an appropriate response using an artificial intelligence model.

[1388] The "means for providing a response result to a user" is a function that sends a response generated by a server to a terminal and allows the user to receive the response in text or voice.

[1389] "Means for managing the user's schedule information and sending reminder notifications" refers to a system that saves the schedule entered by the user and generates reminder notifications at pre-set times to notify the user.

[1390] "Means of collecting and analyzing user health data and notifying users in the event of abnormalities" refers to a function that stores the health information entered by users in a database, detects abnormalities using a machine learning algorithm, and notifies users of the results.

[1391] "Means for converting voice input into text data using voice recognition technology" refers to technology that converts data input by voice by a user into text data using voice recognition technology and transmits it to a server.

[1392] A "machine learning algorithm" is a mathematical model that analyzes large amounts of data to learn patterns and trends and make future predictions and decisions.

[1393] A "generative artificial intelligence model" refers to an algorithm that understands human natural language and generates appropriate responses based on that language, and is a fundamental technology for natural language processing.

[1394] A "database" is a storage system that centrally manages various data such as user information, schedule information, and health data, and allows efficient access.

[1395] The system of the present invention is designed to enable a user to efficiently perform tasks related to daily life and health management. Specific embodiments for carrying out the present invention will be described in detail below.

[1396] System Configuration

[1397] The system consists of three main components: users, devices, and servers. Users access the system using devices such as smartphones and tablets. The devices send input data from the users to the server, which processes the data, generates a response, and returns the result to the device.

[1398] Hardware and software used

[1399] Device: An input device such as a smartphone, tablet, or PC.

[1400] Server: Cloud server or on-premise server.

[1401] Database: A database (e.g., MySQL) for managing user information, schedule information, and health data.

[1402] Speech recognition technology: APIs for converting voice input into text (e.g., Google Speech-to-Text).

[1403] Text generation AI model: A generative artificial intelligence model (e.g., GPT-4).

[1404] Speech synthesis technology: APIs for converting text to speech (e.g., Google Text-to-Speech).

[1405] Machine learning algorithms: Algorithms for analyzing health data (e.g., TensorFlow).

[1406] Processing flow

[1407] 1. User authentication:

[1408] User: Enter your username and password on the login screen.

[1409] Terminal: Sends the entered authentication information to the server.

[1410] Server: Accesses the database and checks the authentication information. If authentication is successful, generates a session ID and authentication token and sends them to the device.

[1411] 2. Data entry and submission:

[1412] User: Enters data into the app via text or voice. For example, "Add a meeting for 3 PM."

[1413] Terminal: Converts speech into text and sends the data to the server.

[1414] 3. Response generation using generative artificial intelligence models:

[1415] Server: Sends user input data to the generative artificial intelligence model and generates a response, e.g., "A meeting has been scheduled for 3 PM."

[1416] 4. Response provision:

[1417] Server: Sends the generated response to the terminal.

[1418] Terminal: Provides a response to the user in text or voice.

[1419] 5. Schedule Management and Reminders:

[1420] User: Enters future events into the app. Example: "Set a reminder to take my medicine tomorrow at 10 AM."

[1421] Terminal: Sends input data to the server.

[1422] Server: Saves schedule data to a database and generates notifications at the reminder time.

[1423] Device: Receives notifications and notifies the user.

[1424] 6. Health data collection and anomaly detection:

[1425] User: Enter walking records and health data. Example: "Today's steps are 5000."

[1426] Terminal: Sends data to the server.

[1427] Server: Stores data in a database, analyzes it using machine learning algorithms, and sends notifications to configured contacts if an anomaly is detected.

[1428] Specific examples

[1429] Set schedule reminders:

[1430] User: Says, "Set a reminder to take my medicine tomorrow at 10 AM."

[1431] Device: Converts speech to text and sends it to the server.

[1432] Server: Stores reminders in a database and generates notifications at the scheduled times.

[1433] Device: Receive reminder notifications and notify the user.

[1434] Anomaly detection in health data:

[1435] User: Enters "5000 steps today."

[1436] Terminal: Sends data to the server.

[1437] Server: Analyzes the data and if an abnormality is detected, sends a notification to family members or medical institutions.

[1438] Device: Notify the user that an anomaly has been detected.

[1439] The system of the present invention allows users to efficiently manage their daily lives and health management on a single platform. In addition, the use of generative artificial intelligence models enables appropriate and prompt responses to user input.

[1440] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1441] Step 1: User authentication

[1442] ---

[1443] User:

[1444] Enter your username and password on the login screen. Input data: username (text), password (text).

[1445] Device:

[1446] Receives the entered authentication information and sends it to the server. Output data: Authentication information (JSON format).

[1447] server:

[1448] Accesses the database and checks the authentication information. If successful, generates a session ID and authentication token and sends them to the device. If unsuccessful, returns an error message.

[1449] Input data: Authentication information (JSON format)

[1450] Data calculation: User information matching and token generation

[1451] Output data: Session ID, authentication token (if successful) or error message (if unsuccessful)

[1452] Step 2: Enter and submit data

[1453] ---

[1454] User:

[1455] Enter data into the app by text or voice, for example, "Add a meeting for 3 PM."

[1456] Input data: text or audio data

[1457] Device:

[1458] For voice input, speech recognition technology is used to convert speech into text.

[1459] Input data: Audio data

[1460] Data Computing: Speech-to-Text Conversion

[1461] Output data: Text data

[1462] Device:

[1463] The converted text data or the directly entered text is sent to the server.

[1464] Output data: Input data (JSON format)

[1465] Step 3: Response generation using a generative artificial intelligence model

[1466] ---

[1467] server:

[1468] The input data is analyzed and applied to a generative artificial intelligence model. For example, the prompt sentence "Add a meeting at 3 PM" is input to the generative AI.

[1469] Input data: Text data (prompt)

[1470] Data Computation: Response Generation with Generative AI Models

[1471] Output data: Response text

[1472] server:

[1473] The generated response is temporarily stored and sent to the terminal.

[1474] Output data: Response data (JSON format)

[1475] Step 4: Provide a response

[1476] ---

[1477] Device:

[1478] The response data received from the server is displayed to the user. If audio is desired, a technology is used to convert text data into audio.

[1479] Input data: Response data (JSON format)

[1480] Data Computing: Text-to-Speech

[1481] Output data: Audio data

[1482] Device:

[1483] The response is displayed or played aloud to the user.

[1484] Output data: User presented

[1485] Step 5: Scheduling and Reminders

[1486] ---

[1487] User:

[1488] Enter future events into the app, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[1489] Input data: Schedule data (text)

[1490] Device:

[1491] Sends input data to the server.

[1492] Output data: Schedule data (JSON format)

[1493] server:

[1494] Save schedule data to a database and generate notifications at reminder times.

[1495] Input data: Schedule data

[1496] Data calculation: Schedule saving and reminder setting

[1497] Output data: Reminder notification data

[1498] Device:

[1499] Receive reminder notifications and notify users.

[1500] Input data: Reminder notification data

[1501] Output data: User presented

[1502] Step 6: Health data collection and anomaly detection

[1503] ---

[1504] User:

[1505] Enter your health data into the app. For example, enter "Today's steps are 5,000."

[1506] Input data: Health data (text)

[1507] Device:

[1508] Send the data to the server.

[1509] Output data: Health data (JSON format)

[1510] server:

[1511] Health data is stored in a database and analyzed using machine learning algorithms.

[1512] Input data: Health data

[1513] Data Computing: Data Analysis and Anomaly Detection

[1514] Output data: Analysis results (notification data in case of abnormality)

[1515] server:

[1516] If an abnormality is detected, a notification will be sent to the specified contacts (e.g., family members, medical institutions).

[1517] Output data: Error notification data

[1518] Device:

[1519] Notify the user that an abnormality has been detected and provide instructions on what action to take.

[1520] Input data: Error notification data

[1521] Output data: User presented

[1522] (Application example 1)

[1523] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1524] Modern factories require systems to manage worker health and improve the safety of the work environment. Conventional systems struggle to monitor worker health data and work environment information in real time, and respond immediately when an abnormality is detected. Furthermore, they do not effectively manage individual work schedules or send reminder notifications. This can lead to reduced work efficiency and increased safety risks for workers. A system is needed to resolve these issues and improve work efficiency and safety within factories.

[1525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1526] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and sending a notification when an abnormality occurs, means for monitoring the worker's health data in real time and issuing a warning when an abnormality is detected, means for managing the work schedule and sending reminder notifications, and means for monitoring the temperature, hazardous gas concentration, etc. of the work environment and sending a notification when an abnormal value is detected. This makes it possible to monitor the worker's health condition in real time and take prompt action when an abnormality is detected, ensuring the safety of the work environment and improving work efficiency.

[1527] "Means for receiving data entered by a user" refers to means for receiving and processing text or voice data entered by a user using a device such as a smartphone or tablet.

[1528] "Means for authenticating users" refers to the means for authenticating that a user is a legitimate user using a user name and password.

[1529] "Means for generating responses to user input using generative artificial intelligence" means means for using a generative AI model (e.g., GPT-4) to generate appropriate responses based on input from a user.

[1530] The "means for providing a response result to a user" refers to a means for transmitting a response generated on a server to a user's terminal and presenting it in text or audio format.

[1531] "Means for managing user schedule information and sending reminder notifications" refers to means for saving and managing the plans and schedule information set by the user and sending reminder notifications at designated times.

[1532] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to the means for storing and analyzing health data entered by users on a server, and notifying users in the event of an abnormality.

[1533] "Means for monitoring workers' health data in real time and issuing warnings when abnormalities are detected" refers to means for collecting health data in real time from workers' smartwatches and healthcare devices and issuing warnings when abnormalities are detected.

[1534] The "means for managing work schedules and issuing reminder notifications" is a means for managing the work schedules of workers and issuing reminder notifications based on the set schedules.

[1535] "Means for monitoring the temperature, hazardous gas concentration, etc. of the working environment and issuing a notification if an abnormal value is detected" refers to a means for monitoring the temperature and hazardous gas concentration within a factory using sensors and issuing an immediate notification if an abnormal value is detected.

[1536] The system of this invention includes functions for receiving data entered by users, authenticating users, generating responses using artificial intelligence, providing response results, managing schedules and sending reminders, collecting and analyzing health data, and notifying users when an abnormality occurs. Furthermore, to adapt to a factory environment, the system adds functions for monitoring workers' health data in real time, issuing warnings when an abnormality is detected, and monitoring the work environment.

[1537] System Configuration

[1538] 1. User Device

[1539] Using a smartphone or tablet, the user inputs data by voice or text. The voice data is converted into text using voice recognition technology. The received data is sent to the server.

[1540] 2. Server

[1541] User authentication: The server accesses the database and performs authentication based on the username and password. If authentication is successful, a session is initiated and an authentication token is sent to the user's device. If authentication fails, an error message is returned.

[1542] Response generation using generative AI: The server uses a generative AI model (e.g., GPT-4) to generate a response based on user input. The generated response is temporarily stored on the server and then sent to the user's device.

[1543] Schedule management and reminder notifications: The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, it generates a corresponding notification and sends it to the user's device.

[1544] Health data collection and analysis: Health data is collected in real time from workers' smartwatches and healthcare devices, and analyzed using machine learning algorithms. If an abnormality is detected, notification will be sent to designated contacts.

[1545] Work environment monitoring: Sensors are used to monitor the temperature and hazardous gas concentrations in the work environment, and if abnormal values ​​are detected, workers are immediately notified.

[1546] Work Schedule Management: Manage the work schedules of workers and send reminder notifications based on the set schedule.

[1547] Hardware and software used

[1548] Hardware: smartphones, tablets, smartwatches, sensors

[1549] Software: speech recognition technology, generative AI models (e.g., GPT-4), machine learning algorithms

[1550] Specific examples

[1551] 1. Set schedule reminders

[1552] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[1553] Device: Converts speech to text and sends it to the server.

[1554] Server: Stores reminders in a database and generates notifications at the scheduled times.

[1555] On your device: Receive reminder notifications and notify the user.

[1556] 2. Detecting Anomalies in Health Data

[1557] User: Updates walking log and enters it into the app.

[1558] Terminal: Sends data to the server.

[1559] Server: Analyzes data, detects anomalies, and sends relevant notifications to family members and medical institutions.

[1560] Device: Notify the user that an anomaly has been detected.

[1561] Prompt Sentence Examples

[1562] Implement a program that collects health data for worker ID: 123456 and notifies if any abnormalities are found. The collected data is heart rate and number of steps, and a notification will be sent if the heart rate exceeds 100 or the number of steps is less than 500. A notification will also be sent if the temperature in the work environment exceeds 35 degrees or the concentration of harmful gases exceeds 50. The APIs to be used are as follows.

[1563] Health data collection: https: / / api.smartwatch.com / users / {user_id} / health

[1564] Speech recognition: https: / / api.speech-to-text.com / convert

[1565] Work schedule management: https: / / api.factory-robot.com / schedules

[1566] Environmental monitoring: https: / / api.factory-sensors.com / environment

[1567] Send notification: https: / / api.notification.com / send

[1568] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1569] Step 1:

[1570] Receiving Data Input

[1571] User: Opens the app on their smartphone and enters voice or text data. For example, they might say, "Set a reminder to take my medicine tomorrow at 10 AM."

[1572] Input: Voice or text data

[1573] Terminal: When voice data is input, it is converted into text using voice recognition technology (e.g., API: https: / / api.speech-to-text.com / convert). The converted text data is sent to the server.

[1574] Output: User input as text data

[1575] Step 2:

[1576] User authentication

[1577] Terminal: Provides an interface for entering a username and password, where the user enters their credentials.

[1578] Input: Username and Password

[1579] Device: Sends authentication information to the server.

[1580] Server: Accesses the database and verifies the entered authentication information. If authentication is successful, starts a session, generates an authentication token, and sends it to the device. If authentication fails, generates an error message.

[1581] Output: Authentication token or error message

[1582] Step 3:

[1583] Response Generation

[1584] Server: Based on user input, generates appropriate responses using a generative AI model (e.g., GPT-4).

[1585] Input: User-entered text data

[1586] Server: Temporarily stores the generated response and sends it to the user's device.

[1587] Output: The text data of the generated response

[1588] Step 4:

[1589] Providing a response

[1590] Device: Displays or speaks to the user the generated response received from the server, for example, "A reminder to take your medicine has been set for tomorrow at 10 AM."

[1591] Input: The text data of the generated response

[1592] Terminal: Displaying text data in a user interface or playing it aloud.

[1593] Output: Providing a visual or audio response

[1594] Step 5:

[1595] Schedule management and reminder notifications

[1596] User: Enters an appointment. For example, "Take medicine at 10:00 AM tomorrow."

[1597] Input: User schedule input

[1598] Terminal: Sends schedule data to the server.

[1599] Server: Stores schedule data in a database and sets reminder notification timing.

[1600] Output: Schedule data stored in the database and reminder times

[1601] Step 6:

[1602] Real-time monitoring of health data

[1603] User: Wears a smartwatch or healthcare device to collect health data.

[1604] Input: Health data collected from smartwatches and healthcare devices

[1605] Device: Sends health data to the server.

[1606] Server: Analyzes health data using machine learning algorithms and generates alerts and notifies configured contacts if anomalies are detected.

[1607] Output: Health data analysis results and warning notifications in case of abnormalities

[1608] Step 7:

[1609] Work environment monitoring

[1610] Server: Use sensors to collect environmental data such as temperature and harmful gas concentrations in the factory (e.g., API: https: / / api.factory-sensors.com / environment).

[1611] Input: Environmental data from sensors

[1612] Server: Analyzes environmental data and immediately notifies workers if any abnormal values ​​are detected.

[1613] Output: Environmental data analysis results and notifications in case of abnormalities

[1614] Step 8:

[1615] Work schedule management and reminder notifications

[1616] User: Sets the work schedule, for example, "Start the next process at 10 o'clock."

[1617] Input: User work schedule input

[1618] Server: Stores work schedule data in a database and sets reminder notification timing. When the reminder time arrives, it generates a corresponding notification and sends it to the device.

[1619] Output: Work schedule data and reminder notifications

[1620] Through the above processing steps, this system can realize safety management of workers at the factory site and improvement of work efficiency.

[1621] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1622] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, integrating functions such as user input, authentication, AI response, reminder notifications, health data analysis, emotion recognition, etc. The following describes in detail the function of each component and its processing flow.

[1623] Receiving data entered by the user

[1624] User

[1625] Users use their smartphone or tablet to launch the app and enter data via text or voice, including schedules, health data, daily communication, and emotional state.

[1626] Terminal

[1627] The device receives user input and transmits it to the server as text and voice data, and uses voice recognition technology to convert the voice data into text.

[1628] User authentication

[1629] User

[1630] When users start using the app, they enter their username and password.

[1631] Terminal

[1632] The terminal transmits these authentication data to the server.

[1633] server

[1634] The server accesses the database and performs authentication based on the user information. If authentication is successful, it generates a user ID and starts a session. It sends an authentication token to the terminal, and if authentication fails, it returns an error message.

[1635] Response generation by generative artificial intelligence

[1636] server

[1637] The server uses generative AI to analyze user input and generate appropriate responses, and an emotion engine to recognize the user's emotional state and reflect it in the response.

[1638] Terminal

[1639] The terminal receives the generated response from the server and provides it to the user in text and audio.

[1640] Emotion recognition by emotion engine

[1641] server

[1642] The emotion engine analyzes user input data (text and voice) and recognizes the user's emotional state, for example, identifying whether the user's input contains emotions such as joy, sadness, or anger.

[1643] server

[1644] Based on the recognized emotions, the generative AI adjusts its response and provides appropriate feedback to the user.

[1645] Schedule management and reminder notifications

[1646] User

[1647] Users input their future plans into the app.

[1648] Terminal

[1649] The entered schedule information is sent to the server.

[1650] server

[1651] The server stores the schedule information in a database and sets the timing of reminder notifications. When the time comes, the server generates the reminder content.

[1652] Terminal

[1653] The device will receive reminder notifications and notify the user via text and voice.

[1654] Health data collection and anomaly detection

[1655] User

[1656] Users enter their medication record and walking records into the app.

[1657] Terminal

[1658] The entered health data is sent to the server.

[1659] server

[1660] The server stores health data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, a notification is sent to the specified contacts (e.g., family members, medical institutions).

[1661] Terminal

[1662] Notify users when data is updated and alert them if anomalies are detected.

[1663] Specific examples

[1664] 1. Set schedule reminders

[1665] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1666] Terminal: Sends input to the server.

[1667] Server: The generating AI understands the reminders and stores them in a database.

[1668] Device: Notify that a reminder has been set.

[1669] 2. Use of Emotion Recognition

[1670] User: In everyday conversation, type "I'm very happy today."

[1671] Terminal: Sends input to the server.

[1672] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[1673] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[1674] Terminal: Provides generated responses to the user in text and audio.

[1675] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[1676] The processing flow will be explained below.

[1677] Step 1:

[1678] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[1679] Step 2:

[1680] Terminal: Sends the entered username and password to the server.

[1681] Step 3:

[1682] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[1683] Step 4:

[1684] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[1685] Step 5:

[1686] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1687] Step 6:

[1688] Terminal: Sends user input to the server.

[1689] Step 7:

[1690] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[1691] Step 8:

[1692] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[1693] Step 9:

[1694] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[1695] Step 10:

[1696] Server: Generates a notification based on the reminder content.

[1697] Step 11:

[1698] Device: Receive reminder notifications and notify users via text and voice.

[1699] Step 12:

[1700] User: Enter new data from your medication record or walking log into the app.

[1701] Step 13:

[1702] Terminal: Sends the entered health data to the server.

[1703] Step 14:

[1704] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[1705] Step 15:

[1706] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[1707] Step 16:

[1708] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[1709] Step 17:

[1710] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[1711] Processing flow incorporating emotion recognition

[1712] Step 18:

[1713] User: Type everyday conversations and questions into the app or speak into the microphone.

[1714] Step 19:

[1715] Terminal: Sends user input or voice data to the server.

[1716] Step 20:

[1717] Server: When voice input is received, the voice data is converted into text data using voice recognition technology.

[1718] Step 21:

[1719] Server: Uses an emotion engine to recognize emotions from the user's text or voice data. For example, the emotion engine identifies emotions such as "happy," "sad," and "angry."

[1720] Step 22:

[1721] Server: Based on the recognized emotion, the generative AI adjusts and generates an appropriate response. For example, if the user inputs "I'm happy," the generative AI will generate a response such as "That's great!"

[1722] Step 23:

[1723] Terminal: Receives the generated response and provides it to the user in text and voice.

[1724] Specific examples

[1725] 1. Set schedule reminders

[1726] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1727] Terminal: Sends input to the server.

[1728] Server: The generating AI understands the reminders and stores them in a database.

[1729] Device: Notify that a reminder has been set.

[1730] 2. Use of Emotion Recognition

[1731] User: In everyday conversation, type "I'm very happy today."

[1732] Terminal: Sends input to the server.

[1733] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[1734] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[1735] Terminal: Provides generated responses to the user in text and audio.

[1736] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[1737] Example 2

[1738] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1739] In today's multifunctional digital lifestyle, systems that integrate multiple functions are required to support users' efficient lifestyle and health management. However, conventional systems often provide separate functions for user input data processing, authentication, generative AI responses, emotion recognition, schedule management, and health data analysis, making integrated management difficult. Furthermore, it is difficult to seamlessly integrate these functions, which can result in a poor user experience. Furthermore, systems are unable to properly recognize the user's emotional state and reflect it in responses, resulting in unnatural communication.

[1740] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1741] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative AI model, means for recognizing the user's emotional state using an emotion engine and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, and means for collecting and analyzing the user's health data and notifying in the event of an abnormality. This enables efficient lifestyle and health management for the user and realizes natural communication through emotion recognition.

[1742] "Means for receiving data entered by a user" refers to a device or program that allows a user to enter data in text or voice format via a smartphone or tablet and receive it.

[1743] A "means for authenticating a user" is a device or program that verifies a user's identity and grants appropriate access rights based on authentication information such as a user name and password.

[1744] A "means for generating a response to a user's input using generative artificial intelligence" is a device or program that uses a generative AI model to create an appropriate response based on the user's input data.

[1745] "Means for recognizing a user's emotional state using an emotion engine and reflecting that in responses" refers to a device or program for analyzing a user's input data and identifying emotions, and a mechanism for adjusting responses based on those emotions.

[1746] The "means for providing a response result to a user" refers to a device or program for displaying or audibly providing the generated response to a user.

[1747] The "means for managing the user's schedule information and sending reminder notifications" refers to a device or program that records the schedule information entered by the user and sends reminder notifications at appropriate times.

[1748] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to a device or program that collects health-related data entered by users, analyzes it using a machine learning algorithm, and notifies users in the event of an abnormality.

[1749] A "machine learning algorithm" is an algorithm that uses mathematical and statistical techniques to predict future data patterns or classify data based on past data.

[1750] "Speech recognition technology" is a technology for analyzing voice data and converting it into corresponding text data.

[1751] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, and integrates functions such as user input, authentication, AI response, reminder notification, health data analysis, emotion recognition, etc. A specific embodiment of this system will be described.

[1752] Receiving data entered by the user

[1753] Users use a smartphone or tablet to launch the app and input data via text or voice. The input content includes schedules, health data, daily communication, and emotional state. The device receives this input data and sends it to the server as text and voice data. Voice recognition technology (e.g., a voice recognition API) is used to convert the voice data into text.

[1754] User authentication

[1755] When a user starts using an app, they enter their username and password. The device sends this authentication data to the server. The server accesses a database (for example, an SQL database) and performs authentication based on the user information. If authentication is successful, a token is generated and sent to the device. This allows the user to use the system securely.

[1756] Response generation by generative artificial intelligence

[1757] The server uses a generative AI model to analyze the user's input and generate an appropriate response. It uses an emotion engine to recognize the user's emotional state and reflect it in the response. In this case, the generative AI model uses natural language processing technology, for example. The server sends the generated response to the device, which then provides it to the user in text and voice.

[1758] Emotion recognition by emotion engine

[1759] The server uses an emotion engine to analyze the user's input data (text and voice) to recognize their emotional state. Specifically, it identifies emotions such as joy, sadness, and anger based on the input data. As a result, the generative AI model adjusts the response content based on the recognized emotion and provides feedback to the user.

[1760] Schedule management and reminder notifications

[1761] The user enters future plans into the app. The device sends the entered schedule information to the server. The server saves the schedule information in a database and sets the timing for reminder notifications. When the reminder time arrives, the server generates the reminder content. The device notifies the user of the reminder via text and voice.

[1762] Health data collection and anomaly detection

[1763] Users enter health data such as their medication record and walking records into the app. The device then sends this health data to the server. The server then analyzes the health data stored in the database and uses machine learning algorithms to detect abnormalities. If an abnormality is detected, a notification is sent to set contacts (e.g., family members, medical institutions) and the user is also notified via the device.

[1764] Specific examples

[1765] 1. Set schedule reminders

[1766] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1767] Terminal: Sends input to the server.

[1768] Server: The generative AI model understands the reminders and stores them in a database.

[1769] Device: Notify that a reminder has been set.

[1770] 2. Use of Emotion Recognition

[1771] User: In everyday conversation, type "I'm very happy today."

[1772] Terminal: Sends input to the server.

[1773] Server: The generative AI model analyzes the input, and the emotion engine recognizes the emotion "happy."

[1774] Server: The generative AI model generates an appropriate response based on the user's emotions (e.g., "That's great!").

[1775] Terminal: Provides generated responses to the user in text and audio.

[1776] The system of the present invention streamlines users' daily lives and health management, and provides more natural and effective communication with users by recognizing emotions and responding appropriately. Furthermore, by utilizing natural language processing technology and machine learning algorithms, the system is capable of advanced data analysis and response generation, flexibly responding to diverse user needs.

[1777] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1778] Program processing flow

[1779] Step 1: User enters data

[1780] A user launches the app using a smartphone or tablet, then enters data via text or voice, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[1781] Input: User text or voice data.

[1782] Output: Text data or audio file.

[1783] Specific operation: In the case of voice input, the voice data is converted into text using a speech recognition API. The text data generated is "Set a reminder to take my medicine tomorrow at 10 AM."

[1784] Step 2: User authentication

[1785] The user enters their username and password into the app's login screen.

[1786] The terminal transmits these authentication data to the server.

[1787] The server compares the received authentication data with a database and authenticates the user.

[1788] Input: Username and Password.

[1789] Output: An authentication token or an error message.

[1790] Specific operation: The server accesses a database such as MySQL and verifies whether the username "user123" and password "password123" are correct. If authentication is successful, it generates a JWT token and sends it to the device. If authentication fails, it returns an error message.

[1791] Step 3: Generative AI generates a response

[1792] The server receives the user's text input and sends prompt sentences to a generative AI model (e.g., a natural language processing model) for analysis.

[1793] Input: User input data (e.g., "I'm very happy today").

[1794] Output: The generated response text (e.g., "That's great!").

[1795] Specific operation: Send a prompt to the generative AI model saying, "The user feels happy. Please generate an appropriate response." Receive the generated text response.

[1796] Step 4: Emotion Recognition with the Emotion Engine

[1797] The server sends the user's input data to an emotion engine (e.g., an emotion analysis API) to identify the emotional state.

[1798] Input: User input data (e.g., "I'm very happy today").

[1799] Output: Emotional state (e.g., "happy").

[1800] Specific operation: The server uses the emotion analysis API to recognize the emotion from the user's input. If the analysis result is "happy," that emotion is reflected in the generation AI's response.

[1801] Step 5: Providing response results

[1802] The server sends the generated response to the terminal.

[1803] The device provides the received response to the user, either displaying it as text or converting it to speech and reading it aloud.

[1804] Input: Response text from the generation AI.

[1805] Output: The response (text or audio) provided to the user.

[1806] Specific behavior: The device receives the response "That's great!" and displays it as text or reads it aloud using the speech synthesis API.

[1807] Step 6: Scheduling and Reminders

[1808] Users input their future plans into the app.

[1809] The terminal transmits this schedule information to the server.

[1810] The server stores the schedule information in a database and sets the timing of reminder notifications.

[1811] Input: Schedule information (e.g., "Take your medicine tomorrow at 10 AM").

[1812] Output: Remind notification.

[1813] Specific operation: The server saves the schedule information in a database and manages the timing of reminders using a Cron job. When the reminder time arrives, it sends a notification to the device, informing the user that it is time to take their medicine.

[1814] Step 7: Health data collection and anomaly detection

[1815] Users enter their medication records and walking records into the app.

[1816] The terminal transmits this health data to the server.

[1817] The server stores the health data in a database and analyzes it using machine learning algorithms.

[1818] Input: Health data (e.g. walking records, blood pressure values).

[1819] Output: Anomaly detection notification.

[1820] How it works: The server analyzes health data using machine learning algorithms such as Scikit-learn. If an abnormality is detected, it sends a notification to the specified contacts (e.g., family members, medical institutions), and notifies the user via their device that an abnormality has been detected.

[1821] keyword

[1822] Generative AI model, prompt sentence

[1823] (Application example 2)

[1824] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1825] Conventional systems mainly provide simple responses and notifications to user inputs, but lack the ability to respond or suggest products that take into account the user's emotional and health states. Therefore, there is a need for more personalized and effective systems that contribute to users' lifestyles and health management. Furthermore, to increase customer satisfaction in virtual stores, a personalized shopping assistant that reflects the customer's current emotional and health states is necessary.

[1826] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1827] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for recognizing the user's emotional state and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying the user in the event of an abnormality, and means for suggesting products based on the user's health data and emotional state. This allows the user to receive personalized responses and product suggestions that take into account their emotional state and health condition at any given time, significantly improving the shopping experience in virtual stores.

[1828] "Means for receiving data" is a function for receiving text or voice data entered by a user and sending it to a server.

[1829] "Means for authenticating users" refers to a function that verifies the authentication information (user name and password) entered by the user and confirms the user's legitimacy.

[1830] "Generative artificial intelligence" is a technique that allows computers to generate appropriate responses to specific inputs, often using machine learning or deep learning models (e.g., GPT-3).

[1831] "Means for recognizing emotional states and reflecting them in responses" refers to technology that analyzes the user's input, recognizes the emotions (joy, sadness, anger, etc.) at the time, and reflects that emotional state in responses.

[1832] The "means for providing the response result to the user" is a function for conveying the generated response to the user in text or voice.

[1833] "Means for managing schedule information and sending reminder notifications" is a function that manages the schedules set by the user and sends notifications when the scheduled time approaches.

[1834] "Means for collecting and analyzing health data and notifying in the event of an abnormality" refers to a function that collects and analyzes health data entered by the user and notifies in the event of an abnormality.

[1835] The "means for suggesting products" is a function for suggesting optimal products based on the user's health data and emotional state.

[1836] To implement this invention, the following system must be constructed. First, the device used by the user is assumed to be a smartphone, smart glasses, or a head-mounted display (HMD). The user inputs data through this device, and the data is sent to a server.

[1837] Data Receipt and Authentication

[1838] The user launches the app using a smartphone or HMD and inputs data via voice or text. The user is authenticated by entering a username and password. The authentication information is sent to the server and authenticated. If authentication is successful, the user can use the system.

[1839] User Input and Response Generation

[1840] The server uses a generative AI model (e.g., GPT-3) to analyze the user's input data and generate an appropriate response. An emotion recognition engine is used to analyze the emotional state contained in the user's input and reflect it in the response. For example, if a user inputs "I'm feeling great today," the generative AI model will recognize that emotion and generate a response such as "That's great!"

[1841] Schedule management and reminder notifications

[1842] Users enter their schedule information into the app, and the server stores that information in a database. When it's time for a reminder, the server generates the reminder content and sends it to the user's device. For example, if you enter "Set a reminder to take my medicine tomorrow at 10 a.m.", a reminder will be sent to the user at the specified time.

[1843] Health data analysis and product proposals

[1844] The server collects the user's health data and analyzes it using machine learning algorithms. If an abnormality is detected, the server notifies the user and their designated contacts. The server also makes personalized product suggestions based on the user's health and emotional state. For example, if a user inputs "I feel a little tired today," the server will suggest "relaxation products" based on the user's low level of exercise.

[1845] Hardware and Software

[1846] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[1847] Software: Generative AI models (e.g., GPT-3), emotion recognition engines, health data analysis algorithms, product recommendation engines

[1848] Examples and prompts

[1849] For example, the following process occurs:

[1850] 1. The user puts on the HMD and speaks, "What products do you have available today?"

[1851] 2. The emotion recognition engine recognizes the emotion "happy."

[1852] 3. The generative AI generates a response: "We're having a special sale today!"

[1853] 4. The product suggestion engine suggests a "set of fresh vegetables."

[1854] 5. The HMD presents the generated responses and suggestions to the user visually and audibly.

[1855] An example prompt is:

[1856] User sentiment: Happy

[1857] Health data: Blood pressure: normal, Heart rate: normal, Exercise: low

[1858] User Input: What products do you have available today?

[1859] Response: We have a special sale today!

[1860] The system allows users to receive a personalized shopping experience that takes into account their emotional and health state at any given time.

[1861] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1862] Step 1:

[1863] The user starts up their smartphone or HMD and inputs input data (voice or text). For example, the user might say, "What products do you have available today?" The input data is converted into text data using voice recognition technology.

[1864] Input: User voice or text input

[1865] Data processing: Converting voice data into text using voice recognition technology

[1866] Output: Text data

[1867] Step 2:

[1868] The device sends the converted text data to the server, which authenticates the user if necessary. If authentication is successful, the server sends the input data to the generative AI model.

[1869] Input: Text data (user input)

[1870] Data Processing: User Credential Verification

[1871] Output: Authenticated text data, user ID

[1872] Step 3:

[1873] The server uses a generative AI model to analyze the text data and generate appropriate responses. It also uses an emotion recognition engine to recognize emotional states (e.g., happy, sad) from the input data and reflect them in the response.

[1874] Input: authenticated text data, user ID

[1875] Data processing: Response generation using a generative AI model, emotional state analysis using an emotion recognition engine

[1876] Output: The generated response text

[1877] Step 4:

[1878] The server stores the user's schedule information in a database and sets appropriate reminder times. For example, if a user inputs "Set a reminder to take my medicine at 10:00 AM tomorrow," the server generates the reminder content and sends it to the device when the time comes.

[1879] Input: Schedule information (e.g., time to take medicine)

[1880] Data processing: setting the timing of reminder notifications, generating reminder notification content

[1881] Output: Reminder notification

[1882] Step 5:

[1883] The server collects the user's health data and analyzes it using machine learning algorithms. For example, if the user's step count or blood pressure data is collected, the server analyzes the data to determine whether it is normal or not. If an abnormality is detected, the server sends a notification to the specified contacts (e.g., medical institutions).

[1884] Input: Health data (e.g., number of steps, blood pressure)

[1885] Data processing: Data analysis using machine learning algorithms

[1886] Output: Abnormality detection notification

[1887] Step 6:

[1888] The server then makes personalized product recommendations based on the user's emotional state and health data. For example, if a user inputs "I feel a little tired today" and health data indicates that they are not exercising much, the server will suggest "relaxation products."

[1889] Input: Emotional state, health data

[1890] Data processing: Analysis by product recommendation engine based on emotional state and health data

[1891] Output: Personalized product recommendations

[1892] Step 7:

[1893] The device provides the user with the responses, notifications, and product suggestions sent from the server visually and audibly, for example, by displaying the responses and product suggestions generated through the HMD and providing audio notifications.

[1894] Input: Generated response text, reminders, product suggestions

[1895] Data processing: Converting responses and notifications into visual and audio formats

[1896] Output: Visual and audio notification to the user

[1897] In this way, users can receive a shopping experience that is personalized to their emotional and health state at the time.

[1898] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1899] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1900] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1901] [Fourth embodiment]

[1902] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1903] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1904] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1905] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1906] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1907] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1908] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1909] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1910] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1911] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1912] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1913] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1914] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1915] The system of the present invention is constructed to allow users to perform various tasks related to daily life and health management. The functions of each component and the processing flow will be specifically explained below.

[1916] Receiving data entered by the user

[1917] User

[1918] Users use devices such as smartphones and tablets to input data into the app by text or voice, including schedules, health data, and everyday conversations.

[1919] Terminal

[1920] The device receives user input and sends it to the server as text data. If voice data is input, it is converted into text using voice recognition technology.

[1921] User authentication

[1922] User

[1923] When users start using the app, they enter their username and password.

[1924] Terminal

[1925] The terminal transmits these authentication data to the server.

[1926] server

[1927] The server accesses the database and performs authentication based on the user information. If authentication is successful, a session is started and an authentication token is sent to the terminal. If authentication fails, an error message is returned.

[1928] Response generation by generative artificial intelligence

[1929] server

[1930] The server uses a generative AI (e.g., GPT-4) to generate a response based on the user's input. The generated response is temporarily stored on the server.

[1931] Terminal

[1932] The terminal receives the generated response from the server and provides it to the user in text and audio.

[1933] Schedule management and reminder notifications

[1934] User

[1935] Users input their future plans into the app.

[1936] Terminal

[1937] This input is sent to the server.

[1938] server

[1939] The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, the server generates the corresponding notification.

[1940] Terminal

[1941] The device receives the reminder notification and notifies the user, for example, "A reminder to take your medicine has been set for 10:00 AM."

[1942] Health data collection and anomaly detection

[1943] User

[1944] Users enter their medication record and walking records into the app.

[1945] Terminal

[1946] The entered health data is sent to the server.

[1947] server

[1948] The server stores this data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, it notifies the specified contacts (e.g., family members, medical institutions).

[1949] Terminal

[1950] Reports detected anomalies to users and provides instructions on what to do.

[1951] Specific examples

[1952] 1. Set schedule reminders

[1953] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[1954] Device: Converts speech to text and sends it to the server.

[1955] Server: Stores reminders in a database and generates notifications at the scheduled times.

[1956] Device: Receive reminder notifications and notify the user.

[1957] 2. Detecting Anomalies in Health Data

[1958] User: Updates walking record and enters it into the app.

[1959] Terminal: Sends data to the server.

[1960] Server: Analyzes data, detects abnormalities, and sends relevant notifications to family members and medical institutions.

[1961] Device: Notify the user that an anomaly has been detected.

[1962] The system of the present invention makes it possible to streamline the user's lifestyle and health management and provide comprehensive daily support.

[1963] The processing flow will be explained below.

[1964] Step 1:

[1965] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[1966] Step 2:

[1967] Terminal: Sends the entered username and password to the server.

[1968] Step 3:

[1969] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[1970] Step 4:

[1971] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[1972] Step 5:

[1973] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[1974] Step 6:

[1975] Terminal: Sends user input to the server.

[1976] Step 7:

[1977] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[1978] Step 8:

[1979] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[1980] Step 9:

[1981] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[1982] Step 10:

[1983] Server: Generates a notification based on the reminder content.

[1984] Step 11:

[1985] Device: Receive reminder notifications and notify users via text and voice.

[1986] Step 12:

[1987] User: Enter new data from your medication record or walking log into the app.

[1988] Step 13:

[1989] Terminal: Sends the entered health data to the server.

[1990] Step 14:

[1991] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[1992] Step 15:

[1993] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[1994] Step 16:

[1995] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[1996] Step 17:

[1997] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[1998] Step 18:

[1999] User: Type everyday conversations and questions into the app or speak into the microphone.

[2000] Step 19:

[2001] Terminal: Sends user input or voice data to the server.

[2002] Step 20:

[2003] Server: When voice input is received, it converts it into text using speech recognition technology, and uses generative AI to generate appropriate responses to user questions and conversations.

[2004] Step 21:

[2005] Terminal: Receives the generated response from the server and provides it to the user in text and voice.

[2006] Example 1

[2007] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2008] In modern society, it is important for individual users to efficiently manage their daily lives and health, and there is a growing demand for systems that support this. However, conventional systems do not integrate various functions such as receiving user data, authenticating users, generating responses, managing schedules, and analyzing health data, which makes them difficult to use and makes it difficult to manage efficiently. Furthermore, there are issues with the recognition accuracy and response speed when using voice input.

[2009] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2010] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative artificial intelligence model, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying in the event of an abnormality, and means for converting voice input into text data using voice recognition technology. This enables the user to efficiently manage their daily life and health using a single system, and also improves the recognition accuracy and response speed for voice input.

[2011] The "means for receiving data entered by the user" is a function that receives information entered by the user via text or voice to the server via the terminal.

[2012] A "means of authenticating a user" is the process of identifying a user using authentication information, such as a username and password, and initiating a session on a server.

[2013] "Means for generating a response to user input using a generative artificial intelligence model" refers to a technology that analyzes input data from a user and generates an appropriate response using an artificial intelligence model.

[2014] The "means for providing a response result to a user" is a function that sends a response generated by a server to a terminal and allows the user to receive the response in text or voice.

[2015] "Means for managing the user's schedule information and sending reminder notifications" refers to a system that saves the schedule entered by the user and generates reminder notifications at pre-set times to notify the user.

[2016] "Means of collecting and analyzing user health data and notifying users in the event of abnormalities" refers to a function that stores the health information entered by users in a database, detects abnormalities using a machine learning algorithm, and notifies users of the results.

[2017] "Means for converting voice input into text data using voice recognition technology" refers to technology that converts data input by voice by a user into text data using voice recognition technology and transmits it to a server.

[2018] A "machine learning algorithm" is a mathematical model that analyzes large amounts of data to learn patterns and trends and make future predictions and decisions.

[2019] A "generative artificial intelligence model" refers to an algorithm that understands human natural language and generates appropriate responses based on that language, and is a fundamental technology for natural language processing.

[2020] A "database" is a storage system that centrally manages various data such as user information, schedule information, and health data, and allows efficient access.

[2021] The system of the present invention is designed to enable a user to efficiently perform tasks related to daily life and health management. Specific embodiments for carrying out the present invention will be described in detail below.

[2022] System Configuration

[2023] The system consists of three main components: users, devices, and servers. Users access the system using devices such as smartphones and tablets. The devices send input data from the users to the server, which processes the data, generates a response, and returns the result to the device.

[2024] Hardware and software used

[2025] Device: An input device such as a smartphone, tablet, or PC.

[2026] Server: Cloud server or on-premise server.

[2027] Database: A database (e.g., MySQL) for managing user information, schedule information, and health data.

[2028] Speech recognition technology: APIs for converting voice input into text (e.g., Google Speech-to-Text).

[2029] Text generation AI model: A generative artificial intelligence model (e.g., GPT-4).

[2030] Speech synthesis technology: APIs for converting text to speech (e.g., Google Text-to-Speech).

[2031] Machine learning algorithms: Algorithms for analyzing health data (e.g., TensorFlow).

[2032] Processing flow

[2033] 1. User authentication:

[2034] User: Enter your username and password on the login screen.

[2035] Terminal: Sends the entered authentication information to the server.

[2036] Server: Accesses the database and checks the authentication information. If authentication is successful, generates a session ID and authentication token and sends them to the device.

[2037] 2. Data entry and submission:

[2038] User: Enters data into the app via text or voice. For example, "Add a meeting for 3 PM."

[2039] Terminal: Converts speech into text and sends the data to the server.

[2040] 3. Response generation using generative artificial intelligence models:

[2041] Server: Sends user input data to the generative artificial intelligence model and generates a response, e.g., "A meeting has been scheduled for 3 PM."

[2042] 4. Response provision:

[2043] Server: Sends the generated response to the terminal.

[2044] Terminal: Provides a response to the user in text or voice.

[2045] 5. Schedule Management and Reminders:

[2046] User: Enters future events into the app. Example: "Set a reminder to take my medicine tomorrow at 10 AM."

[2047] Terminal: Sends input data to the server.

[2048] Server: Saves schedule data to a database and generates notifications at the reminder time.

[2049] Device: Receives notifications and notifies the user.

[2050] 6. Health data collection and anomaly detection:

[2051] User: Enter walking records and health data. Example: "Today's steps are 5000."

[2052] Terminal: Sends data to the server.

[2053] Server: Stores data in a database, analyzes it using machine learning algorithms, and sends notifications to configured contacts if an anomaly is detected.

[2054] Specific examples

[2055] Set schedule reminders:

[2056] User: Says, "Set a reminder to take my medicine tomorrow at 10 AM."

[2057] Device: Converts speech to text and sends it to the server.

[2058] Server: Stores reminders in a database and generates notifications at the scheduled times.

[2059] Device: Receive reminder notifications and notify the user.

[2060] Anomaly detection in health data:

[2061] User: Enters "5000 steps today."

[2062] Terminal: Sends data to the server.

[2063] Server: Analyzes the data and if an abnormality is detected, sends a notification to family members or medical institutions.

[2064] Device: Notify the user that an anomaly has been detected.

[2065] The system of the present invention allows users to efficiently manage their daily lives and health management on a single platform. In addition, the use of generative artificial intelligence models enables appropriate and prompt responses to user input.

[2066] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2067] Step 1: User authentication

[2068] ---

[2069] User:

[2070] Enter your username and password on the login screen. Input data: username (text), password (text).

[2071] Device:

[2072] Receives the entered authentication information and sends it to the server. Output data: Authentication information (JSON format).

[2073] server:

[2074] Accesses the database and checks the authentication information. If successful, generates a session ID and authentication token and sends them to the device. If unsuccessful, returns an error message.

[2075] Input data: Authentication information (JSON format)

[2076] Data calculation: User information matching and token generation

[2077] Output data: Session ID, authentication token (if successful) or error message (if unsuccessful)

[2078] Step 2: Enter and submit data

[2079] ---

[2080] User:

[2081] Enter data into the app by text or voice, for example, "Add a meeting for 3 PM."

[2082] Input data: text or audio data

[2083] Device:

[2084] For voice input, speech recognition technology is used to convert speech into text.

[2085] Input data: Audio data

[2086] Data Computing: Speech-to-Text Conversion

[2087] Output data: Text data

[2088] Device:

[2089] The converted text data or the directly entered text is sent to the server.

[2090] Output data: Input data (JSON format)

[2091] Step 3: Response generation using a generative artificial intelligence model

[2092] ---

[2093] server:

[2094] The input data is analyzed and applied to a generative artificial intelligence model. For example, the prompt sentence "Add a meeting at 3 PM" is input to the generative AI.

[2095] Input data: Text data (prompt)

[2096] Data Computation: Response Generation with Generative AI Models

[2097] Output data: Response text

[2098] server:

[2099] The generated response is temporarily stored and sent to the terminal.

[2100] Output data: Response data (JSON format)

[2101] Step 4: Provide a response

[2102] ---

[2103] Device:

[2104] The response data received from the server is displayed to the user. If audio is desired, a technology is used to convert text data into audio.

[2105] Input data: Response data (JSON format)

[2106] Data Computing: Text-to-Speech

[2107] Output data: Audio data

[2108] Device:

[2109] The response is displayed or played aloud to the user.

[2110] Output data: User presented

[2111] Step 5: Scheduling and Reminders

[2112] ---

[2113] User:

[2114] Enter future events into the app, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[2115] Input data: Schedule data (text)

[2116] Device:

[2117] Sends input data to the server.

[2118] Output data: Schedule data (JSON format)

[2119] server:

[2120] Save schedule data to a database and generate notifications at reminder times.

[2121] Input data: Schedule data

[2122] Data calculation: Schedule saving and reminder setting

[2123] Output data: Reminder notification data

[2124] Device:

[2125] Receive reminder notifications and notify users.

[2126] Input data: Reminder notification data

[2127] Output data: User presented

[2128] Step 6: Health data collection and anomaly detection

[2129] ---

[2130] User:

[2131] Enter your health data into the app. For example, enter "Today's steps are 5,000."

[2132] Input data: Health data (text)

[2133] Device:

[2134] Send the data to the server.

[2135] Output data: Health data (JSON format)

[2136] server:

[2137] Health data is stored in a database and analyzed using machine learning algorithms.

[2138] Input data: Health data

[2139] Data Computing: Data Analysis and Anomaly Detection

[2140] Output data: Analysis results (notification data in case of abnormality)

[2141] server:

[2142] If an abnormality is detected, a notification will be sent to the specified contacts (e.g., family members, medical institutions).

[2143] Output data: Error notification data

[2144] Device:

[2145] Notify the user that an abnormality has been detected and provide instructions on what action to take.

[2146] Input data: Error notification data

[2147] Output data: User presented

[2148] (Application example 1)

[2149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2150] Modern factories require systems to manage worker health and improve the safety of the work environment. Conventional systems struggle to monitor worker health data and work environment information in real time, and respond immediately when an abnormality is detected. Furthermore, they do not effectively manage individual work schedules or send reminder notifications. This can lead to reduced work efficiency and increased safety risks for workers. A system is needed to resolve these issues and improve work efficiency and safety within factories.

[2151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2152] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and sending a notification when an abnormality occurs, means for monitoring the worker's health data in real time and issuing a warning when an abnormality is detected, means for managing the work schedule and sending reminder notifications, and means for monitoring the temperature, hazardous gas concentration, etc. of the work environment and sending a notification when an abnormal value is detected. This makes it possible to monitor the worker's health condition in real time and take prompt action when an abnormality is detected, ensuring the safety of the work environment and improving work efficiency.

[2153] "Means for receiving data entered by a user" refers to means for receiving and processing text or voice data entered by a user using a device such as a smartphone or tablet.

[2154] "Means for authenticating users" refers to the means for authenticating that a user is a legitimate user using a user name and password.

[2155] "Means for generating responses to user input using generative artificial intelligence" means means for using a generative AI model (e.g., GPT-4) to generate appropriate responses based on input from a user.

[2156] The "means for providing a response result to a user" refers to a means for transmitting a response generated on a server to a user's terminal and presenting it in text or audio format.

[2157] "Means for managing user schedule information and sending reminder notifications" refers to means for saving and managing the plans and schedule information set by the user and sending reminder notifications at designated times.

[2158] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to the means for storing and analyzing health data entered by users on a server, and notifying users in the event of an abnormality.

[2159] "Means for monitoring workers' health data in real time and issuing warnings when abnormalities are detected" refers to means for collecting health data in real time from workers' smartwatches and healthcare devices and issuing warnings when abnormalities are detected.

[2160] The "means for managing work schedules and issuing reminder notifications" is a means for managing the work schedules of workers and issuing reminder notifications based on the set schedules.

[2161] "Means for monitoring the temperature, hazardous gas concentration, etc. of the working environment and issuing a notification if an abnormal value is detected" refers to a means for monitoring the temperature and hazardous gas concentration within a factory using sensors and issuing an immediate notification if an abnormal value is detected.

[2162] The system of this invention includes functions for receiving data entered by users, authenticating users, generating responses using artificial intelligence, providing response results, managing schedules and sending reminders, collecting and analyzing health data, and notifying users when an abnormality occurs. Furthermore, to adapt to a factory environment, the system adds functions for monitoring workers' health data in real time, issuing warnings when an abnormality is detected, and monitoring the work environment.

[2163] System Configuration

[2164] 1. User Device

[2165] Using a smartphone or tablet, the user inputs data by voice or text. The voice data is converted into text using voice recognition technology. The received data is sent to the server.

[2166] 2. Server

[2167] User authentication: The server accesses the database and performs authentication based on the username and password. If authentication is successful, a session is initiated and an authentication token is sent to the user's device. If authentication fails, an error message is returned.

[2168] Response generation using generative AI: The server uses a generative AI model (e.g., GPT-4) to generate a response based on user input. The generated response is temporarily stored on the server and then sent to the user's device.

[2169] Schedule management and reminder notifications: The server saves the entered schedule data in a database and sets the timing of reminder notifications. When the reminder time arrives, it generates a corresponding notification and sends it to the user's device.

[2170] Health data collection and analysis: Health data is collected in real time from workers' smartwatches and healthcare devices, and analyzed using machine learning algorithms. If an abnormality is detected, notification will be sent to designated contacts.

[2171] Work environment monitoring: Sensors are used to monitor the temperature and hazardous gas concentrations in the work environment, and if abnormal values ​​are detected, workers are immediately notified.

[2172] Work Schedule Management: Manage the work schedules of workers and send reminder notifications based on the set schedule.

[2173] Hardware and software used

[2174] Hardware: smartphones, tablets, smartwatches, sensors

[2175] Software: speech recognition technology, generative AI models (e.g., GPT-4), machine learning algorithms

[2176] Specific examples

[2177] 1. Set schedule reminders

[2178] User: Opens the app on their smartphone and says, "Set a reminder to take my medicine tomorrow at 10 AM."

[2179] Device: Converts speech to text and sends it to the server.

[2180] Server: Stores reminders in a database and generates notifications at the scheduled times.

[2181] On your device: Receive reminder notifications and notify the user.

[2182] 2. Detecting Anomalies in Health Data

[2183] User: Updates walking log and enters it into the app.

[2184] Terminal: Sends data to the server.

[2185] Server: Analyzes data, detects anomalies, and sends relevant notifications to family members and medical institutions.

[2186] Device: Notify the user that an anomaly has been detected.

[2187] Prompt Sentence Examples

[2188] Implement a program that collects health data for worker ID: 123456 and notifies if any abnormalities are found. The collected data is heart rate and number of steps, and a notification will be sent if the heart rate exceeds 100 or the number of steps is less than 500. A notification will also be sent if the temperature in the work environment exceeds 35 degrees or the concentration of harmful gases exceeds 50. The APIs to be used are as follows.

[2189] Health data collection: https: / / api.smartwatch.com / users / {user_id} / health

[2190] Speech recognition: https: / / api.speech-to-text.com / convert

[2191] Work schedule management: https: / / api.factory-robot.com / schedules

[2192] Environmental monitoring: https: / / api.factory-sensors.com / environment

[2193] Send notification: https: / / api.notification.com / send

[2194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2195] Step 1:

[2196] Receiving Data Input

[2197] User: Opens the app on their smartphone and enters voice or text data. For example, they might say, "Set a reminder to take my medicine tomorrow at 10 AM."

[2198] Input: Voice or text data

[2199] Terminal: When voice data is input, it is converted into text using voice recognition technology (e.g., API: https: / / api.speech-to-text.com / convert). The converted text data is sent to the server.

[2200] Output: User input as text data

[2201] Step 2:

[2202] User authentication

[2203] Terminal: Provides an interface for entering a username and password, where the user enters their credentials.

[2204] Input: Username and Password

[2205] Device: Sends authentication information to the server.

[2206] Server: Accesses the database and verifies the entered authentication information. If authentication is successful, starts a session, generates an authentication token, and sends it to the device. If authentication fails, generates an error message.

[2207] Output: Authentication token or error message

[2208] Step 3:

[2209] Response Generation

[2210] Server: Based on user input, generates appropriate responses using a generative AI model (e.g., GPT-4).

[2211] Input: User-entered text data

[2212] Server: Temporarily stores the generated response and sends it to the user's device.

[2213] Output: The text data of the generated response

[2214] Step 4:

[2215] Providing a response

[2216] Device: Displays or speaks to the user the generated response received from the server, for example, "A reminder to take your medicine has been set for tomorrow at 10 AM."

[2217] Input: The text data of the generated response

[2218] Terminal: Displaying text data in a user interface or playing it aloud.

[2219] Output: Providing a visual or audio response

[2220] Step 5:

[2221] Schedule management and reminder notifications

[2222] User: Enters an appointment. For example, "Take medicine at 10:00 AM tomorrow."

[2223] Input: User schedule input

[2224] Terminal: Sends schedule data to the server.

[2225] Server: Stores schedule data in a database and sets reminder notification timing.

[2226] Output: Schedule data stored in the database and reminder times

[2227] Step 6:

[2228] Real-time monitoring of health data

[2229] User: Wears a smartwatch or healthcare device to collect health data.

[2230] Input: Health data collected from smartwatches and healthcare devices

[2231] Device: Sends health data to the server.

[2232] Server: Analyzes health data using machine learning algorithms and generates alerts and notifies configured contacts if anomalies are detected.

[2233] Output: Health data analysis results and warning notifications in case of abnormalities

[2234] Step 7:

[2235] Work environment monitoring

[2236] Server: Use sensors to collect environmental data such as temperature and harmful gas concentrations in the factory (e.g., API: https: / / api.factory-sensors.com / environment).

[2237] Input: Environmental data from sensors

[2238] Server: Analyzes environmental data and immediately notifies workers if any abnormal values ​​are detected.

[2239] Output: Environmental data analysis results and notifications in case of abnormalities

[2240] Step 8:

[2241] Work schedule management and reminder notifications

[2242] User: Sets the work schedule, for example, "Start the next process at 10 o'clock."

[2243] Input: User work schedule input

[2244] Server: Stores work schedule data in a database and sets reminder notification timing. When the reminder time arrives, it generates a corresponding notification and sends it to the device.

[2245] Output: Work schedule data and reminder notifications

[2246] Through the above processing steps, this system can realize safety management of workers at the factory site and improvement of work efficiency.

[2247] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2248] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, integrating functions such as user input, authentication, AI response, reminder notifications, health data analysis, emotion recognition, etc. The following describes in detail the function of each component and its processing flow.

[2249] Receiving data entered by the user

[2250] User

[2251] Users use their smartphone or tablet to launch the app and enter data via text or voice, including schedules, health data, daily communication, and emotional state.

[2252] Terminal

[2253] The device receives user input and transmits it to the server as text and voice data, and uses voice recognition technology to convert the voice data into text.

[2254] User authentication

[2255] User

[2256] When users start using the app, they enter their username and password.

[2257] Terminal

[2258] The terminal transmits these authentication data to the server.

[2259] server

[2260] The server accesses the database and performs authentication based on the user information. If authentication is successful, it generates a user ID and starts a session. It sends an authentication token to the terminal, and if authentication fails, it returns an error message.

[2261] Response generation by generative artificial intelligence

[2262] server

[2263] The server uses generative AI to analyze user input and generate appropriate responses, and an emotion engine to recognize the user's emotional state and reflect it in the response.

[2264] Terminal

[2265] The terminal receives the generated response from the server and provides it to the user in text and audio.

[2266] Emotion recognition by emotion engine

[2267] server

[2268] The emotion engine analyzes user input data (text and voice) and recognizes the user's emotional state, for example, identifying whether the user's input contains emotions such as joy, sadness, or anger.

[2269] server

[2270] Based on the recognized emotions, the generative AI adjusts its response and provides appropriate feedback to the user.

[2271] Schedule management and reminder notifications

[2272] User

[2273] Users input their future plans into the app.

[2274] Terminal

[2275] The entered schedule information is sent to the server.

[2276] server

[2277] The server stores the schedule information in a database and sets the timing of reminder notifications. When the time comes, the server generates the reminder content.

[2278] Terminal

[2279] The device will receive reminder notifications and notify the user via text and voice.

[2280] Health data collection and anomaly detection

[2281] User

[2282] Users enter their medication record and walking records into the app.

[2283] Terminal

[2284] The entered health data is sent to the server.

[2285] server

[2286] The server stores health data in a database and analyzes it using machine learning algorithms. If an abnormality is detected, a notification is sent to the specified contacts (e.g., family members, medical institutions).

[2287] Terminal

[2288] Notify users when data is updated and alert them if anomalies are detected.

[2289] Specific examples

[2290] 1. Set schedule reminders

[2291] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[2292] Terminal: Sends input to the server.

[2293] Server: The generating AI understands the reminders and stores them in a database.

[2294] Device: Notify that a reminder has been set.

[2295] 2. Use of Emotion Recognition

[2296] User: In everyday conversation, type "I'm very happy today."

[2297] Terminal: Sends input to the server.

[2298] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[2299] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[2300] Terminal: Provides generated responses to the user in text and audio.

[2301] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[2302] The processing flow will be explained below.

[2303] Step 1:

[2304] User: Launches the app using a device such as a smartphone or tablet and enters their username and password.

[2305] Step 2:

[2306] Terminal: Sends the entered username and password to the server.

[2307] Step 3:

[2308] Server: Accesses the database and performs authentication based on user information. If authentication is successful, generates a user ID and starts a session. Sends an authentication token to the terminal, and if authentication fails, returns an error message.

[2309] Step 4:

[2310] Terminal: Receives the authentication result and displays the home screen if authentication is successful, or displays an error message on the login screen if authentication is unsuccessful.

[2311] Step 5:

[2312] User: In the app's chat-style input field, types, "Set a reminder to take my medicine tomorrow at 10 AM."

[2313] Step 6:

[2314] Terminal: Sends user input to the server.

[2315] Step 7:

[2316] Server: Analyzes the user's input using the generated AI and understands that it is a request to set a reminder. Saves the reminder information (user ID, time, message) in the database.

[2317] Step 8:

[2318] On the device: Notify the user that a reminder has been set (e.g., "Reminder set").

[2319] Step 9:

[2320] Server: Periodically checks the database for reminder information and retrieves reminders that should be sent when it is time.

[2321] Step 10:

[2322] Server: Generates a notification based on the reminder content.

[2323] Step 11:

[2324] Device: Receive reminder notifications and notify users via text and voice.

[2325] Step 12:

[2326] User: Enter new data from your medication record or walking log into the app.

[2327] Step 13:

[2328] Terminal: Sends the entered health data to the server.

[2329] Step 14:

[2330] Server: Stores new health data in a database and analyzes it using machine learning algorithms.

[2331] Step 15:

[2332] Server: If an abnormality is detected, a notification is sent to pre-defined contacts (e.g., family members, medical institutions).

[2333] Step 16:

[2334] On the device: Inform the user that their data has been updated and also warn them if an anomaly is detected.

[2335] Step 17:

[2336] Family / healthcare provider: Receive notifications, check the user's condition if necessary, and take appropriate measures.

[2337] Processing flow incorporating emotion recognition

[2338] Step 18:

[2339] User: Type everyday conversations and questions into the app or speak into the microphone.

[2340] Step 19:

[2341] Terminal: Sends user input or voice data to the server.

[2342] Step 20:

[2343] Server: When voice input is received, the voice data is converted into text data using voice recognition technology.

[2344] Step 21:

[2345] Server: Uses an emotion engine to recognize emotions from the user's text or voice data. For example, the emotion engine identifies emotions such as "happy," "sad," and "angry."

[2346] Step 22:

[2347] Server: Based on the recognized emotion, the generative AI adjusts and generates an appropriate response. For example, if the user inputs "I'm happy," the generative AI will generate a response such as "That's great!"

[2348] Step 23:

[2349] Terminal: Receives the generated response and provides it to the user in text and voice.

[2350] Specific examples

[2351] 1. Set schedule reminders

[2352] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[2353] Terminal: Sends input to the server.

[2354] Server: The generating AI understands the reminders and stores them in a database.

[2355] Device: Notify that a reminder has been set.

[2356] 2. Use of Emotion Recognition

[2357] User: In everyday conversation, type "I'm very happy today."

[2358] Terminal: Sends input to the server.

[2359] Server: The generation AI analyzes the input, and the emotion engine recognizes the emotion "happy."

[2360] Server: The generative AI generates an appropriate response based on the user's emotions (e.g., "That's great!").

[2361] Terminal: Provides generated responses to the user in text and audio.

[2362] The system of the present invention streamlines the user's daily life and health management, and provides more natural and effective communication with the user by recognizing emotions and responding appropriately.

[2363] Example 2

[2364] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2365] In today's multifunctional digital lifestyle, systems that integrate multiple functions are required to support users' efficient lifestyle and health management. However, conventional systems often provide separate functions for user input data processing, authentication, generative AI responses, emotion recognition, schedule management, and health data analysis, making integrated management difficult. Furthermore, it is difficult to seamlessly integrate these functions, which can result in a poor user experience. Furthermore, systems are unable to properly recognize the user's emotional state and reflect it in responses, resulting in unnatural communication.

[2366] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2367] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using a generative AI model, means for recognizing the user's emotional state using an emotion engine and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, and means for collecting and analyzing the user's health data and notifying in the event of an abnormality. This enables efficient lifestyle and health management for the user and realizes natural communication through emotion recognition.

[2368] "Means for receiving data entered by a user" refers to a device or program that allows a user to enter data in text or voice format via a smartphone or tablet and receive it.

[2369] A "means for authenticating a user" is a device or program that verifies a user's identity and grants appropriate access rights based on authentication information such as a user name and password.

[2370] A "means for generating a response to a user's input using generative artificial intelligence" is a device or program that uses a generative AI model to create an appropriate response based on the user's input data.

[2371] "Means for recognizing a user's emotional state using an emotion engine and reflecting that in responses" refers to a device or program for analyzing a user's input data and identifying emotions, and a mechanism for adjusting responses based on those emotions.

[2372] The "means for providing a response result to a user" refers to a device or program for displaying or audibly providing the generated response to a user.

[2373] The "means for managing the user's schedule information and sending reminder notifications" refers to a device or program that records the schedule information entered by the user and sends reminder notifications at appropriate times.

[2374] "Means for collecting and analyzing user health data and notifying users in the event of an abnormality" refers to a device or program that collects health-related data entered by users, analyzes it using a machine learning algorithm, and notifies users in the event of an abnormality.

[2375] A "machine learning algorithm" is an algorithm that uses mathematical and statistical techniques to predict future data patterns or classify data based on past data.

[2376] "Speech recognition technology" is a technology for analyzing voice data and converting it into corresponding text data.

[2377] The system of the present invention is a multi-functional system designed to support users' daily lives and health management, and integrates functions such as user input, authentication, AI response, reminder notification, health data analysis, emotion recognition, etc. A specific embodiment of this system will be described.

[2378] Receiving data entered by the user

[2379] Users use a smartphone or tablet to launch the app and input data via text or voice. The input content includes schedules, health data, daily communication, and emotional state. The device receives this input data and sends it to the server as text and voice data. Voice recognition technology (e.g., a voice recognition API) is used to convert the voice data into text.

[2380] User authentication

[2381] When a user starts using an app, they enter their username and password. The device sends this authentication data to the server. The server accesses a database (for example, an SQL database) and performs authentication based on the user information. If authentication is successful, a token is generated and sent to the device. This allows the user to use the system securely.

[2382] Response generation by generative artificial intelligence

[2383] The server uses a generative AI model to analyze the user's input and generate an appropriate response. It uses an emotion engine to recognize the user's emotional state and reflect it in the response. In this case, the generative AI model uses natural language processing technology, for example. The server sends the generated response to the device, which then provides it to the user in text and voice.

[2384] Emotion recognition by emotion engine

[2385] The server uses an emotion engine to analyze the user's input data (text and voice) to recognize their emotional state. Specifically, it identifies emotions such as joy, sadness, and anger based on the input data. As a result, the generative AI model adjusts the response content based on the recognized emotion and provides feedback to the user.

[2386] Schedule management and reminder notifications

[2387] The user enters future plans into the app. The device sends the entered schedule information to the server. The server saves the schedule information in a database and sets the timing for reminder notifications. When the reminder time arrives, the server generates the reminder content. The device notifies the user of the reminder via text and voice.

[2388] Health data collection and anomaly detection

[2389] Users enter health data such as their medication record and walking records into the app. The device then sends this health data to the server. The server then analyzes the health data stored in the database and uses machine learning algorithms to detect abnormalities. If an abnormality is detected, a notification is sent to set contacts (e.g., family members, medical institutions) and the user is also notified via the device.

[2390] Specific examples

[2391] 1. Set schedule reminders

[2392] User: Opens the app on their phone and types, "Set a reminder to take my medicine tomorrow at 10 AM."

[2393] Terminal: Sends input to the server.

[2394] Server: The generative AI model understands the reminders and stores them in a database.

[2395] Device: Notify that a reminder has been set.

[2396] 2. Use of Emotion Recognition

[2397] User: In everyday conversation, type "I'm very happy today."

[2398] Terminal: Sends input to the server.

[2399] Server: The generative AI model analyzes the input, and the emotion engine recognizes the emotion "happy."

[2400] Server: The generative AI model generates an appropriate response based on the user's emotions (e.g., "That's great!").

[2401] Terminal: Provides generated responses to the user in text and audio.

[2402] The system of the present invention streamlines users' daily lives and health management, and provides more natural and effective communication with users by recognizing emotions and responding appropriately. Furthermore, by utilizing natural language processing technology and machine learning algorithms, the system is capable of advanced data analysis and response generation, flexibly responding to diverse user needs.

[2403] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2404] Program processing flow

[2405] Step 1: User enters data

[2406] A user launches the app using a smartphone or tablet, then enters data via text or voice, for example, "Set a reminder to take my medicine tomorrow at 10 AM."

[2407] Input: User text or voice data.

[2408] Output: Text data or audio file.

[2409] Specific operation: In the case of voice input, the voice data is converted into text using a speech recognition API. The text data generated is "Set a reminder to take my medicine tomorrow at 10 AM."

[2410] Step 2: User authentication

[2411] The user enters their username and password into the app's login screen.

[2412] The terminal transmits these authentication data to the server.

[2413] The server compares the received authentication data with a database and authenticates the user.

[2414] Input: Username and Password.

[2415] Output: An authentication token or an error message.

[2416] Specific operation: The server accesses a database such as MySQL and verifies whether the username "user123" and password "password123" are correct. If authentication is successful, it generates a JWT token and sends it to the device. If authentication fails, it returns an error message.

[2417] Step 3: Generative AI generates a response

[2418] The server receives the user's text input and sends prompt sentences to a generative AI model (e.g., a natural language processing model) for analysis.

[2419] Input: User input data (e.g., "I'm very happy today").

[2420] Output: The generated response text (e.g., "That's great!").

[2421] Specific operation: Send a prompt to the generative AI model saying, "The user feels happy. Please generate an appropriate response." Receive the generated text response.

[2422] Step 4: Emotion Recognition with the Emotion Engine

[2423] The server sends the user's input data to an emotion engine (e.g., an emotion analysis API) to identify the emotional state.

[2424] Input: User input data (e.g., "I'm very happy today").

[2425] Output: Emotional state (e.g., "happy").

[2426] Specific operation: The server uses the emotion analysis API to recognize the emotion from the user's input. If the analysis result is "happy," that emotion is reflected in the generation AI's response.

[2427] Step 5: Providing response results

[2428] The server sends the generated response to the terminal.

[2429] The device provides the received response to the user, either displaying it as text or converting it to speech and reading it aloud.

[2430] Input: Response text from the generation AI.

[2431] Output: The response (text or audio) provided to the user.

[2432] Specific behavior: The device receives the response "That's great!" and displays it as text or reads it aloud using the speech synthesis API.

[2433] Step 6: Scheduling and Reminders

[2434] Users input their future plans into the app.

[2435] The terminal transmits this schedule information to the server.

[2436] The server stores the schedule information in a database and sets the timing of reminder notifications.

[2437] Input: Schedule information (e.g., "Take your medicine tomorrow at 10 AM").

[2438] Output: Remind notification.

[2439] Specific operation: The server saves the schedule information in a database and manages the timing of reminders using a Cron job. When the reminder time arrives, it sends a notification to the device, informing the user that it is time to take their medicine.

[2440] Step 7: Health data collection and anomaly detection

[2441] Users enter their medication records and walking records into the app.

[2442] The terminal transmits this health data to the server.

[2443] The server stores the health data in a database and analyzes it using machine learning algorithms.

[2444] Input: Health data (e.g. walking records, blood pressure values).

[2445] Output: Anomaly detection notification.

[2446] How it works: The server analyzes health data using machine learning algorithms such as Scikit-learn. If an abnormality is detected, it sends a notification to the specified contacts (e.g., family members, medical institutions), and notifies the user via their device that an abnormality has been detected.

[2447] keyword

[2448] Generative AI model, prompt sentence

[2449] (Application example 2)

[2450] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2451] Conventional systems mainly provide simple responses and notifications to user inputs, but lack the ability to respond or suggest products that take into account the user's emotional and health states. Therefore, there is a need for more personalized and effective systems that contribute to users' lifestyles and health management. Furthermore, to increase customer satisfaction in virtual stores, a personalized shopping assistant that reflects the customer's current emotional and health states is necessary.

[2452] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2453] In this invention, the server includes means for receiving data entered by the user, means for authenticating the user, means for generating a response to the user's input using generative artificial intelligence, means for recognizing the user's emotional state and reflecting it in the response, means for providing the response result to the user, means for managing the user's schedule information and sending reminder notifications, means for collecting and analyzing the user's health data and notifying the user in the event of an abnormality, and means for suggesting products based on the user's health data and emotional state. This allows the user to receive personalized responses and product suggestions that take into account their emotional state and health condition at any given time, significantly improving the shopping experience in virtual stores.

[2454] "Means for receiving data" is a function for receiving text or voice data entered by a user and sending it to a server.

[2455] "Means for authenticating users" refers to a function that verifies the authentication information (user name and password) entered by the user and confirms the user's legitimacy.

[2456] "Generative artificial intelligence" is a technique that allows computers to generate appropriate responses to specific inputs, often using machine learning or deep learning models (e.g., GPT-3).

[2457] "Means for recognizing emotional states and reflecting them in responses" refers to technology that analyzes the user's input, recognizes the emotions (joy, sadness, anger, etc.) at the time, and reflects that emotional state in responses.

[2458] The "means for providing the response result to the user" is a function for conveying the generated response to the user in text or voice.

[2459] "Means for managing schedule information and sending reminder notifications" is a function that manages the schedules set by the user and sends notifications when the scheduled time approaches.

[2460] "Means for collecting and analyzing health data and notifying in the event of an abnormality" refers to a function that collects and analyzes health data entered by the user and notifies in the event of an abnormality.

[2461] The "means for suggesting products" is a function for suggesting optimal products based on the user's health data and emotional state.

[2462] To implement this invention, the following system must be constructed. First, the device used by the user is assumed to be a smartphone, smart glasses, or a head-mounted display (HMD). The user inputs data through this device, and the data is sent to a server.

[2463] Data Receipt and Authentication

[2464] The user launches the app using a smartphone or HMD and inputs data via voice or text. The user is authenticated by entering a username and password. The authentication information is sent to the server and authenticated. If authentication is successful, the user can use the system.

[2465] User Input and Response Generation

[2466] The server uses a generative AI model (e.g., GPT-3) to analyze the user's input data and generate an appropriate response. An emotion recognition engine is used to analyze the emotional state contained in the user's input and reflect it in the response. For example, if a user inputs "I'm feeling great today," the generative AI model will recognize that emotion and generate a response such as "That's great!"

[2467] Schedule management and reminder notifications

[2468] Users enter their schedule information into the app, and the server stores that information in a database. When it's time for a reminder, the server generates the reminder content and sends it to the user's device. For example, if you enter "Set a reminder to take my medicine tomorrow at 10 a.m.", a reminder will be sent to the user at the specified time.

[2469] Health data analysis and product proposals

[2470] The server collects the user's health data and analyzes it using machine learning algorithms. If an abnormality is detected, the server notifies the user and their designated contacts. The server also makes personalized product suggestions based on the user's health and emotional state. For example, if a user inputs "I feel a little tired today," the server will suggest "relaxation products" based on the user's low level of exercise.

[2471] Hardware and Software

[2472] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[2473] Software: Generative AI models (e.g., GPT-3), emotion recognition engines, health data analysis algorithms, product recommendation engines

[2474] Examples and prompts

[2475] For example, the following process occurs:

[2476] 1. The user puts on the HMD and speaks, "What products do you have available today?"

[2477] 2. The emotion recognition engine recognizes the emotion "happy."

[2478] 3. The generative AI generates a response: "We're having a special sale today!"

[2479] 4. The product suggestion engine suggests a "set of fresh vegetables."

[2480] 5. The HMD presents the generated responses and suggestions to the user visually and audibly.

[2481] An example prompt is:

[2482] User sentiment: Happy

[2483] Health data: Blood pressure: normal, Heart rate: normal, Exercise: low

[2484] User Input: What products do you have available today?

[2485] Response: We have a special sale today!

[2486] The system allows users to receive a personalized shopping experience that takes into account their emotional and health state at any given time.

[2487] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2488] Step 1:

[2489] The user starts up their smartphone or HMD and inputs input data (voice or text). For example, the user might say, "What products do you have available today?" The input data is converted into text data using voice recognition technology.

[2490] Input: User voice or text input

[2491] Data processing: Converting voice data into text using voice recognition technology

[2492] Output: Text data

[2493] Step 2:

[2494] The device sends the converted text data to the server, which authenticates the user if necessary. If authentication is successful, the server sends the input data to the generative AI model.

[2495] Input: Text data (user input)

[2496] Data Processing: User Credential Verification

[2497] Output: Authenticated text data, user ID

[2498] Step 3:

[2499] The server uses a generative AI model to analyze the text data and generate appropriate responses. It also uses an emotion recognition engine to recognize emotional states (e.g., happy, sad) from the input data and reflect them in the response.

[2500] Input: authenticated text data, user ID

[2501] Data processing: Response generation using a generative AI model, emotional state analysis using an emotion recognition engine

[2502] Output: The generated response text

[2503] Step 4:

[2504] The server stores the user's schedule information in a database and sets appropriate reminder times. For example, if a user inputs "Set a reminder to take my medicine at 10:00 AM tomorrow," the server generates the reminder content and sends it to the device when the time comes.

[2505] Input: Schedule information (e.g., time to take medicine)

[2506] Data processing: setting the timing of reminder notifications, generating reminder notification content

[2507] Output: Reminder notification

[2508] Step 5:

[2509] The server collects the user's health data and analyzes it using machine learning algorithms. For example, if the user's step count or blood pressure data is collected, the server analyzes the data to determine whether it is normal or not. If an abnormality is detected, the server sends a notification to the specified contacts (e.g., medical institutions).

[2510] Input: Health data (e.g., number of steps, blood pressure)

[2511] Data processing: Data analysis using machine learning algorithms

[2512] Output: Abnormality detection notification

[2513] Step 6:

[2514] The server then makes personalized product recommendations based on the user's emotional state and health data. For example, if a user inputs "I feel a little tired today" and health data indicates that they are not exercising much, the server will suggest "relaxation products."

[2515] Input: Emotional state, health data

[2516] Data processing: Analysis by product recommendation engine based on emotional state and health data

[2517] Output: Personalized product recommendations

[2518] Step 7:

[2519] The device provides the user with the responses, notifications, and product suggestions sent from the server visually and audibly, for example, by displaying the responses and product suggestions generated through the HMD and providing audio notifications.

[2520] Input: Generated response text, reminders, product suggestions

[2521] Data processing: Converting responses and notifications into visual and audio formats

[2522] Output: Visual and audio notification to the user

[2523] In this way, users can receive a shopping experience that is personalized to their emotional and health state at the time.

[2524] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2525] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2526] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2527] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2528] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2529] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2530] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2531] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2532] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2533] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2534] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Serv...

Claims

1. means for receiving user input data; a means for authenticating a user; means for generating a response to a user input using generative artificial intelligence; a means for providing the response results to the user; A means to manage the user's schedule information and send reminder notifications; A means of collecting and analyzing user health data and notifying users in the event of an abnormality; A system including:

2. The system of claim 1, wherein a machine learning algorithm is used in analyzing the health data.

3. 10. The system of claim 1, further comprising means for receiving the user's input as voice data and text data and converting the voice data into text data using voice recognition techniques.

Images