System

Abstract

A system is provided.SOLUTION: A system comprising: means for communicating with a person to be cared for using a generative AI technique; means for analyzing an acquired image; means for transmitting the text data and an analysis result to a caregiver or a medical facility; and means for issuing an alert when an abnormality is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many countries, including Japan, are facing a rapidly aging society with a declining birthrate, resulting in an increasing number of elderly people requiring care. At the same time, difficulties in recruiting caregivers and rising turnover rates mean a serious labor shortage in the care field. This problem leads to a decline in the quality of care services and increases the burden on both care recipients and caregivers. Furthermore, the traditional care system does not adequately provide care tailored to individual health conditions and emotions. Therefore, a new system is needed to reduce the workload and improve the quality of care services. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. Specifically, the present invention provides a system including a means for communicating with a care recipient using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for transmitting the text data and analysis results to a caregiver or a medical institution, and a means for issuing an alert when an abnormality is detected. The present invention also provides a system further including a means for generating feedback for the care recipient using generative AI technology and a means for providing the feedback in the form of audio or visual data. The present invention also provides a system including a means for analyzing big data and proposing an optimized care plan for the care recipient, and a means for sharing the care plan with a caregiver or a medical institution. In this way, a system is constructed that can improve the quality of care services, reduce the burden on caregivers, and achieve higher satisfaction.

[0006] "Generative AI technology" is a type of artificial intelligence technology that utilizes functions such as natural language processing and image recognition to interact with users and interpret and generate data.

[0007] "Care recipients" refers to elderly people and people with disabilities who need nursing care services, and primarily those who need assistance with daily life.

[0008] "Voice data" refers to voice data collected through a voice input device such as a microphone, and mainly includes speech information from the user.

[0009] "Text data" refers to text information converted from voice data using voice recognition technology, and is data in sentence format that can be used by the generation AI for analysis.

[0010] "Image data" refers to visual information collected through a camera or other image capture device, including facial expressions and movements of the care recipient.

[0011] "Analysis" is the process of processing collected data (audio data and image data) and extracting information such as health status and emotions.

[0012] "Caregiver" refers to people who play a supporting role in those receiving care, particularly professional care workers working in care facilities or homes.

[0013] "Medical institution" refers to a hospital, clinic, or other facility providing medical services, and refers to a place for the health management and treatment of care recipients.

[0014] An "alert" refers to a warning message sent when an abnormality or emergency occurs, and is a means of notifying caregivers and medical institutions when a prompt response is required.

[0015] "Feedback" refers to response information such as advice or warnings to the care recipient, and is provided in audio or visual form using generative AI technology.

[0016] "Big data" refers to large and diverse data sets, and includes technologies for discovering new insights and trends by analyzing these data sets.

[0017] A "care plan" refers to an individual care plan created based on the health and living situation of the person receiving care, and is a document that details the content and goals of daily care. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system that uses generative AI technology to provide personalized care services to individuals. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement appropriate care plans.

[0040] System Configuration

[0041] The system includes the following major components:

[0042] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[0043] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[0044] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[0045] 4. Database: Stores the care recipient's profile, health status, and history data.

[0046] 5. Feedback module: Generates advice and feedback for the care recipient.

[0047] 6. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[0048] 7. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[0049] Program processing

[0050] 1. User registration and profile creation

[0051] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[0052] Terminal: Sends the entered information to the server.

[0053] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0054] 2. Start a conversation session with the generative AI

[0055] User: Launches the application and begins interacting with the generative AI.

[0056] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[0057] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[0058] 3. Acquisition of audio and image data

[0059] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[0060] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data. The captured image data is passed to the image analysis module, where facial expressions and movements are analyzed.

[0061] 4. Data analysis and feedback

[0062] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[0063] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data and image analysis results.

[0064] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[0065] Device: Provides audio and visual feedback to the care recipient.

[0066] 5. Alert Notifications and Continuous Monitoring

[0067] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0068] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[0069] 6. Big data analysis and care plan proposals

[0070] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0071] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0072] Specific examples

[0073] 1. User registration and profile creation

[0074] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[0075] 2. Start a conversation session with the generative AI

[0076] User: Launches the application and begins a dialogue with the AI ​​generator. The AI ​​generator asks, "How are you feeling today?", to which the user responds, "I'm not feeling too well."

[0077] 3. Acquisition of audio and image data

[0078] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[0079] 4. Data analysis and feedback

[0080] Device: The text data and image analysis results are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[0081] 5. Alert Notifications and Continuous Monitoring

[0082] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[0083] 6. Big data analysis and care plan proposals

[0084] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve the plan.

[0085] The above is a description of the mode for carrying out the invention. It is expected that this system will provide individually optimized care and reduce the burden on caregivers.

[0086] The processing flow will be explained below.

[0087] Step 1: User Registration

[0088] User: Installs the application on the tablet device and launches it.

[0089] Device: Displays a registration screen and prompts the user to create an account.

[0090] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[0091] Terminal: Sends the entered information to the server.

[0092] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0093] Step 2: Create your profile

[0094] Device: Displays a profile creation screen and prompts the user to enter personal information.

[0095] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[0096] Terminal: Sends the entered information to the server.

[0097] Server: The received information is linked to the user account and stored in a database.

[0098] Step 3: Start interacting with generative AI

[0099] User: Launches the application and begins an interactive session with the generative AI.

[0100] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[0101] Step 4: Acquiring audio and visual data

[0102] User: Tells the generated AI about their physical condition, wishes, etc.

[0103] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[0104] Step 5: Convert audio data to text

[0105] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[0106] Step 6: Analyzing the image data

[0107] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[0108] Step 7: Send data to the server

[0109] Terminal: Sends text data generated by voice recognition and image analysis results to the server.

[0110] Step 8: Analysis by generative AI

[0111] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on the text data and image analysis results.

[0112] Server: Stores the evaluation results in a database.

[0113] Step 9: Generate and provide feedback

[0114] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[0115] Server: Sends the generated feedback to the device.

[0116] Device: Provides audio and visual feedback to the care recipient.

[0117] Step 10: Alert Notification

[0118] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0119] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[0120] Step 11: Continuous monitoring and data collection

[0121] Server: Continuously collects data from the care recipient and stores it as a log.

[0122] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[0123] Step 12: Big data analysis and care plan proposal

[0124] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0125] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] In an aging society, there is a need to improve the quality of nursing care services, provide personalized care that meets the individual needs of care recipients, and reduce the burden on caregivers and medical institutions. However, conventional systems have difficulty accurately understanding the health and emotional state of care recipients in real time and providing appropriate feedback and care plans. As a result, there has been a lack of efficient means to improve the quality of nursing care services.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes: means for communicating with the care recipient using AI technology; means for converting acquired voice data into text data; means for analyzing acquired image data; means for transmitting the text data and analysis results; means for issuing an alert when an abnormality is detected; means for a user to input voice and image data; means for transmitting the voice and image data to the server; means for generating feedback for the care recipient using AI technology; means for providing the feedback in audio or visual form; means for generating feedback based on data received by the AI ​​from the server; means for transmitting the feedback from the server to a terminal; means for analyzing big data and proposing an optimized care plan for the care recipient; means for sharing the care plan with a caregiver or a medical institution; means for analyzing the acquired big data; means for generating a care plan based on the analysis results; and means for notifying the caregiver or a medical institution of the care plan from the server. This enables real-time monitoring of the condition of the care recipient and providing appropriate feedback and a personalized care plan.

[0131] "Generative AI technology" is a technology that uses natural language processing and machine learning algorithms to engage in natural dialogue with humans and generate appropriate responses and information.

[0132] "Care recipients" refers to elderly people or individuals with physical disabilities who require nursing or medical assistance.

[0133] "Means of communication" refers to the method by which the system has the functionality to exchange information with the care recipient via voice or text.

[0134] "Audio data" refers to information that has been digitally recorded from the voice of the person receiving care.

[0135] "Text data" refers to information in which voice data is expressed as a string of characters.

[0136] The term "means for converting voice data into text data" refers to a method for converting voice data into corresponding text data using voice recognition technology.

[0137] "Image data" refers to information that digitally records the facial expressions and movements of the person receiving care.

[0138] "Means for analyzing image data" refers to a method of using image processing technology to recognize the facial expressions and movements of the person being cared for and analyze their meaning.

[0139] "Means for transmitting text data and analysis results" refers to the communication function for transmitting the text data and image analysis results acquired by the system to a remote server or other device.

[0140] "Means for sending an alert" refers to a method that has the function of sending a notification when an abnormality is detected and warning caregivers and medical institutions.

[0141] "Means for inputting voice and image data" refers to a method having the function of capturing voice and image data from the care recipient and inputting them into the system.

[0142] "Means by which the generating AI generates feedback based on data received from the server" refers to a method by which the generating AI creates appropriate feedback for the care recipient based on data received from the server.

[0143] "Means for providing audio or visual feedback" refers to a method that has the function of providing the generated feedback to the care recipient as audio and images.

[0144] "Means of analyzing big data" refers to methods that use technology to analyze large amounts of data and identify patterns and trends.

[0145] "Means for proposing an optimized care plan" refers to a method that has the function of creating and proposing an optimal care plan for a care recipient based on the analysis results.

[0146] "Means for sharing a care plan with a caregiver or a medical institution" refers to a method that has a function for sharing a generated care plan with a caregiver or a medical institution through communication.

[0147] A "server" is a computing device that serves as the core of a system and plays a central role in storing, analyzing, and transmitting data.

[0148] "Terminal" refers to a device (e.g., tablet, smartphone) that a user directly operates to communicate.

[0149] These definitions provide a clear understanding of the technical scope and function of the invention and serve as a basis for patent prosecution and enforcement.

[0150] This invention relates to a system that provides personalized care services to individuals using generative AI technology. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement an appropriate care plan. The system includes the following main components:

[0151] System Configuration

[0152] 1. Generative AI Engine: This is the main software component for natural dialogue with the care recipient. The generative AI engine uses natural language processing and machine learning algorithms.

[0153] 2. Speech recognition module: This module recognizes the voice of the care recipient and converts it into text data. Specifically, it uses the Google Cloud Speech-to-Text API.

[0154] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses OpenCV, TensorFlow, etc.

[0155] 4. Database: A database system for storing the care recipient's profile, health status, and past history data. MongoDB or MySQL is used.

[0156] 5. Feedback module: This module generates advice and feedback for the care recipient. It works in conjunction with the generation AI engine to generate appropriate feedback.

[0157] 6. Alert module: This module sends alerts to caregivers and medical institutions when an abnormality is detected. It utilizes Twilio, SMTP, etc.

[0158] 7. Big Data Analysis Engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[0159] Program processing

[0160] 1. User registration and profile creation

[0161] User: Installs the application on a tablet device and launches it. When launching the application for the first time, the user enters the required information (name, email address, password, age, gender, medical history, allergies, current health condition) and creates a profile.

[0162] Terminal: Sends the entered information to the server. HTTPS is used as the communication protocol.

[0163] Server: Stores the received data in a database and notifies the device that account creation is complete.

[0164] 2. Start a conversation session with the generative AI

[0165] User: Launches the application and taps the "Start Interaction" button.

[0166] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[0167] User: Answers the generated AI verbally about their physical condition and wishes.

[0168] 3. Acquisition of audio and image data

[0169] Device: Records the user's voice data and captures facial expressions and movements with a camera.

[0170] Terminal: Sends voice data to the voice recognition module, which converts the voice into text data. Also sends captured image data to the image analysis module, which analyzes facial expressions and movements.

[0171] 4. Data analysis and feedback

[0172] Terminal: Transmits the converted text data and image analysis results to the server.

[0173] Server: The generation AI engine evaluates the health and emotional state of the care recipient based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[0174] Terminal: The generated feedback is converted into speech using a speech synthesis engine, and is played back to the user and also displayed on the screen as a text message.

[0175] 5. Alert Notifications and Continuous Monitoring

[0176] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[0177] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[0178] 6. Big data analysis and care plan proposals

[0179] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0180] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to an electronic health record (EHR) system.

[0181] Specific examples

[0182] User: The application starts a dialogue, and the generative AI asks, "How is your day going?" The user responds verbally, "I'm feeling a little tired today."

[0183] Device: Records voice data and converts it into "I'm a little tired today" using a voice recognition module. Captures the user's facial expression using a camera, and analyzes the tired facial data using an image analysis module.

[0184] Server: Based on the received data, the generated AI evaluates the level of fatigue and generates feedback such as, "You seem tired. Please take a short rest."

[0185] Device: The feedback is synthesized into speech and played to the user. The same message is also displayed as text on the screen.

[0186] Prompt Sentence Examples

[0187] "Please explain how the generative AI engine assesses the emotional state of the care recipient."

[0188] "Please explain the process for alert notification when the system detects an abnormality."

[0189] These specific embodiments enable the system to provide high-quality, personalized care services to care recipients, thereby reducing the burden on caregivers and medical institutions.

[0190] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0191] Step 1: Register and create your profile

[0192] User: Installs the application on a tablet device and launches it by tapping the icon. On the sign-up screen, the user enters their name, email address, and password, then presses the "Next" button. Next, the user enters profile information such as age, gender, medical history, allergies, and current health condition, and presses the "Submit" button.

[0193] Input: Name, email address, password, age, gender, medical history, allergies, health status, and other information.

[0194] Output: The user's account information is saved in the database.

[0195] Terminal: The information entered by the user is sent to the server using the HTTPS protocol. If the transmission is successful, a confirmation message saying "Your account has been created" is displayed.

[0196] Server: Saves the received user information in the database. After confirming that the data was saved correctly, it sends a response to the device indicating that the account creation is complete.

[0197] Step 2: Start a conversation session with the generative AI

[0198] User: Launches the application and taps the "Start Interaction" button.

[0199] Input: The action to start the interaction.

[0200] Output: A greeting and question from the generated AI are displayed.

[0201] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[0202] User: Answers the generated AI verbally about their physical condition and wishes.

[0203] Step 3: Acquiring audio and image data

[0204] Device: Records the user's voice data through a microphone and captures facial expressions and movements with a camera.

[0205] Input: User's voice and image data.

[0206] Output: Data transfer to speech recognition module and image analysis module.

[0207] Terminal: The recorded voice data is sent to the voice recognition module, which then converts the voice into text. The captured image data is also sent to the image analysis module, which analyzes facial expressions and movements.

[0208] Step 4: Analyze data and provide feedback

[0209] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[0210] Input: Text data converted by the speech recognition module and the results of the image analysis module.

[0211] Output: Feedback generation.

[0212] Server: The generation AI engine evaluates the user's health and emotional state based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[0213] Terminal: The generated feedback is converted into speech by a speech synthesis engine and played back to the user. It is also displayed on the screen as a text message.

[0214] Step 5: Alerting and continuous monitoring

[0215] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[0216] Input: Data in which an anomaly was detected.

[0217] Output: Alert notification.

[0218] Server: Collects data to continuously monitor the user's status and periodically stores it in a database.

[0219] Step 6: Big data analysis and proposing care plans

[0220] Server: The big data analysis engine analyzes the accumulated data and generates a care plan optimized for the user.

[0221] Input: Accumulated big data.

[0222] Output: Optimized care plan.

[0223] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to the Electronic Health Record (EHR) system.

[0224] (Application example 1)

[0225] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0226] In modern society, providing personalized meal plans based on an individual's health status and dietary preferences is extremely important. In particular, food delivery services can provide greater convenience and satisfaction by suggesting optimal meals that take into account the user's health status and dietary restrictions. However, conventional systems have difficulty achieving such high levels of personalization, and there is a lack of technology to provide meal plans tailored to the user's health status.

[0227] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0228] In this invention, the server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and the analysis results and transmitting the plan to a delivery company, and a means for issuing an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[0229] "Generative AI technology" is a technology that uses natural language processing and machine learning models to interact with users and analyze data.

[0230] "Means of communication" refers to devices or programs that use generative AI technology to engage in voice or text dialogue with users.

[0231] The "means for converting acquired voice data into text data" refers to a device or program for converting a user's voice into text format using voice recognition technology.

[0232] "Means for analyzing acquired image data" refers to devices or programs that recognize and analyze the user's facial expressions and movements captured using image analysis technology.

[0233] The "means for proposing a personalized meal plan" is a device or program for generating and proposing an optimal meal plan based on the user's health condition and dietary preferences.

[0234] The "means for communicating to the delivery company" refers to a device or program for notifying the delivery company of the generated meal plan and making meal arrangements.

[0235] The "means for issuing an alert when an abnormality is detected" refers to a device or program for issuing a warning when an abnormality is detected in the user's health condition.

[0236] A "means for generating feedback" is a device or program that automatically generates appropriate advice or suggestions based on the user's input data.

[0237] "Audio or visual providing means" refers to a device or program for providing the generated feedback to the user audio or visually.

[0238] "Means for analyzing big data" refers to devices and programs that collect and analyze large amounts of data and extract useful information based on user behavior and health status.

[0239] The "means for proposing an optimized meal plan" refers to a device or program that uses the results of analyzing large amounts of data to generate and propose an optimal meal plan that meets the individual needs of a user.

[0240] "Means for sharing" refers to a device or program for sharing the generated meal plan and feedback with other parties or systems in collaboration with them.

[0241] This invention relates to a system that uses generative AI technology to propose personalized meal plans and arrange food delivery. The system aims to provide optimal meal plans based on the user's health condition and dietary preferences, and to realize these plans in cooperation with delivery companies.

[0242] System Configuration

[0243] The system includes the following major components:

[0244] 1. Generative AI engine:

[0245] It uses natural language processing and machine learning models (e.g., GPT-4) to interact with users and analyze their needs and health status.

[0246] 2. Speech Recognition Module:

[0247] The user's voice is converted into text data using the Google Speech-to-Text API or similar.

[0248] 3. Image Analysis Module:

[0249] Using OpenCV, TensorFlow, etc., the system analyzes the user's facial expressions and movements to assess their health condition.

[0250] 4. Database:

[0251] It stores user profiles, health status, dietary preferences, and historical data in databases such as MySQL and MongoDB.

[0252] 5. Feedback module:

[0253] Use a chatbot framework (such as Dialogflow) to generate feedback and advice for the user and provide it in audio or visual form.

[0254] 6. Alerts Module:

[0255] If an abnormality is detected, an alert will be sent to prompt appropriate action.

[0256] 7. Big Data Analysis Engine:

[0257] It analyzes the large amount of data collected and proposes an optimized meal plan to the user.

[0258] Program processing

[0259] The server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and analysis results, a means for transmitting the plan to a delivery company, and a means for sending an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[0260] Specific example explanation

[0261] A user can use their smartphone to start a conversation with the generative AI. For example, if they ask, "What would you recommend for lunch?", the generative AI will respond with, "Hello, Name. You seem a little tired today. I'd like to suggest a healthy salad and soup."

[0262] Example prompt sentence:

[0263] User: What would you recommend for lunch?

[0264] Generative AI: Hello, Name. You seem a little tired today. I suggest a healthy salad and soup.

[0265] The user's voice data is converted to text using the Google Speech-to-Text API, and their facial expressions and movements are analyzed using OpenCV and TensorFlow. This data is sent to a server where a generative AI engine analyzes it and proposes the optimal meal plan for the user. The proposed plan is then shared with the delivery company, and delivery is arranged.

[0266] For example, if a user responds, "I'm not feeling well," the generative AI will respond with, "I'll suggest a porridge that's easy to digest," providing appropriate feedback. If an abnormality is detected, an alert will be sent, prompting the user to take appropriate action.

[0267] This system will enable advanced personalization based on the user's health condition and individual needs, significantly improving the quality of food delivery services.

[0268] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0269] Step 1:

[0270] User registration and profile creation

[0271] The user installs and launches the application on their smartphone. They create an account by entering information such as their name, email address, password, health status, dietary preferences, and allergies. The entered information is sent by the device to the server. The server stores the received information in a database and sends a response to the device indicating that the account has been created.

[0272] Step 2:

[0273] Start a conversation session with generative AI

[0274] The user launches the application and begins a dialogue with the generation AI. The generation AI asks on the device, "Hello, Name. What kind of meal would you like to have today?" The user responds verbally about their meal preferences and physical condition. Input: User's voice data. Output: Text data about their meal preferences.

[0275] Step 3:

[0276] Acquisition of audio and image data

[0277] The device records voice data from the user and captures facial expressions and movements with a camera. The recorded voice data is converted into text data using a voice recognition module (Google Speech-to-Text API). The captured image data is analyzed for the user's facial expressions and movements using an image analysis module (OpenCV, TensorFlow). Input: User's voice data and image data. Output: Text data and analysis results.

[0278] Step 4:

[0279] Data analysis and meal plan suggestions

[0280] The device sends the converted text data and image analysis results to the server. The server analyzes this data using a generative AI engine (GPT-4) to evaluate the user's health condition and preferences. The server generates a personalized meal plan based on the evaluation results. The proposed meal plan is presented by a feedback module (Chatbot framework: Dialogflow) with the message, "For today's meals, we suggest these to suit your physical condition." Input: Text data and image analysis results. Output: Personalized meal plan.

[0281] Step 5:

[0282] Delivery arrangements and alert notifications

[0283] The server arranges an order with a delivery company based on the generated meal plan. The user's meal plan is notified to the delivery company, which then arranges delivery. If an abnormality is detected, the server also sends an alert to caregivers and medical institutions via the alert module. Input: Generated meal plan. Output: Order to delivery company, and alert notification as necessary.

[0284] Step 6:

[0285] Big data analysis and service improvement

[0286] The server analyzes the collected data using a big data analysis engine and improves the algorithm that optimizes each user's meal plan. Furthermore, the service is continuously improved based on user feedback and health status. This makes it possible to propose meal plans that are optimized for each user. Input: Large amounts of analyzed data. Output: Optimized meal plans and improved algorithms.

[0287] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0288] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[0289] System Configuration

[0290] The system includes the following major components:

[0291] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[0292] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[0293] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[0294] 4. Emotion engine: Recognizes emotions from the care recipient's voice and facial expression data.

[0295] 5. Database: Stores the profile, health status and history data of the care recipient.

[0296] 6. Feedback module: Generates advice and feedback for the care recipient.

[0297] 7. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[0298] 8. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[0299] Program processing

[0300] 1. User registration and profile creation

[0301] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[0302] Terminal: Sends the entered information to the server.

[0303] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0304] 2. Start a conversation session with the generative AI

[0305] User: Launches the application and begins interacting with the generative AI.

[0306] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[0307] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[0308] 3. Acquisition of audio and image data

[0309] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[0310] Terminal: The recorded voice data is converted into text by the voice recognition module, and the image data is analyzed by the image analysis module.

[0311] 4. Emotion Recognition

[0312] Terminal: The emotion engine analyzes the voice and image data to identify the emotions of the person receiving care.

[0313] Terminal: Sends the identified emotion data to the server.

[0314] 5. Data analysis and feedback

[0315] Terminal: Sends the text data converted by the voice recognition module, the results of the image analysis module, and the data from the emotion engine to the server.

[0316] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[0317] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[0318] Device: Provides audio and visual feedback to the care recipient.

[0319] 6. Alert Notifications and Continuous Monitoring

[0320] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0321] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[0322] 7. Big data analysis and care plan proposals

[0323] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0324] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0325] Specific examples

[0326] 1. User registration and profile creation

[0327] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[0328] 2. Start a conversation session with the generative AI

[0329] User: Launches the application and begins a conversation with the AI ​​generator. The AI ​​generator asks, "Hello, how are you today?", to which the user responds, "I'm not feeling too well."

[0330] 3. Acquisition of audio and image data

[0331] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[0332] 4. Emotion Recognition

[0333] Terminal: The emotion engine analyzes the voice and image data to identify the user's emotion. The emotion data is sent to the server.

[0334] 5. Data analysis and feedback

[0335] Device: Text data, image analysis results, and emotion data are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[0336] 6. Alert Notifications and Continuous Monitoring

[0337] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[0338] 7. Big data analysis and care plan proposals

[0339] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve services.

[0340] This concludes the description of the embodiment of the invention. It is expected that this system will provide more individually optimized care through emotion recognition and reduce the burden on caregivers.

[0341] The processing flow will be explained below.

[0342] Step 1: User Registration

[0343] User: Installs the application on the tablet device and launches it.

[0344] Device: Displays a registration screen and prompts the user to create an account.

[0345] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[0346] Terminal: Sends the entered information to the server.

[0347] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0348] Step 2: Create your profile

[0349] Device: Displays a profile creation screen and prompts the user to enter personal information.

[0350] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[0351] Terminal: Sends the entered information to the server.

[0352] Server: The received information is linked to the user account and stored in a database.

[0353] Step 3: Start interacting with generative AI

[0354] User: Launches the application and begins an interactive session with the generative AI.

[0355] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[0356] Step 4: Acquiring audio and visual data

[0357] User: Tells the generated AI about their physical condition, wishes, etc.

[0358] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[0359] Step 5: Convert audio data to text

[0360] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[0361] Step 6: Analyzing the image data

[0362] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[0363] Step 7: Emotion Recognition

[0364] Terminal: Passes image analysis and voice recognition data to the emotion engine to recognize the user's emotions.

[0365] Terminal: Sends the recognized emotion data to the server.

[0366] Step 8: Send data to the server

[0367] Terminal: Sends text data generated by voice recognition, image analysis results, and emotion data to the server.

[0368] Step 9: Analysis by generative AI

[0369] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[0370] Server: Stores the evaluation results in a database.

[0371] Step 10: Generate and provide feedback

[0372] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[0373] Server: Sends the generated feedback to the device.

[0374] Device: Provides audio and visual feedback to the care recipient.

[0375] Step 11: Alert Notification

[0376] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0377] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[0378] Step 12: Continuous monitoring and data collection

[0379] Server: Continuously collects data from the care recipient and stores it as a log.

[0380] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[0381] Step 13: Big data analysis and care plan proposal

[0382] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0383] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0384] Example 2

[0385] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0386] In today's nursing care industry, there is a need to accurately grasp the physical and mental state of care recipients and provide appropriate care tailored to their individual needs and emotions. However, conventional systems lack the technology to analyze the emotions and health status of care recipients in real time and provide optimal feedback and care plans, which contributes to the increased burden on caregivers. Furthermore, their ability to effectively analyze large amounts of data and issue appropriate alerts is limited.

[0387] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0388] In this invention, the server includes means for communicating with the care recipient using generative AI technology, means for converting acquired voice data into text data, means for analyzing acquired image data, means for recognizing emotions using an emotion engine, means for transmitting the text data, analysis results, and emotion data to a caregiver or a medical institution, and means for issuing an alert when an abnormality is detected. This makes it possible to grasp the emotional and health state of the care recipient in real time, provide optimal feedback and care plans, and immediately detect and notify abnormalities.

[0389] "Generative AI technology" refers to technology that uses artificial intelligence to generate natural language and engage in dialogue.

[0390] "Communication" refers to the process of engaging in voice and text dialogue with the care recipient through generative AI technology.

[0391] "Audio data" refers to the voice of the care recipient recorded through a microphone and expressed in digital form.

[0392] "Text data" refers to data that has been converted from voice data using a voice recognition module so that it can be handled as text information.

[0393] "Image data" refers to visual information captured through a camera or other image input device.

[0394] "Analysis" is the process of processing collected data using specific algorithms to extract meaningful information.

[0395] An "emotion engine" refers to algorithms and technologies for recognizing the emotional state of a care recipient from voice and image data.

[0396] "Emotion data" refers to information indicating an emotional state output by the emotion engine after analyzing audio and image data.

[0397] "Communication" refers to the process of sending and sharing the generated text data, analysis results, and emotional data with caregivers or medical institutions.

[0398] An "alert" is a notification sent to a caregiver or medical institution as a warning when an abnormality is detected.

[0399] "Feedback" refers to advice and comments that the generative AI technology provides to the care recipient based on analyzed data.

[0400] "Big data" refers to large data sets and the techniques and methods for analyzing them to extract useful information.

[0401] A "care plan" is a plan formulated to provide optimal nursing care services to a care recipient.

[0402] "Sharing" refers to the process of sharing and exchanging information such as generated care plans among stakeholders.

[0403] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[0404] System Configuration

[0405] The system includes the following major components:

[0406] 1. Generative AI engine: This is the main component for natural conversation with the care recipient. It uses Google Cloud Speech-to-Text API and OpenAI GPT model.

[0407] 2. Speech recognition module: This module converts the voice of the care recipient into text data. It uses the Google Cloud Speech-to-Text API, etc.

[0408] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses TensorFlow, OpenCV, etc.

[0409] 4. Emotion Engine: Algorithms and technologies for recognizing emotions from the voice and facial expression data of the care recipient. This includes tools for extracting emotions from voice and deep learning models for facial expression analysis.

[0410] 5. Database: Data storage for storing the care recipient's profile, health status, and history data. SQL or NoSQL databases are used.

[0411] 6. Feedback module: A program that generates advice and feedback for the care recipient. It works in conjunction with the generative AI engine.

[0412] 7. Alert module: This is a system for notifying caregivers and medical institutions when an abnormality is detected. It includes the function of sending emails and SMS.

[0413] 8. Big data analysis engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[0414] Program processing

[0415] The program processing of this system is as follows.

[0416] 1. The user installs and launches the application on their tablet device. The user creates an account by entering the required information (such as name, email address, and password), and then enters their profile information (e.g., name, age, gender, medical history, allergies, and current health condition).

[0417] 2. The device sends the entered information to the server, which stores the received information in a database. The server then sends a response to the device indicating that account creation is complete.

[0418] 3. The user launches the application and begins a conversation with the AI. The AI ​​greets the user via voice and text, saying, "Hello, Name. How are you today?" and asks about the user's health and wishes.

[0419] 4. When the user responds by voice, the device records the voice data and captures their facial expressions and movements with the camera. The recorded voice data is converted into text by the voice recognition module, and the image data is analyzed by the image analysis module.

[0420] 5. The device uses the emotion engine to analyze the voice and image data to identify the user's emotion, and the identified emotion data is sent to the server.

[0421] 6. The server combines the text data converted by the speech recognition module, the results of the image analysis module, and the emotion data, and analyzes them using a generative AI engine. The feedback module generates appropriate feedback based on the analysis results, and provides it to the user in the form of voice and text messages, such as "You seem tired. Please take a short rest."

[0422] 7. If the server detects an abnormality, the alert module will be activated and an alert will be sent to the caregiver or medical institution via email or SMS.

[0423] 8. The server continuously monitors the condition of the care recipient and stores the data in a database. The collected data is analyzed by a big data analysis engine to generate an optimal care plan. This care plan is shared with caregivers and medical institutions and used to improve services.

[0424] Specific examples

[0425] As a concrete example, consider a scenario in which a user installs an application, fills out a profile, and creates an account. In this system, the device sends the user's input information (e.g., name, age, medical history, etc.) to the server, which stores it in a database. When the user initiates a dialogue with the generative AI, the generative AI asks, "Hello, how are you today?" to which the user replies, "I'm not feeling well." The device records voice data and captures the user's facial expressions with a camera. This voice data is converted into text using the Google Cloud Speech-to-Text API, and the image data is analyzed using OpenCV. The emotion engine analyzes this data and identifies the user's emotions. The identified emotion data is sent to the server, which evaluates the care recipient's condition and generates appropriate feedback.

[0426] In this way, the present invention provides a specific means for understanding the emotions and health status of a care recipient in real time and providing individually optimized care services.

[0427] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0428] Step 1:

[0429] User: Installs and launches the application on the tablet device. After installing the application, the user launches the application for the first time and begins the account creation process by following the on-screen instructions.

[0430] Input: Tablet device, application installation file

[0431] Output: Installed applications

[0432] Step 2:

[0433] User: Enter information such as name, email address, and password on the account creation screen and tap the "Register" button.

[0434] Terminal: Sends the entered information to the server.

[0435] Input: Name, Email Address, Password

[0436] Output: Account information sent to the server

[0437] Step 3:

[0438] Server: Saves the received account information in a database and sends a response to the terminal indicating that the account has been created.

[0439] Input: Account information

[0440] Output: Account information stored in the database, response to the terminal

[0441] Step 4:

[0442] User: After creating an account, proceed to the screen where you enter your profile information, including your name, age, gender, medical history, allergies, current health condition, etc. Once you have completed the entry, tap the "Save" button.

[0443] Device: Sends the entered profile information to the server.

[0444] Input: Profile information

[0445] Output: Profile information sent to the server

[0446] Step 5:

[0447] Server: Saves the received profile information in a database and sends a response to the terminal indicating that the profile has been created.

[0448] Input: Profile information

[0449] Output: Profile information stored in the database, response to the device

[0450] Step 6:

[0451] User: Taps the "Start interactive session" button on the application's home screen.

[0452] Device: The generative AI engine will launch and say "Hello, Name. How are you today?" The same text will also be displayed on the screen.

[0453] Input: Button to start an interactive session, Generative AI engine

[0454] Output: Greeting speech and text

[0455] Step 7:

[0456] User: Answers questions from the generated AI by voice, such as about their physical condition and wishes.

[0457] Device: Records the user's voice data using a microphone.

[0458] Input: Voice response to generative AI

[0459] Output: Recorded audio data

[0460] Step 8:

[0461] Terminal: The camera simultaneously captures the user's facial expressions and movements, and the captured images are sent to the image analysis module in real time.

[0462] Input: User's facial expressions and movements

[0463] Output: Captured image data

[0464] Step 9:

[0465] On the device: The recorded voice data is converted into text by a speech recognition module, using the Google Cloud Speech-to-Text API.

[0466] Input: Recorded audio data

[0467] Output: Converted text data

[0468] Step 10:

[0469] Terminal: The captured image data is analyzed using an image analysis module (e.g., using TensorFlow or OpenCV).

[0470] Input: Captured image data

[0471] Output: Analyzed image data

[0472] Step 11:

[0473] Terminal: The emotion engine analyzes the voice and image data to identify the user's emotions.

[0474] Input: converted text data, analyzed image data

[0475] Output: Identified emotion data

[0476] Step 12:

[0477] Terminal: Sends the identified emotion data to the server.

[0478] Input: Identified emotion data

[0479] Output: Emotion data sent to the server

[0480] Step 13:

[0481] Server: The integrated data (text data converted by the voice recognition module, image analysis results, and emotion data) is analyzed using a generative AI engine to evaluate the health and emotional state of the care recipient.

[0482] Input: Text data, analysis results, emotion data

[0483] Output: Assessed health and emotional states

[0484] Step 14:

[0485] Server: The feedback module generates appropriate feedback based on the evaluation results (e.g., "You seem tired. Please take a rest.").

[0486] Input: Assessed health and emotional states

[0487] Output: Generated feedback

[0488] Step 15:

[0489] Terminal: Provides generated feedback to the user as voice and text messages.

[0490] Input: Generated feedback

[0491] Output: Feedback as audio and text messages

[0492] Step 16:

[0493] Server: If an abnormality is detected, the alert module is activated and notifies caregivers and medical institutions via email or SMS.

[0494] Input: Detected anomaly

[0495] Output: Alert notification to caregivers and medical institutions

[0496] Step 17:

[0497] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[0498] Input: Status data of care recipient

[0499] Output: Data stored in the database

[0500] Step 18:

[0501] Server: The big data analytics engine analyzes the accumulated data and finds trends and patterns.

[0502] Input: Accumulated data

[0503] Output: Trends and patterns as analytical results

[0504] Step 19:

[0505] Server: Generates optimal care plans based on the analysis results and shares them with caregivers and medical institutions.

[0506] Input: Trend and pattern analysis results

[0507] Output: Generated care plans and their sharing

[0508] The above are the specific processing steps of this system. This system makes it possible to accurately grasp the condition of the care recipient and provide prompt and appropriate care.

[0509] (Application example 2)

[0510] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0511] Conventional security systems have difficulty assessing a user's emotional state in real time, making it difficult to take appropriate action in emergencies. They also lack the means to provide immediate feedback or alerts to ensure user safety. In particular, there is a need for a system that can monitor a user's state over the long term and implement optimal security measures without compromising personal privacy.

[0512] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for communicating with the care recipient using generative AI technology, means for converting acquired voice data into text data, means for analyzing acquired image data, means for identifying emotions based on voice recognition and image analysis data, means for transmitting the text data, analysis results, and emotional data to a caregiver or a medical institution, and means for issuing an alert when an abnormality is detected. This allows the user's emotional state to be evaluated in real time and emergency contacts to be notified in the event of an abnormality, ensuring the user's safety and enabling optimal security measures to be taken through long-term data monitoring.

[0513] "Generative AI technology" is software technology that uses natural language processing and machine learning algorithms to engage in natural conversations with humans.

[0514] "Voice data" refers to the voice information input by the user, and is the data that is analyzed by the voice recognition module based on this information.

[0515] "Text data" refers to character information converted from voice data by a voice recognition module.

[0516] "Image data" refers to visual information captured by a camera of a user's facial expressions and movements.

[0517] "Voice recognition and image analysis data" refers to the results of analyzing acquired voice data and image data and converting them into various types of information.

[0518] "Means for identifying emotions" refers to a technology or system that analyzes audio and image data to assess a user's emotional state.

[0519] "Means of transmitting to caregivers or medical institutions" refers to technology that transmits user analysis data to caregivers or medical institutions in real time.

[0520] "Means for sending an alert" refers to a technology that sends warning information to a pre-set emergency contact when the system detects an abnormal condition.

[0521] "Big data analysis" is a technology that collects and analyzes large amounts of data and uses the results to derive new insights and patterns.

[0522] "Security measures" refer to the defensive measures and response procedures established to ensure user safety.

[0523] "Feedback" refers to advice and suggestions generated by the system based on the user's behavior and emotional state.

[0524] "Means for notifying emergency contacts" refers to technology that automatically notifies the set emergency contacts when the system detects an abnormality.

[0525] This invention relates to a security system that uses generative AI technology and an emotion engine to analyze user emotions in real time to improve safety. The system collects and analyzes voice and image data, and issues an emergency notification if it detects an abnormal emotional state.

[0526] System Configuration

[0527] The system consists of the following major components:

[0528] 1. Generative AI engine: This is the primary software for engaging in natural dialogue with users.

[0529] 2. Speech recognition module: Converts collected voice data into text data.

[0530] 3. Image analysis module: Analyzes the acquired image data and evaluates the user's facial expressions and movements.

[0531] 4. Emotion Engine: Identify user emotions from audio and image data.

[0532] 5. Database: Stores user profiles and historical data.

[0533] 6. Feedback module: Generates advice and feedback for the user.

[0534] 7. Alert module: Notifies emergency contacts when an abnormality is detected.

[0535] 8. Big Data Analysis Engine: Analyzes collected data and proposes security plans.

[0536] Hardware and Software Used

[0537] Hardware used

[0538] Smartphone: Equipped with a camera and microphone to collect audio and image data.

[0539] Server: Processes the collected data and stores the analysis results.

[0540] Software used

[0541] Speech Recognition Module: Uses Google Speech-to-Text API.

[0542] Image Analysis Module: Analyzes image data in real time using OpenCV.

[0543] Emotion Engine: Identifies emotions using Azure Emotion API or Hume AI.

[0544] Database: Use AWS DynamoDB or Firebase to store data.

[0545] Feedback module: Uses a custom message generation algorithm.

[0546] Alert Module: Uses the Twilio API to notify emergency contacts.

[0547] Details of data processing and calculation

[0548] Acquisition of audio and image data

[0549] Device: Records voice data from the user and captures facial expressions and movements with the camera. This data is converted to text using the Google Speech-to-Text API, and image data is analyzed using OpenCV.

[0550] Emotion Recognition in Action

[0551] On the device: The emotion engine analyzes the voice and image data to identify the user's emotion. This information is sent to the server.

[0552] Analyzing data and providing feedback

[0553] Server: Based on the results of voice and image analysis, the user's emotional state is evaluated, and the feedback module generates appropriate advice. Real-time voice and visual feedback is provided to the user.

[0554] Alert Notifications

[0555] Server: If an anomaly is detected, the alert module immediately sends a notification to emergency contacts.

[0556] Specific examples

[0557] If a user comes home late at night and is captured looking nervous:

[0558] Application: The generative AI asks, "Good work. You're feeling a little nervous today, aren't you?"

[0559] User: "Yes, I'm a little nervous."

[0560] Application: The emotion engine detects the user's anxiety, and the alert module notifies emergency contacts in real time.

[0561] Example of input prompt for generative AI model

[0562] Analyze the user's voice and facial expressions to assess their current emotional state. If anxiety or tension is increasing, generate appropriate feedback and alert emergency contacts if necessary.

[0563]

[0564] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0565] Step 1:

[0566] User registration and profile creation

[0567] User: Installs the application on a smartphone and launches it. Creates an account by entering name, email address, password, age, gender, and profile information. Specifically, enters the required information into the application's input form and presses the registration button.

[0568] Input: Name, email address, password, age, gender and profile information.

[0569] Terminal: Sends the entered information to the server. Specifically, it sends data to the server using an HTTP request.

[0570] Output: Registration data sent to the server.

[0571] Server: Stores the received information in a database and sends an account creation completion response to the terminal.

[0572] Output: Account creation complete response.

[0573] Step 2:

[0574] Start a conversation session with generative AI

[0575] User: Launches the application and begins interacting with the generation AI by pressing the start button on the application's main screen.

[0576] Input: Instruction to start a dialogue.

[0577] Terminal: The generated AI asks, "Hello, Name. How are you doing today?"

[0578] Output: Question from the generative AI.

[0579] User: Answers the generated AI verbally about their physical condition and emotions. Specifically, they speak into a microphone.

[0580] Input: User's voice data.

[0581] Step 3:

[0582] Acquisition of audio and image data

[0583] Device: Records voice data from the user and captures facial expressions and movements with a camera. Specifically, the device uses the smartphone's microphone and camera to capture voice and images.

[0584] Input: Audio data, image data.

[0585] Terminal: The voice recognition module converts the recorded voice data into text, and the image data is analyzed by the image analysis module.

[0586] Output: Text data, analysis results.

[0587] Step 4:

[0588] Emotion Recognition in Action

[0589] On the device: The emotion engine analyzes audio and image data to identify the user's emotions. Specifically, emotion analysis is performed using Azure Emotion API and Hume AI.

[0590] Input: Text data, image analysis data.

[0591] Output: Emotion data.

[0592] Terminal: Sends the identified emotion data to the server. Specifically, it sends the emotion data using an HTTP request.

[0593] Output: Emotion data transmission.

[0594] Step 5:

[0595] Analyzing data and providing feedback

[0596] Server: Analyzes data to assess the user's health and emotional state based on the results of the voice recognition and image analysis modules and emotional data. Specifically, it uses a generative AI engine to integrate and evaluate multiple data sources.

[0597] Input: text data, image analysis results, emotion data.

[0598] Output: Health status assessment, emotional status assessment.

[0599] Server: The feedback module generates appropriate feedback to the user, such as advice like "Take a short rest."

[0600] Output: Feedback.

[0601] Terminal: Provides generated feedback to the user in audio and visual form.

[0602] Output: Provide feedback.

[0603] Step 6:

[0604] Alerting and Continuous Monitoring

[0605] Server: If an anomaly is detected, the alert module notifies the user's emergency contacts by sending an alert via SMS or phone call using the Twilio API.

[0606] Input: Emotion data, evaluation results.

[0607] Output: Alert notification.

[0608] Server: Continuously monitors the user's status and accumulates the collected data in a database. Specifically, data is periodically added to the database.

[0609] Output: Database updated.

[0610] Step 7:

[0611] Big data analysis and security plan proposals

[0612] Server: The big data analysis engine analyzes the collected data and generates a security plan optimized for the user. Specifically, it integrates long-term data and analyzes user behavior patterns and emotional tendencies.

[0613] Input: Collected data.

[0614] Output: Optimized security plan.

[0615] Server: Shares the generated security plan with the user and their emergency contacts via email or in-app notification.

[0616] Output: Security plan share.

[0617] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0618] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0620] [Second embodiment]

[0621] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0622] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0623] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0625] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0627] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0628] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0629] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0630] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0631] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0632] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0633] This invention relates to a system that uses generative AI technology to provide personalized care services to individuals. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement appropriate care plans.

[0634] System Configuration

[0635] The system includes the following major components:

[0636] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[0637] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[0638] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[0639] 4. Database: Stores the care recipient's profile, health status, and history data.

[0640] 5. Feedback module: Generates advice and feedback for the care recipient.

[0641] 6. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[0642] 7. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[0643] Program processing

[0644] 1. User registration and profile creation

[0645] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[0646] Terminal: Sends the entered information to the server.

[0647] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0648] 2. Start a conversation session with the generative AI

[0649] User: Launches the application and begins interacting with the generative AI.

[0650] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[0651] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[0652] 3. Acquisition of audio and image data

[0653] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[0654] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data. The captured image data is passed to the image analysis module, where facial expressions and movements are analyzed.

[0655] 4. Data analysis and feedback

[0656] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[0657] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data and image analysis results.

[0658] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[0659] Device: Provides audio and visual feedback to the care recipient.

[0660] 5. Alert Notifications and Continuous Monitoring

[0661] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0662] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[0663] 6. Big data analysis and care plan proposals

[0664] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0665] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0666] Specific examples

[0667] 1. User registration and profile creation

[0668] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[0669] 2. Start a conversation session with the generative AI

[0670] User: Launches the application and begins a dialogue with the AI ​​generator. The AI ​​generator asks, "How are you feeling today?", to which the user responds, "I'm not feeling too well."

[0671] 3. Acquisition of audio and image data

[0672] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[0673] 4. Data analysis and feedback

[0674] Device: The text data and image analysis results are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[0675] 5. Alert Notifications and Continuous Monitoring

[0676] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[0677] 6. Big data analysis and care plan proposals

[0678] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve the plan.

[0679] The above is a description of the mode for carrying out the invention. It is expected that this system will provide individually optimized care and reduce the burden on caregivers.

[0680] The processing flow will be explained below.

[0681] Step 1: User Registration

[0682] User: Installs the application on the tablet device and launches it.

[0683] Device: Displays a registration screen and prompts the user to create an account.

[0684] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[0685] Terminal: Sends the entered information to the server.

[0686] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0687] Step 2: Create your profile

[0688] Device: Displays a profile creation screen and prompts the user to enter personal information.

[0689] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[0690] Terminal: Sends the entered information to the server.

[0691] Server: The received information is linked to the user account and stored in a database.

[0692] Step 3: Start interacting with generative AI

[0693] User: Launches the application and begins an interactive session with the generative AI.

[0694] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[0695] Step 4: Acquiring audio and visual data

[0696] User: Tells the generated AI about their physical condition, wishes, etc.

[0697] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[0698] Step 5: Convert audio data to text

[0699] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[0700] Step 6: Analyzing the image data

[0701] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[0702] Step 7: Send data to the server

[0703] Terminal: Sends text data generated by voice recognition and image analysis results to the server.

[0704] Step 8: Analysis by generative AI

[0705] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on the text data and image analysis results.

[0706] Server: Stores the evaluation results in a database.

[0707] Step 9: Generate and provide feedback

[0708] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[0709] Server: Sends the generated feedback to the device.

[0710] Device: Provides audio and visual feedback to the care recipient.

[0711] Step 10: Alert Notification

[0712] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0713] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[0714] Step 11: Continuous monitoring and data collection

[0715] Server: Continuously collects data from the care recipient and stores it as a log.

[0716] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[0717] Step 12: Big data analysis and care plan proposal

[0718] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0719] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0720] Example 1

[0721] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] In an aging society, there is a need to improve the quality of nursing care services, provide personalized care that meets the individual needs of care recipients, and reduce the burden on caregivers and medical institutions. However, conventional systems have difficulty accurately understanding the health and emotional state of care recipients in real time and providing appropriate feedback and care plans. As a result, there has been a lack of efficient means to improve the quality of nursing care services.

[0723] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0724] In this invention, the server includes: means for communicating with the care recipient using AI technology; means for converting acquired voice data into text data; means for analyzing acquired image data; means for transmitting the text data and analysis results; means for issuing an alert when an abnormality is detected; means for a user to input voice and image data; means for transmitting the voice and image data to the server; means for generating feedback for the care recipient using AI technology; means for providing the feedback in audio or visual form; means for generating feedback based on data received by the AI ​​from the server; means for transmitting the feedback from the server to a terminal; means for analyzing big data and proposing an optimized care plan for the care recipient; means for sharing the care plan with a caregiver or a medical institution; means for analyzing the acquired big data; means for generating a care plan based on the analysis results; and means for notifying the caregiver or a medical institution of the care plan from the server. This enables real-time monitoring of the condition of the care recipient and providing appropriate feedback and a personalized care plan.

[0725] "Generative AI technology" is a technology that uses natural language processing and machine learning algorithms to engage in natural dialogue with humans and generate appropriate responses and information.

[0726] "Care recipients" refers to elderly people or individuals with physical disabilities who require nursing or medical assistance.

[0727] "Means of communication" refers to the method by which the system has the functionality to exchange information with the care recipient via voice or text.

[0728] "Audio data" refers to information that has been digitally recorded from the voice of the person receiving care.

[0729] "Text data" refers to information in which voice data is expressed as a string of characters.

[0730] The term "means for converting voice data into text data" refers to a method for converting voice data into corresponding text data using voice recognition technology.

[0731] "Image data" refers to information that digitally records the facial expressions and movements of the person receiving care.

[0732] "Means for analyzing image data" refers to a method of using image processing technology to recognize the facial expressions and movements of the person being cared for and analyze their meaning.

[0733] "Means for transmitting text data and analysis results" refers to the communication function for transmitting the text data and image analysis results acquired by the system to a remote server or other device.

[0734] "Means for sending an alert" refers to a method that has the function of sending a notification when an abnormality is detected and warning caregivers and medical institutions.

[0735] "Means for inputting voice and image data" refers to a method having the function of capturing voice and image data from the care recipient and inputting them into the system.

[0736] "Means by which the generating AI generates feedback based on data received from the server" refers to a method by which the generating AI creates appropriate feedback for the care recipient based on data received from the server.

[0737] "Means for providing audio or visual feedback" refers to a method that has the function of providing the generated feedback to the care recipient as audio and images.

[0738] "Means of analyzing big data" refers to methods that use technology to analyze large amounts of data and identify patterns and trends.

[0739] "Means for proposing an optimized care plan" refers to a method that has the function of creating and proposing an optimal care plan for a care recipient based on the analysis results.

[0740] "Means for sharing a care plan with a caregiver or a medical institution" refers to a method that has a function for sharing a generated care plan with a caregiver or a medical institution through communication.

[0741] A "server" is a computing device that serves as the core of a system and plays a central role in storing, analyzing, and transmitting data.

[0742] "Terminal" refers to a device (e.g., tablet, smartphone) that a user directly operates to communicate.

[0743] These definitions provide a clear understanding of the technical scope and function of the invention and serve as a basis for patent prosecution and enforcement.

[0744] This invention relates to a system that provides personalized care services to individuals using generative AI technology. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement an appropriate care plan. The system includes the following main components:

[0745] System Configuration

[0746] 1. Generative AI Engine: This is the main software component for natural dialogue with the care recipient. The generative AI engine uses natural language processing and machine learning algorithms.

[0747] 2. Speech recognition module: This module recognizes the voice of the care recipient and converts it into text data. Specifically, it uses the Google Cloud Speech-to-Text API.

[0748] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses OpenCV, TensorFlow, etc.

[0749] 4. Database: A database system for storing the care recipient's profile, health status, and past history data. MongoDB or MySQL is used.

[0750] 5. Feedback module: This module generates advice and feedback for the care recipient. It works in conjunction with the generation AI engine to generate appropriate feedback.

[0751] 6. Alert module: This module sends alerts to caregivers and medical institutions when an abnormality is detected. It utilizes Twilio, SMTP, etc.

[0752] 7. Big Data Analysis Engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[0753] Program processing

[0754] 1. User registration and profile creation

[0755] User: Installs the application on a tablet device and launches it. When launching the application for the first time, the user enters the required information (name, email address, password, age, gender, medical history, allergies, current health condition) and creates a profile.

[0756] Terminal: Sends the entered information to the server. HTTPS is used as the communication protocol.

[0757] Server: Stores the received data in a database and notifies the device that account creation is complete.

[0758] 2. Start a conversation session with the generative AI

[0759] User: Launches the application and taps the "Start Interaction" button.

[0760] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[0761] User: Answers the generated AI verbally about their physical condition and wishes.

[0762] 3. Acquisition of audio and image data

[0763] Device: Records the user's voice data and captures facial expressions and movements with a camera.

[0764] Terminal: Sends voice data to the voice recognition module, which converts the voice into text data. Also sends captured image data to the image analysis module, which analyzes facial expressions and movements.

[0765] 4. Data analysis and feedback

[0766] Terminal: Transmits the converted text data and image analysis results to the server.

[0767] Server: The generation AI engine evaluates the health and emotional state of the care recipient based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[0768] Terminal: The generated feedback is converted into speech using a speech synthesis engine, and is played back to the user and also displayed on the screen as a text message.

[0769] 5. Alert Notifications and Continuous Monitoring

[0770] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[0771] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[0772] 6. Big data analysis and care plan proposals

[0773] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0774] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to an electronic health record (EHR) system.

[0775] Specific examples

[0776] User: The application starts a dialogue, and the generative AI asks, "How is your day going?" The user responds verbally, "I'm feeling a little tired today."

[0777] Device: Records voice data and converts it into "I'm a little tired today" using a voice recognition module. Captures the user's facial expression using a camera, and analyzes the tired facial data using an image analysis module.

[0778] Server: Based on the received data, the generated AI evaluates the level of fatigue and generates feedback such as, "You seem tired. Please take a short rest."

[0779] Device: The feedback is synthesized into speech and played to the user. The same message is also displayed as text on the screen.

[0780] Prompt Sentence Examples

[0781] "Please explain how the generative AI engine assesses the emotional state of the care recipient."

[0782] "Please explain the process for alert notification when the system detects an abnormality."

[0783] These specific embodiments enable the system to provide high-quality, personalized care services to care recipients, thereby reducing the burden on caregivers and medical institutions.

[0784] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0785] Step 1: Register and create your profile

[0786] User: Installs the application on a tablet device and launches it by tapping the icon. On the sign-up screen, the user enters their name, email address, and password, then presses the "Next" button. Next, the user enters profile information such as age, gender, medical history, allergies, and current health condition, and presses the "Submit" button.

[0787] Input: Name, email address, password, age, gender, medical history, allergies, health status, and other information.

[0788] Output: The user's account information is saved in the database.

[0789] Terminal: The information entered by the user is sent to the server using the HTTPS protocol. If the transmission is successful, a confirmation message saying "Your account has been created" is displayed.

[0790] Server: Saves the received user information in the database. After confirming that the data was saved correctly, it sends a response to the device indicating that the account creation is complete.

[0791] Step 2: Start a conversation session with the generative AI

[0792] User: Launches the application and taps the "Start Interaction" button.

[0793] Input: The action to start the interaction.

[0794] Output: A greeting and question from the generated AI are displayed.

[0795] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[0796] User: Answers the generated AI verbally about their physical condition and wishes.

[0797] Step 3: Acquiring audio and image data

[0798] Device: Records the user's voice data through a microphone and captures facial expressions and movements with a camera.

[0799] Input: User's voice and image data.

[0800] Output: Data transfer to speech recognition module and image analysis module.

[0801] Terminal: The recorded voice data is sent to the voice recognition module, which then converts the voice into text. The captured image data is also sent to the image analysis module, which analyzes facial expressions and movements.

[0802] Step 4: Analyze data and provide feedback

[0803] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[0804] Input: Text data converted by the speech recognition module and the results of the image analysis module.

[0805] Output: Feedback generation.

[0806] Server: The generation AI engine evaluates the user's health and emotional state based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[0807] Terminal: The generated feedback is converted into speech by a speech synthesis engine and played back to the user. It is also displayed on the screen as a text message.

[0808] Step 5: Alerting and continuous monitoring

[0809] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[0810] Input: Data in which an anomaly was detected.

[0811] Output: Alert notification.

[0812] Server: Collects data to continuously monitor the user's status and periodically stores it in a database.

[0813] Step 6: Big data analysis and proposing care plans

[0814] Server: The big data analysis engine analyzes the accumulated data and generates a care plan optimized for the user.

[0815] Input: Accumulated big data.

[0816] Output: Optimized care plan.

[0817] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to the Electronic Health Record (EHR) system.

[0818] (Application example 1)

[0819] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0820] In modern society, providing personalized meal plans based on an individual's health status and dietary preferences is extremely important. In particular, food delivery services can provide greater convenience and satisfaction by suggesting optimal meals that take into account the user's health status and dietary restrictions. However, conventional systems have difficulty achieving such high levels of personalization, and there is a lack of technology to provide meal plans tailored to the user's health status.

[0821] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0822] In this invention, the server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and the analysis results and transmitting the plan to a delivery company, and a means for issuing an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[0823] "Generative AI technology" is a technology that uses natural language processing and machine learning models to interact with users and analyze data.

[0824] "Means of communication" refers to devices or programs that use generative AI technology to engage in voice or text dialogue with users.

[0825] The "means for converting acquired voice data into text data" refers to a device or program for converting a user's voice into text format using voice recognition technology.

[0826] "Means for analyzing acquired image data" refers to devices or programs that recognize and analyze the user's facial expressions and movements captured using image analysis technology.

[0827] The "means for proposing a personalized meal plan" is a device or program for generating and proposing an optimal meal plan based on the user's health condition and dietary preferences.

[0828] The "means for communicating to the delivery company" refers to a device or program for notifying the delivery company of the generated meal plan and making meal arrangements.

[0829] The "means for issuing an alert when an abnormality is detected" refers to a device or program for issuing a warning when an abnormality is detected in the user's health condition.

[0830] A "means for generating feedback" is a device or program that automatically generates appropriate advice or suggestions based on the user's input data.

[0831] "Audio or visual providing means" refers to a device or program for providing the generated feedback to the user audio or visually.

[0832] "Means for analyzing big data" refers to devices and programs that collect and analyze large amounts of data and extract useful information based on user behavior and health status.

[0833] The "means for proposing an optimized meal plan" refers to a device or program that uses the results of analyzing large amounts of data to generate and propose an optimal meal plan that meets the individual needs of a user.

[0834] "Means for sharing" refers to a device or program for sharing the generated meal plan and feedback with other parties or systems in collaboration with them.

[0835] This invention relates to a system that uses generative AI technology to propose personalized meal plans and arrange food delivery. The system aims to provide optimal meal plans based on the user's health condition and dietary preferences, and to realize these plans in cooperation with delivery companies.

[0836] System Configuration

[0837] The system includes the following major components:

[0838] 1. Generative AI engine:

[0839] It uses natural language processing and machine learning models (e.g., GPT-4) to interact with users and analyze their needs and health status.

[0840] 2. Speech Recognition Module:

[0841] The user's voice is converted into text data using the Google Speech-to-Text API or similar.

[0842] 3. Image Analysis Module:

[0843] Using OpenCV, TensorFlow, etc., the system analyzes the user's facial expressions and movements to assess their health condition.

[0844] 4. Database:

[0845] It stores user profiles, health status, dietary preferences, and historical data in databases such as MySQL and MongoDB.

[0846] 5. Feedback module:

[0847] Use a chatbot framework (such as Dialogflow) to generate feedback and advice for the user and provide it in audio or visual form.

[0848] 6. Alerts Module:

[0849] If an abnormality is detected, an alert will be sent to prompt appropriate action.

[0850] 7. Big Data Analysis Engine:

[0851] It analyzes the large amount of data collected and proposes an optimized meal plan to the user.

[0852] Program processing

[0853] The server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and analysis results, a means for transmitting the plan to a delivery company, and a means for sending an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[0854] Specific example explanation

[0855] A user can use their smartphone to start a conversation with the generative AI. For example, if they ask, "What would you recommend for lunch?", the generative AI will respond with, "Hello, Name. You seem a little tired today. I'd like to suggest a healthy salad and soup."

[0856] Example prompt sentence:

[0857] User: What would you recommend for lunch?

[0858] Generative AI: Hello, Name. You seem a little tired today. I suggest a healthy salad and soup.

[0859] The user's voice data is converted to text using the Google Speech-to-Text API, and their facial expressions and movements are analyzed using OpenCV and TensorFlow. This data is sent to a server where a generative AI engine analyzes it and proposes the optimal meal plan for the user. The proposed plan is then shared with the delivery company, and delivery is arranged.

[0860] For example, if a user responds, "I'm not feeling well," the generative AI will respond with, "I'll suggest a porridge that's easy to digest," providing appropriate feedback. If an abnormality is detected, an alert will be sent, prompting the user to take appropriate action.

[0861] This system will enable advanced personalization based on the user's health condition and individual needs, significantly improving the quality of food delivery services.

[0862] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0863] Step 1:

[0864] User registration and profile creation

[0865] The user installs and launches the application on their smartphone. They create an account by entering information such as their name, email address, password, health status, dietary preferences, and allergies. The entered information is sent by the device to the server. The server stores the received information in a database and sends a response to the device indicating that the account has been created.

[0866] Step 2:

[0867] Start a conversation session with generative AI

[0868] The user launches the application and begins a dialogue with the generation AI. The generation AI asks on the device, "Hello, Name. What kind of meal would you like to have today?" The user responds verbally about their meal preferences and physical condition. Input: User's voice data. Output: Text data about their meal preferences.

[0869] Step 3:

[0870] Acquisition of audio and image data

[0871] The device records voice data from the user and captures facial expressions and movements with a camera. The recorded voice data is converted into text data using a voice recognition module (Google Speech-to-Text API). The captured image data is analyzed for the user's facial expressions and movements using an image analysis module (OpenCV, TensorFlow). Input: User's voice data and image data. Output: Text data and analysis results.

[0872] Step 4:

[0873] Data analysis and meal plan suggestions

[0874] The device sends the converted text data and image analysis results to the server. The server analyzes this data using a generative AI engine (GPT-4) to evaluate the user's health condition and preferences. The server generates a personalized meal plan based on the evaluation results. The proposed meal plan is presented by a feedback module (Chatbot framework: Dialogflow) with the message, "For today's meals, we suggest these to suit your physical condition." Input: Text data and image analysis results. Output: Personalized meal plan.

[0875] Step 5:

[0876] Delivery arrangements and alert notifications

[0877] The server arranges an order with a delivery company based on the generated meal plan. The user's meal plan is notified to the delivery company, which then arranges delivery. If an abnormality is detected, the server also sends an alert to caregivers and medical institutions via the alert module. Input: Generated meal plan. Output: Order to delivery company, and alert notification as necessary.

[0878] Step 6:

[0879] Big data analysis and service improvement

[0880] The server analyzes the collected data using a big data analysis engine and improves the algorithm that optimizes each user's meal plan. Furthermore, the service is continuously improved based on user feedback and health status. This makes it possible to propose meal plans that are optimized for each user. Input: Large amounts of analyzed data. Output: Optimized meal plans and improved algorithms.

[0881] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0882] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[0883] System Configuration

[0884] The system includes the following major components:

[0885] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[0886] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[0887] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[0888] 4. Emotion engine: Recognizes emotions from the care recipient's voice and facial expression data.

[0889] 5. Database: Stores the profile, health status and history data of the care recipient.

[0890] 6. Feedback module: Generates advice and feedback for the care recipient.

[0891] 7. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[0892] 8. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[0893] Program processing

[0894] 1. User registration and profile creation

[0895] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[0896] Terminal: Sends the entered information to the server.

[0897] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0898] 2. Start a conversation session with the generative AI

[0899] User: Launches the application and begins interacting with the generative AI.

[0900] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[0901] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[0902] 3. Acquisition of audio and image data

[0903] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[0904] Terminal: The recorded voice data is converted into text by the voice recognition module, and the image data is analyzed by the image analysis module.

[0905] 4. Emotion Recognition

[0906] Terminal: The emotion engine analyzes the voice and image data to identify the emotions of the person receiving care.

[0907] Terminal: Sends the identified emotion data to the server.

[0908] 5. Data analysis and feedback

[0909] Terminal: Sends the text data converted by the voice recognition module, the results of the image analysis module, and the data from the emotion engine to the server.

[0910] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[0911] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[0912] Device: Provides audio and visual feedback to the care recipient.

[0913] 6. Alert Notifications and Continuous Monitoring

[0914] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0915] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[0916] 7. Big data analysis and care plan proposals

[0917] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0918] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0919] Specific examples

[0920] 1. User registration and profile creation

[0921] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[0922] 2. Start a conversation session with the generative AI

[0923] User: Launches the application and begins a conversation with the AI ​​generator. The AI ​​generator asks, "Hello, how are you today?", to which the user responds, "I'm not feeling too well."

[0924] 3. Acquisition of audio and image data

[0925] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[0926] 4. Emotion Recognition

[0927] Terminal: The emotion engine analyzes the voice and image data to identify the user's emotion. The emotion data is sent to the server.

[0928] 5. Data analysis and feedback

[0929] Device: Text data, image analysis results, and emotion data are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[0930] 6. Alert Notifications and Continuous Monitoring

[0931] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[0932] 7. Big data analysis and care plan proposals

[0933] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve services.

[0934] This concludes the description of the embodiment of the invention. It is expected that this system will provide more individually optimized care through emotion recognition and reduce the burden on caregivers.

[0935] The processing flow will be explained below.

[0936] Step 1: User Registration

[0937] User: Installs the application on the tablet device and launches it.

[0938] Device: Displays a registration screen and prompts the user to create an account.

[0939] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[0940] Terminal: Sends the entered information to the server.

[0941] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[0942] Step 2: Create your profile

[0943] Device: Displays a profile creation screen and prompts the user to enter personal information.

[0944] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[0945] Terminal: Sends the entered information to the server.

[0946] Server: The received information is linked to the user account and stored in a database.

[0947] Step 3: Start interacting with generative AI

[0948] User: Launches the application and begins an interactive session with the generative AI.

[0949] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[0950] Step 4: Acquiring audio and visual data

[0951] User: Tells the generated AI about their physical condition, wishes, etc.

[0952] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[0953] Step 5: Convert audio data to text

[0954] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[0955] Step 6: Analyzing the image data

[0956] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[0957] Step 7: Emotion Recognition

[0958] Terminal: Passes image analysis and voice recognition data to the emotion engine to recognize the user's emotions.

[0959] Terminal: Sends the recognized emotion data to the server.

[0960] Step 8: Send data to the server

[0961] Terminal: Sends text data generated by voice recognition, image analysis results, and emotion data to the server.

[0962] Step 9: Analysis by generative AI

[0963] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[0964] Server: Stores the evaluation results in a database.

[0965] Step 10: Generate and provide feedback

[0966] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[0967] Server: Sends the generated feedback to the device.

[0968] Device: Provides audio and visual feedback to the care recipient.

[0969] Step 11: Alert Notification

[0970] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[0971] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[0972] Step 12: Continuous monitoring and data collection

[0973] Server: Continuously collects data from the care recipient and stores it as a log.

[0974] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[0975] Step 13: Big data analysis and care plan proposal

[0976] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[0977] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[0978] Example 2

[0979] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0980] In today's nursing care industry, there is a need to accurately grasp the physical and mental state of care recipients and provide appropriate care tailored to their individual needs and emotions. However, conventional systems lack the technology to analyze the emotions and health status of care recipients in real time and provide optimal feedback and care plans, which contributes to the increased burden on caregivers. Furthermore, their ability to effectively analyze large amounts of data and issue appropriate alerts is limited.

[0981] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0982] In this invention, the server includes means for communicating with the care recipient using generative AI technology, means for converting acquired voice data into text data, means for analyzing acquired image data, means for recognizing emotions using an emotion engine, means for transmitting the text data, analysis results, and emotion data to a caregiver or a medical institution, and means for issuing an alert when an abnormality is detected. This makes it possible to grasp the emotional and health state of the care recipient in real time, provide optimal feedback and care plans, and immediately detect and notify abnormalities.

[0983] "Generative AI technology" refers to technology that uses artificial intelligence to generate natural language and engage in dialogue.

[0984] "Communication" refers to the process of engaging in voice and text dialogue with the care recipient through generative AI technology.

[0985] "Audio data" refers to the voice of the care recipient recorded through a microphone and expressed in digital form.

[0986] "Text data" refers to data that has been converted from voice data using a voice recognition module so that it can be handled as text information.

[0987] "Image data" refers to visual information captured through a camera or other image input device.

[0988] "Analysis" is the process of processing collected data using specific algorithms to extract meaningful information.

[0989] An "emotion engine" refers to algorithms and technologies for recognizing the emotional state of a care recipient from voice and image data.

[0990] "Emotion data" refers to information indicating an emotional state output by the emotion engine after analyzing audio and image data.

[0991] "Communication" refers to the process of sending and sharing the generated text data, analysis results, and emotional data with caregivers or medical institutions.

[0992] An "alert" is a notification sent to a caregiver or medical institution as a warning when an abnormality is detected.

[0993] "Feedback" refers to advice and comments that the generative AI technology provides to the care recipient based on analyzed data.

[0994] "Big data" refers to large data sets and the techniques and methods for analyzing them to extract useful information.

[0995] A "care plan" is a plan formulated to provide optimal nursing care services to a care recipient.

[0996] "Sharing" refers to the process of sharing and exchanging information such as generated care plans among stakeholders.

[0997] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[0998] System Configuration

[0999] The system includes the following major components:

[1000] 1. Generative AI engine: This is the main component for natural conversation with the care recipient. It uses Google Cloud Speech-to-Text API and OpenAI GPT model.

[1001] 2. Speech recognition module: This module converts the voice of the care recipient into text data. It uses the Google Cloud Speech-to-Text API, etc.

[1002] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses TensorFlow, OpenCV, etc.

[1003] 4. Emotion Engine: Algorithms and technologies for recognizing emotions from the voice and facial expression data of the care recipient. This includes tools for extracting emotions from voice and deep learning models for facial expression analysis.

[1004] 5. Database: Data storage for storing the care recipient's profile, health status, and history data. SQL or NoSQL databases are used.

[1005] 6. Feedback module: A program that generates advice and feedback for the care recipient. It works in conjunction with the generative AI engine.

[1006] 7. Alert module: This is a system for notifying caregivers and medical institutions when an abnormality is detected. It includes the function of sending emails and SMS.

[1007] 8. Big data analysis engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[1008] Program processing

[1009] The program processing of this system is as follows.

[1010] 1. The user installs and launches the application on their tablet device. The user creates an account by entering the required information (such as name, email address, and password), and then enters their profile information (e.g., name, age, gender, medical history, allergies, and current health condition).

[1011] 2. The device sends the entered information to the server, which stores the received information in a database. The server then sends a response to the device indicating that account creation is complete.

[1012] 3. The user launches the application and begins a conversation with the AI. The AI ​​greets the user via voice and text, saying, "Hello, Name. How are you today?" and asks about the user's health and wishes.

[1013] 4. When the user responds by voice, the device records the voice data and captures their facial expressions and movements with the camera. The recorded voice data is converted into text by the voice recognition module, and the image data is analyzed by the image analysis module.

[1014] 5. The device uses the emotion engine to analyze the voice and image data to identify the user's emotion, and the identified emotion data is sent to the server.

[1015] 6. The server combines the text data converted by the speech recognition module, the results of the image analysis module, and the emotion data, and analyzes them using a generative AI engine. The feedback module generates appropriate feedback based on the analysis results, and provides it to the user in the form of voice and text messages, such as "You seem tired. Please take a short rest."

[1016] 7. If the server detects an abnormality, the alert module will be activated and an alert will be sent to the caregiver or medical institution via email or SMS.

[1017] 8. The server continuously monitors the condition of the care recipient and stores the data in a database. The collected data is analyzed by a big data analysis engine to generate an optimal care plan. This care plan is shared with caregivers and medical institutions and used to improve services.

[1018] Specific examples

[1019] As a concrete example, consider a scenario in which a user installs an application, fills out a profile, and creates an account. In this system, the device sends the user's input information (e.g., name, age, medical history, etc.) to the server, which stores it in a database. When the user initiates a dialogue with the generative AI, the generative AI asks, "Hello, how are you today?" to which the user replies, "I'm not feeling well." The device records voice data and captures the user's facial expressions with a camera. This voice data is converted into text using the Google Cloud Speech-to-Text API, and the image data is analyzed using OpenCV. The emotion engine analyzes this data and identifies the user's emotions. The identified emotion data is sent to the server, which evaluates the care recipient's condition and generates appropriate feedback.

[1020] In this way, the present invention provides a specific means for understanding the emotions and health status of a care recipient in real time and providing individually optimized care services.

[1021] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1022] Step 1:

[1023] User: Installs and launches the application on the tablet device. After installing the application, the user launches the application for the first time and begins the account creation process by following the on-screen instructions.

[1024] Input: Tablet device, application installation file

[1025] Output: Installed applications

[1026] Step 2:

[1027] User: Enter information such as name, email address, and password on the account creation screen and tap the "Register" button.

[1028] Terminal: Sends the entered information to the server.

[1029] Input: Name, Email Address, Password

[1030] Output: Account information sent to the server

[1031] Step 3:

[1032] Server: Saves the received account information in a database and sends a response to the terminal indicating that the account has been created.

[1033] Input: Account information

[1034] Output: Account information stored in the database, response to the terminal

[1035] Step 4:

[1036] User: After creating an account, proceed to the screen where you enter your profile information, including your name, age, gender, medical history, allergies, current health condition, etc. Once you have completed the entry, tap the "Save" button.

[1037] Device: Sends the entered profile information to the server.

[1038] Input: Profile information

[1039] Output: Profile information sent to the server

[1040] Step 5:

[1041] Server: Saves the received profile information in a database and sends a response to the terminal indicating that the profile has been created.

[1042] Input: Profile information

[1043] Output: Profile information stored in the database, response to the device

[1044] Step 6:

[1045] User: Taps the "Start interactive session" button on the application's home screen.

[1046] Device: The generative AI engine will launch and say "Hello, Name. How are you today?" The same text will also be displayed on the screen.

[1047] Input: Button to start an interactive session, Generative AI engine

[1048] Output: Greeting speech and text

[1049] Step 7:

[1050] User: Answers questions from the generated AI by voice, such as about their physical condition and wishes.

[1051] Device: Records the user's voice data using a microphone.

[1052] Input: Voice response to generative AI

[1053] Output: Recorded audio data

[1054] Step 8:

[1055] Terminal: The camera simultaneously captures the user's facial expressions and movements, and the captured images are sent to the image analysis module in real time.

[1056] Input: User's facial expressions and movements

[1057] Output: Captured image data

[1058] Step 9:

[1059] On the device: The recorded voice data is converted into text by a speech recognition module, using the Google Cloud Speech-to-Text API.

[1060] Input: Recorded audio data

[1061] Output: Converted text data

[1062] Step 10:

[1063] Terminal: The captured image data is analyzed using an image analysis module (e.g., using TensorFlow or OpenCV).

[1064] Input: Captured image data

[1065] Output: Analyzed image data

[1066] Step 11:

[1067] Terminal: The emotion engine analyzes the voice and image data to identify the user's emotions.

[1068] Input: converted text data, analyzed image data

[1069] Output: Identified emotion data

[1070] Step 12:

[1071] Terminal: Sends the identified emotion data to the server.

[1072] Input: Identified emotion data

[1073] Output: Emotion data sent to the server

[1074] Step 13:

[1075] Server: The integrated data (text data converted by the voice recognition module, image analysis results, and emotion data) is analyzed using a generative AI engine to evaluate the health and emotional state of the care recipient.

[1076] Input: Text data, analysis results, emotion data

[1077] Output: Assessed health and emotional states

[1078] Step 14:

[1079] Server: The feedback module generates appropriate feedback based on the evaluation results (e.g., "You seem tired. Please take a rest.").

[1080] Input: Assessed health and emotional states

[1081] Output: Generated feedback

[1082] Step 15:

[1083] Terminal: Provides generated feedback to the user as voice and text messages.

[1084] Input: Generated feedback

[1085] Output: Feedback as audio and text messages

[1086] Step 16:

[1087] Server: If an abnormality is detected, the alert module is activated and notifies caregivers and medical institutions via email or SMS.

[1088] Input: Detected anomaly

[1089] Output: Alert notification to caregivers and medical institutions

[1090] Step 17:

[1091] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[1092] Input: Status data of care recipient

[1093] Output: Data stored in the database

[1094] Step 18:

[1095] Server: The big data analytics engine analyzes the accumulated data and finds trends and patterns.

[1096] Input: Accumulated data

[1097] Output: Trends and patterns as analytical results

[1098] Step 19:

[1099] Server: Generates optimal care plans based on the analysis results and shares them with caregivers and medical institutions.

[1100] Input: Trend and pattern analysis results

[1101] Output: Generated care plans and their sharing

[1102] The above are the specific processing steps of this system. This system makes it possible to accurately grasp the condition of the care recipient and provide prompt and appropriate care.

[1103] (Application example 2)

[1104] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1105] Conventional security systems have difficulty assessing a user's emotional state in real time, making it difficult to take appropriate action in emergencies. They also lack the means to provide immediate feedback or alerts to ensure user safety. In particular, there is a need for a system that can monitor a user's state over the long term and implement optimal security measures without compromising personal privacy.

[1106] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for communicating with the care recipient using generative AI technology, means for converting acquired voice data into text data, means for analyzing acquired image data, means for identifying emotions based on voice recognition and image analysis data, means for transmitting the text data, analysis results, and emotional data to a caregiver or a medical institution, and means for issuing an alert when an abnormality is detected. This allows the user's emotional state to be evaluated in real time and emergency contacts to be notified in the event of an abnormality, ensuring the user's safety and enabling optimal security measures to be taken through long-term data monitoring.

[1107] "Generative AI technology" is software technology that uses natural language processing and machine learning algorithms to engage in natural conversations with humans.

[1108] "Voice data" refers to the voice information input by the user, and is the data that is analyzed by the voice recognition module based on this information.

[1109] "Text data" refers to character information converted from voice data by a voice recognition module.

[1110] "Image data" refers to visual information captured by a camera of a user's facial expressions and movements.

[1111] "Voice recognition and image analysis data" refers to the results of analyzing acquired voice data and image data and converting them into various types of information.

[1112] "Means for identifying emotions" refers to a technology or system that analyzes audio and image data to assess a user's emotional state.

[1113] "Means of transmitting to caregivers or medical institutions" refers to technology that transmits user analysis data to caregivers or medical institutions in real time.

[1114] "Means for sending an alert" refers to a technology that sends warning information to a pre-set emergency contact when the system detects an abnormal condition.

[1115] "Big data analysis" is a technology that collects and analyzes large amounts of data and uses the results to derive new insights and patterns.

[1116] "Security measures" refer to the defensive measures and response procedures established to ensure user safety.

[1117] "Feedback" refers to advice and suggestions generated by the system based on the user's behavior and emotional state.

[1118] "Means for notifying emergency contacts" refers to technology that automatically notifies the set emergency contacts when the system detects an abnormality.

[1119] This invention relates to a security system that uses generative AI technology and an emotion engine to analyze user emotions in real time to improve safety. The system collects and analyzes voice and image data, and issues an emergency notification if it detects an abnormal emotional state.

[1120] System Configuration

[1121] The system consists of the following major components:

[1122] 1. Generative AI engine: This is the primary software for engaging in natural dialogue with users.

[1123] 2. Speech recognition module: Converts collected voice data into text data.

[1124] 3. Image analysis module: Analyzes the acquired image data and evaluates the user's facial expressions and movements.

[1125] 4. Emotion Engine: Identify user emotions from audio and image data.

[1126] 5. Database: Stores user profiles and historical data.

[1127] 6. Feedback module: Generates advice and feedback for the user.

[1128] 7. Alert module: Notifies emergency contacts when an abnormality is detected.

[1129] 8. Big Data Analysis Engine: Analyzes collected data and proposes security plans.

[1130] Hardware and Software Used

[1131] Hardware used

[1132] Smartphone: Equipped with a camera and microphone to collect audio and image data.

[1133] Server: Processes the collected data and stores the analysis results.

[1134] Software used

[1135] Speech Recognition Module: Uses Google Speech-to-Text API.

[1136] Image Analysis Module: Analyzes image data in real time using OpenCV.

[1137] Emotion Engine: Identifies emotions using Azure Emotion API or Hume AI.

[1138] Database: Use AWS DynamoDB or Firebase to store data.

[1139] Feedback module: Uses a custom message generation algorithm.

[1140] Alert Module: Uses the Twilio API to notify emergency contacts.

[1141] Details of data processing and calculation

[1142] Acquisition of audio and image data

[1143] Device: Records voice data from the user and captures facial expressions and movements with the camera. This data is converted to text using the Google Speech-to-Text API, and image data is analyzed using OpenCV.

[1144] Emotion Recognition in Action

[1145] On the device: The emotion engine analyzes the voice and image data to identify the user's emotion. This information is sent to the server.

[1146] Analyzing data and providing feedback

[1147] Server: Based on the results of voice and image analysis, the user's emotional state is evaluated, and the feedback module generates appropriate advice. Real-time voice and visual feedback is provided to the user.

[1148] Alert Notifications

[1149] Server: If an anomaly is detected, the alert module immediately sends a notification to emergency contacts.

[1150] Specific examples

[1151] If a user comes home late at night and is captured looking nervous:

[1152] Application: The generative AI asks, "Good work. You're feeling a little nervous today, aren't you?"

[1153] User: "Yes, I'm a little nervous."

[1154] Application: The emotion engine detects the user's anxiety, and the alert module notifies emergency contacts in real time.

[1155] Example of input prompt for generative AI model

[1156] Analyze the user's voice and facial expressions to assess their current emotional state. If anxiety or tension is increasing, generate appropriate feedback and alert emergency contacts if necessary.

[1157]

[1158] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1159] Step 1:

[1160] User registration and profile creation

[1161] User: Installs the application on a smartphone and launches it. Creates an account by entering name, email address, password, age, gender, and profile information. Specifically, enters the required information into the application's input form and presses the registration button.

[1162] Input: Name, email address, password, age, gender and profile information.

[1163] Terminal: Sends the entered information to the server. Specifically, it sends data to the server using an HTTP request.

[1164] Output: Registration data sent to the server.

[1165] Server: Stores the received information in a database and sends an account creation completion response to the terminal.

[1166] Output: Account creation complete response.

[1167] Step 2:

[1168] Start a conversation session with generative AI

[1169] User: Launches the application and begins interacting with the generation AI by pressing the start button on the application's main screen.

[1170] Input: Instruction to start a dialogue.

[1171] Terminal: The generated AI asks, "Hello, Name. How are you doing today?"

[1172] Output: Question from the generative AI.

[1173] User: Answers the generated AI verbally about their physical condition and emotions. Specifically, they speak into a microphone.

[1174] Input: User's voice data.

[1175] Step 3:

[1176] Acquisition of audio and image data

[1177] Device: Records voice data from the user and captures facial expressions and movements with a camera. Specifically, the device uses the smartphone's microphone and camera to capture voice and images.

[1178] Input: Audio data, image data.

[1179] Terminal: The voice recognition module converts the recorded voice data into text, and the image data is analyzed by the image analysis module.

[1180] Output: Text data, analysis results.

[1181] Step 4:

[1182] Emotion Recognition in Action

[1183] On the device: The emotion engine analyzes audio and image data to identify the user's emotions. Specifically, emotion analysis is performed using Azure Emotion API and Hume AI.

[1184] Input: Text data, image analysis data.

[1185] Output: Emotion data.

[1186] Terminal: Sends the identified emotion data to the server. Specifically, it sends the emotion data using an HTTP request.

[1187] Output: Emotion data transmission.

[1188] Step 5:

[1189] Analyzing data and providing feedback

[1190] Server: Analyzes data to assess the user's health and emotional state based on the results of the voice recognition and image analysis modules and emotional data. Specifically, it uses a generative AI engine to integrate and evaluate multiple data sources.

[1191] Input: text data, image analysis results, emotion data.

[1192] Output: Health status assessment, emotional status assessment.

[1193] Server: The feedback module generates appropriate feedback to the user, such as advice like "Take a short rest."

[1194] Output: Feedback.

[1195] Terminal: Provides generated feedback to the user in audio and visual form.

[1196] Output: Provide feedback.

[1197] Step 6:

[1198] Alerting and Continuous Monitoring

[1199] Server: If an anomaly is detected, the alert module notifies the user's emergency contacts by sending an alert via SMS or phone call using the Twilio API.

[1200] Input: Emotion data, evaluation results.

[1201] Output: Alert notification.

[1202] Server: Continuously monitors the user's status and accumulates the collected data in a database. Specifically, data is periodically added to the database.

[1203] Output: Database updated.

[1204] Step 7:

[1205] Big data analysis and security plan proposals

[1206] Server: The big data analysis engine analyzes the collected data and generates a security plan optimized for the user. Specifically, it integrates long-term data and analyzes user behavior patterns and emotional tendencies.

[1207] Input: Collected data.

[1208] Output: Optimized security plan.

[1209] Server: Shares the generated security plan with the user and their emergency contacts via email or in-app notification.

[1210] Output: Security plan share.

[1211] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1212] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1213] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1214] [Third embodiment]

[1215] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1216] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1217] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1218] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1219] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1221] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1222] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1223] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1224] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1225] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1226] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1227] This invention relates to a system that uses generative AI technology to provide personalized care services to individuals. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement appropriate care plans.

[1228] System Configuration

[1229] The system includes the following major components:

[1230] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[1231] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[1232] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[1233] 4. Database: Stores the care recipient's profile, health status, and history data.

[1234] 5. Feedback module: Generates advice and feedback for the care recipient.

[1235] 6. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[1236] 7. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[1237] Program processing

[1238] 1. User registration and profile creation

[1239] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[1240] Terminal: Sends the entered information to the server.

[1241] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[1242] 2. Start a conversation session with the generative AI

[1243] User: Launches the application and begins interacting with the generative AI.

[1244] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[1245] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[1246] 3. Acquisition of audio and image data

[1247] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[1248] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data. The captured image data is passed to the image analysis module, where facial expressions and movements are analyzed.

[1249] 4. Data analysis and feedback

[1250] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[1251] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data and image analysis results.

[1252] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[1253] Device: Provides audio and visual feedback to the care recipient.

[1254] 5. Alert Notifications and Continuous Monitoring

[1255] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[1256] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[1257] 6. Big data analysis and care plan proposals

[1258] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1259] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[1260] Specific examples

[1261] 1. User registration and profile creation

[1262] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[1263] 2. Start a conversation session with the generative AI

[1264] User: Launches the application and begins a dialogue with the AI ​​generator. The AI ​​generator asks, "How are you feeling today?", to which the user responds, "I'm not feeling too well."

[1265] 3. Acquisition of audio and image data

[1266] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[1267] 4. Data analysis and feedback

[1268] Device: The text data and image analysis results are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[1269] 5. Alert Notifications and Continuous Monitoring

[1270] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[1271] 6. Big data analysis and care plan proposals

[1272] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve the plan.

[1273] The above is a description of the mode for carrying out the invention. It is expected that this system will provide individually optimized care and reduce the burden on caregivers.

[1274] The processing flow will be explained below.

[1275] Step 1: User Registration

[1276] User: Installs the application on the tablet device and launches it.

[1277] Device: Displays a registration screen and prompts the user to create an account.

[1278] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[1279] Terminal: Sends the entered information to the server.

[1280] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[1281] Step 2: Create your profile

[1282] Device: Displays a profile creation screen and prompts the user to enter personal information.

[1283] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[1284] Terminal: Sends the entered information to the server.

[1285] Server: The received information is linked to the user account and stored in a database.

[1286] Step 3: Start interacting with generative AI

[1287] User: Launches the application and begins an interactive session with the generative AI.

[1288] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[1289] Step 4: Acquiring audio and visual data

[1290] User: Tells the generated AI about their physical condition, wishes, etc.

[1291] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[1292] Step 5: Convert audio data to text

[1293] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[1294] Step 6: Analyzing the image data

[1295] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[1296] Step 7: Send data to the server

[1297] Terminal: Sends text data generated by voice recognition and image analysis results to the server.

[1298] Step 8: Analysis by generative AI

[1299] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on the text data and image analysis results.

[1300] Server: Stores the evaluation results in a database.

[1301] Step 9: Generate and provide feedback

[1302] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[1303] Server: Sends the generated feedback to the device.

[1304] Device: Provides audio and visual feedback to the care recipient.

[1305] Step 10: Alert Notification

[1306] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[1307] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[1308] Step 11: Continuous monitoring and data collection

[1309] Server: Continuously collects data from the care recipient and stores it as a log.

[1310] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[1311] Step 12: Big data analysis and care plan proposal

[1312] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1313] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[1314] Example 1

[1315] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1316] In an aging society, there is a need to improve the quality of nursing care services, provide personalized care that meets the individual needs of care recipients, and reduce the burden on caregivers and medical institutions. However, conventional systems have difficulty accurately understanding the health and emotional state of care recipients in real time and providing appropriate feedback and care plans. As a result, there has been a lack of efficient means to improve the quality of nursing care services.

[1317] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1318] In this invention, the server includes: means for communicating with the care recipient using AI technology; means for converting acquired voice data into text data; means for analyzing acquired image data; means for transmitting the text data and analysis results; means for issuing an alert when an abnormality is detected; means for a user to input voice and image data; means for transmitting the voice and image data to the server; means for generating feedback for the care recipient using AI technology; means for providing the feedback in audio or visual form; means for generating feedback based on data received by the AI ​​from the server; means for transmitting the feedback from the server to a terminal; means for analyzing big data and proposing an optimized care plan for the care recipient; means for sharing the care plan with a caregiver or a medical institution; means for analyzing the acquired big data; means for generating a care plan based on the analysis results; and means for notifying the caregiver or a medical institution of the care plan from the server. This enables real-time monitoring of the condition of the care recipient and providing appropriate feedback and a personalized care plan.

[1319] "Generative AI technology" is a technology that uses natural language processing and machine learning algorithms to engage in natural dialogue with humans and generate appropriate responses and information.

[1320] "Care recipients" refers to elderly people or individuals with physical disabilities who require nursing or medical assistance.

[1321] "Means of communication" refers to the method by which the system has the functionality to exchange information with the care recipient via voice or text.

[1322] "Audio data" refers to information that has been digitally recorded from the voice of the person receiving care.

[1323] "Text data" refers to information in which voice data is expressed as a string of characters.

[1324] The term "means for converting voice data into text data" refers to a method for converting voice data into corresponding text data using voice recognition technology.

[1325] "Image data" refers to information that digitally records the facial expressions and movements of the person receiving care.

[1326] "Means for analyzing image data" refers to a method of using image processing technology to recognize the facial expressions and movements of the person being cared for and analyze their meaning.

[1327] "Means for transmitting text data and analysis results" refers to the communication function for transmitting the text data and image analysis results acquired by the system to a remote server or other device.

[1328] "Means for sending an alert" refers to a method that has the function of sending a notification when an abnormality is detected and warning caregivers and medical institutions.

[1329] "Means for inputting voice and image data" refers to a method having the function of capturing voice and image data from the care recipient and inputting them into the system.

[1330] "Means by which the generating AI generates feedback based on data received from the server" refers to a method by which the generating AI creates appropriate feedback for the care recipient based on data received from the server.

[1331] "Means for providing audio or visual feedback" refers to a method that has the function of providing the generated feedback to the care recipient as audio and images.

[1332] "Means of analyzing big data" refers to methods that use technology to analyze large amounts of data and identify patterns and trends.

[1333] "Means for proposing an optimized care plan" refers to a method that has the function of creating and proposing an optimal care plan for a care recipient based on the analysis results.

[1334] "Means for sharing a care plan with a caregiver or a medical institution" refers to a method that has a function for sharing a generated care plan with a caregiver or a medical institution through communication.

[1335] A "server" is a computing device that serves as the core of a system and plays a central role in storing, analyzing, and transmitting data.

[1336] "Terminal" refers to a device (e.g., tablet, smartphone) that a user directly operates to communicate.

[1337] These definitions provide a clear understanding of the technical scope and function of the invention and serve as a basis for patent prosecution and enforcement.

[1338] This invention relates to a system that provides personalized care services to individuals using generative AI technology. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement an appropriate care plan. The system includes the following main components:

[1339] System Configuration

[1340] 1. Generative AI Engine: This is the main software component for natural dialogue with the care recipient. The generative AI engine uses natural language processing and machine learning algorithms.

[1341] 2. Speech recognition module: This module recognizes the voice of the care recipient and converts it into text data. Specifically, it uses the Google Cloud Speech-to-Text API.

[1342] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses OpenCV, TensorFlow, etc.

[1343] 4. Database: A database system for storing the care recipient's profile, health status, and past history data. MongoDB or MySQL is used.

[1344] 5. Feedback module: This module generates advice and feedback for the care recipient. It works in conjunction with the generation AI engine to generate appropriate feedback.

[1345] 6. Alert module: This module sends alerts to caregivers and medical institutions when an abnormality is detected. It utilizes Twilio, SMTP, etc.

[1346] 7. Big Data Analysis Engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[1347] Program processing

[1348] 1. User registration and profile creation

[1349] User: Installs the application on a tablet device and launches it. When launching the application for the first time, the user enters the required information (name, email address, password, age, gender, medical history, allergies, current health condition) and creates a profile.

[1350] Terminal: Sends the entered information to the server. HTTPS is used as the communication protocol.

[1351] Server: Stores the received data in a database and notifies the device that account creation is complete.

[1352] 2. Start a conversation session with the generative AI

[1353] User: Launches the application and taps the "Start Interaction" button.

[1354] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[1355] User: Answers the generated AI verbally about their physical condition and wishes.

[1356] 3. Acquisition of audio and image data

[1357] Device: Records the user's voice data and captures facial expressions and movements with a camera.

[1358] Terminal: Sends voice data to the voice recognition module, which converts the voice into text data. Also sends captured image data to the image analysis module, which analyzes facial expressions and movements.

[1359] 4. Data analysis and feedback

[1360] Terminal: Transmits the converted text data and image analysis results to the server.

[1361] Server: The generation AI engine evaluates the health and emotional state of the care recipient based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[1362] Terminal: The generated feedback is converted into speech using a speech synthesis engine, and is played back to the user and also displayed on the screen as a text message.

[1363] 5. Alert Notifications and Continuous Monitoring

[1364] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[1365] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[1366] 6. Big data analysis and care plan proposals

[1367] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1368] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to an electronic health record (EHR) system.

[1369] Specific examples

[1370] User: The application starts a dialogue, and the generative AI asks, "How is your day going?" The user responds verbally, "I'm feeling a little tired today."

[1371] Device: Records voice data and converts it into "I'm a little tired today" using a voice recognition module. Captures the user's facial expression using a camera, and analyzes the tired facial data using an image analysis module.

[1372] Server: Based on the received data, the generated AI evaluates the level of fatigue and generates feedback such as, "You seem tired. Please take a short rest."

[1373] Device: The feedback is synthesized into speech and played to the user. The same message is also displayed as text on the screen.

[1374] Prompt Sentence Examples

[1375] "Please explain how the generative AI engine assesses the emotional state of the care recipient."

[1376] "Please explain the process for alert notification when the system detects an abnormality."

[1377] These specific embodiments enable the system to provide high-quality, personalized care services to care recipients, thereby reducing the burden on caregivers and medical institutions.

[1378] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1379] Step 1: Register and create your profile

[1380] User: Installs the application on a tablet device and launches it by tapping the icon. On the sign-up screen, the user enters their name, email address, and password, then presses the "Next" button. Next, the user enters profile information such as age, gender, medical history, allergies, and current health condition, and presses the "Submit" button.

[1381] Input: Name, email address, password, age, gender, medical history, allergies, health status, and other information.

[1382] Output: The user's account information is saved in the database.

[1383] Terminal: The information entered by the user is sent to the server using the HTTPS protocol. If the transmission is successful, a confirmation message saying "Your account has been created" is displayed.

[1384] Server: Saves the received user information in the database. After confirming that the data was saved correctly, it sends a response to the device indicating that the account creation is complete.

[1385] Step 2: Start a conversation session with the generative AI

[1386] User: Launches the application and taps the "Start Interaction" button.

[1387] Input: The action to start the interaction.

[1388] Output: A greeting and question from the generated AI are displayed.

[1389] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[1390] User: Answers the generated AI verbally about their physical condition and wishes.

[1391] Step 3: Acquiring audio and image data

[1392] Device: Records the user's voice data through a microphone and captures facial expressions and movements with a camera.

[1393] Input: User's voice and image data.

[1394] Output: Data transfer to speech recognition module and image analysis module.

[1395] Terminal: The recorded voice data is sent to the voice recognition module, which then converts the voice into text. The captured image data is also sent to the image analysis module, which analyzes facial expressions and movements.

[1396] Step 4: Analyze data and provide feedback

[1397] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[1398] Input: Text data converted by the speech recognition module and the results of the image analysis module.

[1399] Output: Feedback generation.

[1400] Server: The generation AI engine evaluates the user's health and emotional state based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[1401] Terminal: The generated feedback is converted into speech by a speech synthesis engine and played back to the user. It is also displayed on the screen as a text message.

[1402] Step 5: Alerting and continuous monitoring

[1403] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[1404] Input: Data in which an anomaly was detected.

[1405] Output: Alert notification.

[1406] Server: Collects data to continuously monitor the user's status and periodically stores it in a database.

[1407] Step 6: Big data analysis and proposing care plans

[1408] Server: The big data analysis engine analyzes the accumulated data and generates a care plan optimized for the user.

[1409] Input: Accumulated big data.

[1410] Output: Optimized care plan.

[1411] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to the Electronic Health Record (EHR) system.

[1412] (Application example 1)

[1413] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1414] In modern society, providing personalized meal plans based on an individual's health status and dietary preferences is extremely important. In particular, food delivery services can provide greater convenience and satisfaction by suggesting optimal meals that take into account the user's health status and dietary restrictions. However, conventional systems have difficulty achieving such high levels of personalization, and there is a lack of technology to provide meal plans tailored to the user's health status.

[1415] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1416] In this invention, the server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and the analysis results and transmitting the plan to a delivery company, and a means for issuing an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[1417] "Generative AI technology" is a technology that uses natural language processing and machine learning models to interact with users and analyze data.

[1418] "Means of communication" refers to devices or programs that use generative AI technology to engage in voice or text dialogue with users.

[1419] The "means for converting acquired voice data into text data" refers to a device or program for converting a user's voice into text format using voice recognition technology.

[1420] "Means for analyzing acquired image data" refers to devices or programs that recognize and analyze the user's facial expressions and movements captured using image analysis technology.

[1421] The "means for proposing a personalized meal plan" is a device or program for generating and proposing an optimal meal plan based on the user's health condition and dietary preferences.

[1422] The "means for communicating to the delivery company" refers to a device or program for notifying the delivery company of the generated meal plan and making meal arrangements.

[1423] The "means for issuing an alert when an abnormality is detected" refers to a device or program for issuing a warning when an abnormality is detected in the user's health condition.

[1424] A "means for generating feedback" is a device or program that automatically generates appropriate advice or suggestions based on the user's input data.

[1425] "Audio or visual providing means" refers to a device or program for providing the generated feedback to the user audio or visually.

[1426] "Means for analyzing big data" refers to devices and programs that collect and analyze large amounts of data and extract useful information based on user behavior and health status.

[1427] The "means for proposing an optimized meal plan" refers to a device or program that uses the results of analyzing large amounts of data to generate and propose an optimal meal plan that meets the individual needs of a user.

[1428] "Means for sharing" refers to a device or program for sharing the generated meal plan and feedback with other parties or systems in collaboration with them.

[1429] This invention relates to a system that uses generative AI technology to propose personalized meal plans and arrange food delivery. The system aims to provide optimal meal plans based on the user's health condition and dietary preferences, and to realize these plans in cooperation with delivery companies.

[1430] System Configuration

[1431] The system includes the following major components:

[1432] 1. Generative AI engine:

[1433] It uses natural language processing and machine learning models (e.g., GPT-4) to interact with users and analyze their needs and health status.

[1434] 2. Speech Recognition Module:

[1435] The user's voice is converted into text data using the Google Speech-to-Text API or similar.

[1436] 3. Image Analysis Module:

[1437] Using OpenCV, TensorFlow, etc., the system analyzes the user's facial expressions and movements to assess their health condition.

[1438] 4. Database:

[1439] It stores user profiles, health status, dietary preferences, and historical data in databases such as MySQL and MongoDB.

[1440] 5. Feedback module:

[1441] Use a chatbot framework (such as Dialogflow) to generate feedback and advice for the user and provide it in audio or visual form.

[1442] 6. Alerts Module:

[1443] If an abnormality is detected, an alert will be sent to prompt appropriate action.

[1444] 7. Big Data Analysis Engine:

[1445] It analyzes the large amount of data collected and proposes an optimized meal plan to the user.

[1446] Program processing

[1447] The server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and analysis results, a means for transmitting the plan to a delivery company, and a means for sending an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[1448] Specific example explanation

[1449] A user can use their smartphone to start a conversation with the generative AI. For example, if they ask, "What would you recommend for lunch?", the generative AI will respond with, "Hello, Name. You seem a little tired today. I'd like to suggest a healthy salad and soup."

[1450] Example prompt sentence:

[1451] User: What would you recommend for lunch?

[1452] Generative AI: Hello, Name. You seem a little tired today. I suggest a healthy salad and soup.

[1453] The user's voice data is converted to text using the Google Speech-to-Text API, and their facial expressions and movements are analyzed using OpenCV and TensorFlow. This data is sent to a server where a generative AI engine analyzes it and proposes the optimal meal plan for the user. The proposed plan is then shared with the delivery company, and delivery is arranged.

[1454] For example, if a user responds, "I'm not feeling well," the generative AI will respond with, "I'll suggest a porridge that's easy to digest," providing appropriate feedback. If an abnormality is detected, an alert will be sent, prompting the user to take appropriate action.

[1455] This system will enable advanced personalization based on the user's health condition and individual needs, significantly improving the quality of food delivery services.

[1456] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1457] Step 1:

[1458] User registration and profile creation

[1459] The user installs and launches the application on their smartphone. They create an account by entering information such as their name, email address, password, health status, dietary preferences, and allergies. The entered information is sent by the device to the server. The server stores the received information in a database and sends a response to the device indicating that the account has been created.

[1460] Step 2:

[1461] Start a conversation session with generative AI

[1462] The user launches the application and begins a dialogue with the generation AI. The generation AI asks on the device, "Hello, Name. What kind of meal would you like to have today?" The user responds verbally about their meal preferences and physical condition. Input: User's voice data. Output: Text data about their meal preferences.

[1463] Step 3:

[1464] Acquisition of audio and image data

[1465] The device records voice data from the user and captures facial expressions and movements with a camera. The recorded voice data is converted into text data using a voice recognition module (Google Speech-to-Text API). The captured image data is analyzed for the user's facial expressions and movements using an image analysis module (OpenCV, TensorFlow). Input: User's voice data and image data. Output: Text data and analysis results.

[1466] Step 4:

[1467] Data analysis and meal plan suggestions

[1468] The device sends the converted text data and image analysis results to the server. The server analyzes this data using a generative AI engine (GPT-4) to evaluate the user's health condition and preferences. The server generates a personalized meal plan based on the evaluation results. The proposed meal plan is presented by a feedback module (Chatbot framework: Dialogflow) with the message, "For today's meals, we suggest these to suit your physical condition." Input: Text data and image analysis results. Output: Personalized meal plan.

[1469] Step 5:

[1470] Delivery arrangements and alert notifications

[1471] The server arranges an order with a delivery company based on the generated meal plan. The user's meal plan is notified to the delivery company, which then arranges delivery. If an abnormality is detected, the server also sends an alert to caregivers and medical institutions via the alert module. Input: Generated meal plan. Output: Order to delivery company, and alert notification as necessary.

[1472] Step 6:

[1473] Big data analysis and service improvement

[1474] The server analyzes the collected data using a big data analysis engine and improves the algorithm that optimizes each user's meal plan. Furthermore, the service is continuously improved based on user feedback and health status. This makes it possible to propose meal plans that are optimized for each user. Input: Large amounts of analyzed data. Output: Optimized meal plans and improved algorithms.

[1475] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1476] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[1477] System Configuration

[1478] The system includes the following major components:

[1479] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[1480] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[1481] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[1482] 4. Emotion engine: Recognizes emotions from the care recipient's voice and facial expression data.

[1483] 5. Database: Stores the profile, health status and history data of the care recipient.

[1484] 6. Feedback module: Generates advice and feedback for the care recipient.

[1485] 7. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[1486] 8. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[1487] Program processing

[1488] 1. User registration and profile creation

[1489] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[1490] Terminal: Sends the entered information to the server.

[1491] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[1492] 2. Start a conversation session with the generative AI

[1493] User: Launches the application and begins interacting with the generative AI.

[1494] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[1495] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[1496] 3. Acquisition of audio and image data

[1497] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[1498] Terminal: The recorded voice data is converted into text by the voice recognition module, and the image data is analyzed by the image analysis module.

[1499] 4. Emotion Recognition

[1500] Terminal: The emotion engine analyzes the voice and image data to identify the emotions of the person receiving care.

[1501] Terminal: Sends the identified emotion data to the server.

[1502] 5. Data analysis and feedback

[1503] Terminal: Sends the text data converted by the voice recognition module, the results of the image analysis module, and the data from the emotion engine to the server.

[1504] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[1505] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[1506] Device: Provides audio and visual feedback to the care recipient.

[1507] 6. Alert Notifications and Continuous Monitoring

[1508] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[1509] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[1510] 7. Big data analysis and care plan proposals

[1511] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1512] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[1513] Specific examples

[1514] 1. User registration and profile creation

[1515] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[1516] 2. Start a conversation session with the generative AI

[1517] User: Launches the application and begins a conversation with the AI ​​generator. The AI ​​generator asks, "Hello, how are you today?", to which the user responds, "I'm not feeling too well."

[1518] 3. Acquisition of audio and image data

[1519] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[1520] 4. Emotion Recognition

[1521] Terminal: The emotion engine analyzes the voice and image data to identify the user's emotion. The emotion data is sent to the server.

[1522] 5. Data analysis and feedback

[1523] Device: Text data, image analysis results, and emotion data are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[1524] 6. Alert Notifications and Continuous Monitoring

[1525] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[1526] 7. Big data analysis and care plan proposals

[1527] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve services.

[1528] This concludes the description of the embodiment of the invention. It is expected that this system will provide more individually optimized care through emotion recognition and reduce the burden on caregivers.

[1529] The processing flow will be explained below.

[1530] Step 1: User Registration

[1531] User: Installs the application on the tablet device and launches it.

[1532] Device: Displays a registration screen and prompts the user to create an account.

[1533] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[1534] Terminal: Sends the entered information to the server.

[1535] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[1536] Step 2: Create your profile

[1537] Device: Displays a profile creation screen and prompts the user to enter personal information.

[1538] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[1539] Terminal: Sends the entered information to the server.

[1540] Server: The received information is linked to the user account and stored in a database.

[1541] Step 3: Start interacting with generative AI

[1542] User: Launches the application and begins an interactive session with the generative AI.

[1543] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[1544] Step 4: Acquiring audio and visual data

[1545] User: Tells the generated AI about their physical condition, wishes, etc.

[1546] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[1547] Step 5: Convert audio data to text

[1548] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[1549] Step 6: Analyzing the image data

[1550] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[1551] Step 7: Emotion Recognition

[1552] Terminal: Passes image analysis and voice recognition data to the emotion engine to recognize the user's emotions.

[1553] Terminal: Sends the recognized emotion data to the server.

[1554] Step 8: Send data to the server

[1555] Terminal: Sends text data generated by voice recognition, image analysis results, and emotion data to the server.

[1556] Step 9: Analysis by generative AI

[1557] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[1558] Server: Stores the evaluation results in a database.

[1559] Step 10: Generate and provide feedback

[1560] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[1561] Server: Sends the generated feedback to the device.

[1562] Device: Provides audio and visual feedback to the care recipient.

[1563] Step 11: Alert Notification

[1564] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[1565] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[1566] Step 12: Continuous monitoring and data collection

[1567] Server: Continuously collects data from the care recipient and stores it as a log.

[1568] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[1569] Step 13: Big data analysis and care plan proposal

[1570] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1571] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[1572] Example 2

[1573] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1574] In today's nursing care industry, there is a need to accurately grasp the physical and mental state of care recipients and provide appropriate care tailored to their individual needs and emotions. However, conventional systems lack the technology to analyze the emotions and health status of care recipients in real time and provide optimal feedback and care plans, which contributes to the increased burden on caregivers. Furthermore, their ability to effectively analyze large amounts of data and issue appropriate alerts is limited.

[1575] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1576] In this invention, the server includes means for communicating with the care recipient using generative AI technology, means for converting acquired voice data into text data, means for analyzing acquired image data, means for recognizing emotions using an emotion engine, means for transmitting the text data, analysis results, and emotion data to a caregiver or a medical institution, and means for issuing an alert when an abnormality is detected. This makes it possible to grasp the emotional and health state of the care recipient in real time, provide optimal feedback and care plans, and immediately detect and notify abnormalities.

[1577] "Generative AI technology" refers to technology that uses artificial intelligence to generate natural language and engage in dialogue.

[1578] "Communication" refers to the process of engaging in voice and text dialogue with the care recipient through generative AI technology.

[1579] "Audio data" refers to the voice of the care recipient recorded through a microphone and expressed in digital form.

[1580] "Text data" refers to data that has been converted from voice data using a voice recognition module so that it can be handled as text information.

[1581] "Image data" refers to visual information captured through a camera or other image input device.

[1582] "Analysis" is the process of processing collected data using specific algorithms to extract meaningful information.

[1583] An "emotion engine" refers to algorithms and technologies for recognizing the emotional state of a care recipient from voice and image data.

[1584] "Emotion data" refers to information indicating an emotional state output by the emotion engine after analyzing audio and image data.

[1585] "Communication" refers to the process of sending and sharing the generated text data, analysis results, and emotional data with caregivers or medical institutions.

[1586] An "alert" is a notification sent to a caregiver or medical institution as a warning when an abnormality is detected.

[1587] "Feedback" refers to advice and comments that the generative AI technology provides to the care recipient based on analyzed data.

[1588] "Big data" refers to large data sets and the techniques and methods for analyzing them to extract useful information.

[1589] A "care plan" is a plan formulated to provide optimal nursing care services to a care recipient.

[1590] "Sharing" refers to the process of sharing and exchanging information such as generated care plans among stakeholders.

[1591] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[1592] System Configuration

[1593] The system includes the following major components:

[1594] 1. Generative AI engine: This is the main component for natural conversation with the care recipient. It uses Google Cloud Speech-to-Text API and OpenAI GPT model.

[1595] 2. Speech recognition module: This module converts the voice of the care recipient into text data. It uses the Google Cloud Speech-to-Text API, etc.

[1596] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses TensorFlow, OpenCV, etc.

[1597] 4. Emotion Engine: Algorithms and technologies for recognizing emotions from the voice and facial expression data of the care recipient. This includes tools for extracting emotions from voice and deep learning models for facial expression analysis.

[1598] 5. Database: Data storage for storing the care recipient's profile, health status, and history data. SQL or NoSQL databases are used.

[1599] 6. Feedback module: A program that generates advice and feedback for the care recipient. It works in conjunction with the generative AI engine.

[1600] 7. Alert module: This is a system for notifying caregivers and medical institutions when an abnormality is detected. It includes the function of sending emails and SMS.

[1601] 8. Big data analysis engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[1602] Program processing

[1603] The program processing of this system is as follows.

[1604] 1. The user installs and launches the application on their tablet device. The user creates an account by entering the required information (such as name, email address, and password), and then enters their profile information (e.g., name, age, gender, medical history, allergies, and current health condition).

[1605] 2. The device sends the entered information to the server, which stores the received information in a database. The server then sends a response to the device indicating that account creation is complete.

[1606] 3. The user launches the application and begins a conversation with the AI. The AI ​​greets the user via voice and text, saying, "Hello, Name. How are you today?" and asks about the user's health and wishes.

[1607] 4. When the user responds by voice, the device records the voice data and captures their facial expressions and movements with the camera. The recorded voice data is converted into text by the voice recognition module, and the image data is analyzed by the image analysis module.

[1608] 5. The device uses the emotion engine to analyze the voice and image data to identify the user's emotion, and the identified emotion data is sent to the server.

[1609] 6. The server combines the text data converted by the speech recognition module, the results of the image analysis module, and the emotion data, and analyzes them using a generative AI engine. The feedback module generates appropriate feedback based on the analysis results, and provides it to the user in the form of voice and text messages, such as "You seem tired. Please take a short rest."

[1610] 7. If the server detects an abnormality, the alert module will be activated and an alert will be sent to the caregiver or medical institution via email or SMS.

[1611] 8. The server continuously monitors the condition of the care recipient and stores the data in a database. The collected data is analyzed by a big data analysis engine to generate an optimal care plan. This care plan is shared with caregivers and medical institutions and used to improve services.

[1612] Specific examples

[1613] As a concrete example, consider a scenario in which a user installs an application, fills out a profile, and creates an account. In this system, the device sends the user's input information (e.g., name, age, medical history, etc.) to the server, which stores it in a database. When the user initiates a dialogue with the generative AI, the generative AI asks, "Hello, how are you today?" to which the user replies, "I'm not feeling well." The device records voice data and captures the user's facial expressions with a camera. This voice data is converted into text using the Google Cloud Speech-to-Text API, and the image data is analyzed using OpenCV. The emotion engine analyzes this data and identifies the user's emotions. The identified emotion data is sent to the server, which evaluates the care recipient's condition and generates appropriate feedback.

[1614] In this way, the present invention provides a specific means for understanding the emotions and health status of a care recipient in real time and providing individually optimized care services.

[1615] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1616] Step 1:

[1617] User: Installs and launches the application on the tablet device. After installing the application, the user launches the application for the first time and begins the account creation process by following the on-screen instructions.

[1618] Input: Tablet device, application installation file

[1619] Output: Installed applications

[1620] Step 2:

[1621] User: Enter information such as name, email address, and password on the account creation screen and tap the "Register" button.

[1622] Terminal: Sends the entered information to the server.

[1623] Input: Name, Email Address, Password

[1624] Output: Account information sent to the server

[1625] Step 3:

[1626] Server: Saves the received account information in a database and sends a response to the terminal indicating that the account has been created.

[1627] Input: Account information

[1628] Output: Account information stored in the database, response to the terminal

[1629] Step 4:

[1630] User: After creating an account, proceed to the screen where you enter your profile information, including your name, age, gender, medical history, allergies, current health condition, etc. Once you have completed the entry, tap the "Save" button.

[1631] Device: Sends the entered profile information to the server.

[1632] Input: Profile information

[1633] Output: Profile information sent to the server

[1634] Step 5:

[1635] Server: Saves the received profile information in a database and sends a response to the terminal indicating that the profile has been created.

[1636] Input: Profile information

[1637] Output: Profile information stored in the database, response to the device

[1638] Step 6:

[1639] User: Taps the "Start interactive session" button on the application's home screen.

[1640] Device: The generative AI engine will launch and say "Hello, Name. How are you today?" The same text will also be displayed on the screen.

[1641] Input: Button to start an interactive session, Generative AI engine

[1642] Output: Greeting speech and text

[1643] Step 7:

[1644] User: Answers questions from the generated AI by voice, such as about their physical condition and wishes.

[1645] Device: Records the user's voice data using a microphone.

[1646] Input: Voice response to generative AI

[1647] Output: Recorded audio data

[1648] Step 8:

[1649] Terminal: The camera simultaneously captures the user's facial expressions and movements, and the captured images are sent to the image analysis module in real time.

[1650] Input: User's facial expressions and movements

[1651] Output: Captured image data

[1652] Step 9:

[1653] On the device: The recorded voice data is converted into text by a speech recognition module, using the Google Cloud Speech-to-Text API.

[1654] Input: Recorded audio data

[1655] Output: Converted text data

[1656] Step 10:

[1657] Terminal: The captured image data is analyzed using an image analysis module (e.g., using TensorFlow or OpenCV).

[1658] Input: Captured image data

[1659] Output: Analyzed image data

[1660] Step 11:

[1661] Terminal: The emotion engine analyzes the voice and image data to identify the user's emotions.

[1662] Input: converted text data, analyzed image data

[1663] Output: Identified emotion data

[1664] Step 12:

[1665] Terminal: Sends the identified emotion data to the server.

[1666] Input: Identified emotion data

[1667] Output: Emotion data sent to the server

[1668] Step 13:

[1669] Server: The integrated data (text data converted by the voice recognition module, image analysis results, and emotion data) is analyzed using a generative AI engine to evaluate the health and emotional state of the care recipient.

[1670] Input: Text data, analysis results, emotion data

[1671] Output: Assessed health and emotional states

[1672] Step 14:

[1673] Server: The feedback module generates appropriate feedback based on the evaluation results (e.g., "You seem tired. Please take a rest.").

[1674] Input: Assessed health and emotional states

[1675] Output: Generated feedback

[1676] Step 15:

[1677] Terminal: Provides generated feedback to the user as voice and text messages.

[1678] Input: Generated feedback

[1679] Output: Feedback as audio and text messages

[1680] Step 16:

[1681] Server: If an abnormality is detected, the alert module is activated and notifies caregivers and medical institutions via email or SMS.

[1682] Input: Detected anomaly

[1683] Output: Alert notification to caregivers and medical institutions

[1684] Step 17:

[1685] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[1686] Input: Status data of care recipient

[1687] Output: Data stored in the database

[1688] Step 18:

[1689] Server: The big data analytics engine analyzes the accumulated data and finds trends and patterns.

[1690] Input: Accumulated data

[1691] Output: Trends and patterns as analytical results

[1692] Step 19:

[1693] Server: Generates optimal care plans based on the analysis results and shares them with caregivers and medical institutions.

[1694] Input: Trend and pattern analysis results

[1695] Output: Generated care plans and their sharing

[1696] The above are the specific processing steps of this system. This system makes it possible to accurately grasp the condition of the care recipient and provide prompt and appropriate care.

[1697] (Application example 2)

[1698] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1699] Conventional security systems have difficulty assessing a user's emotional state in real time, making it difficult to take appropriate action in emergencies. They also lack the means to provide immediate feedback or alerts to ensure user safety. In particular, there is a need for a system that can monitor a user's state over the long term and implement optimal security measures without compromising personal privacy.

[1700] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for communicating with the care recipient using generative AI technology, means for converting acquired voice data into text data, means for analyzing acquired image data, means for identifying emotions based on voice recognition and image analysis data, means for transmitting the text data, analysis results, and emotional data to a caregiver or a medical institution, and means for issuing an alert when an abnormality is detected. This allows the user's emotional state to be evaluated in real time and emergency contacts to be notified in the event of an abnormality, ensuring the user's safety and enabling optimal security measures to be taken through long-term data monitoring.

[1701] "Generative AI technology" is software technology that uses natural language processing and machine learning algorithms to engage in natural conversations with humans.

[1702] "Voice data" refers to the voice information input by the user, and is the data that is analyzed by the voice recognition module based on this information.

[1703] "Text data" refers to character information converted from voice data by a voice recognition module.

[1704] "Image data" refers to visual information captured by a camera of a user's facial expressions and movements.

[1705] "Voice recognition and image analysis data" refers to the results of analyzing acquired voice data and image data and converting them into various types of information.

[1706] "Means for identifying emotions" refers to a technology or system that analyzes audio and image data to assess a user's emotional state.

[1707] "Means of transmitting to caregivers or medical institutions" refers to technology that transmits user analysis data to caregivers or medical institutions in real time.

[1708] "Means for sending an alert" refers to a technology that sends warning information to a pre-set emergency contact when the system detects an abnormal condition.

[1709] "Big data analysis" is a technology that collects and analyzes large amounts of data and uses the results to derive new insights and patterns.

[1710] "Security measures" refer to the defensive measures and response procedures established to ensure user safety.

[1711] "Feedback" refers to advice and suggestions generated by the system based on the user's behavior and emotional state.

[1712] "Means for notifying emergency contacts" refers to technology that automatically notifies the set emergency contacts when the system detects an abnormality.

[1713] This invention relates to a security system that uses generative AI technology and an emotion engine to analyze user emotions in real time to improve safety. The system collects and analyzes voice and image data, and issues an emergency notification if it detects an abnormal emotional state.

[1714] System Configuration

[1715] The system consists of the following major components:

[1716] 1. Generative AI engine: This is the primary software for engaging in natural dialogue with users.

[1717] 2. Speech recognition module: Converts collected voice data into text data.

[1718] 3. Image analysis module: Analyzes the acquired image data and evaluates the user's facial expressions and movements.

[1719] 4. Emotion Engine: Identify user emotions from audio and image data.

[1720] 5. Database: Stores user profiles and historical data.

[1721] 6. Feedback module: Generates advice and feedback for the user.

[1722] 7. Alert module: Notifies emergency contacts when an abnormality is detected.

[1723] 8. Big Data Analysis Engine: Analyzes collected data and proposes security plans.

[1724] Hardware and Software Used

[1725] Hardware used

[1726] Smartphone: Equipped with a camera and microphone to collect audio and image data.

[1727] Server: Processes the collected data and stores the analysis results.

[1728] Software used

[1729] Speech Recognition Module: Uses Google Speech-to-Text API.

[1730] Image Analysis Module: Analyzes image data in real time using OpenCV.

[1731] Emotion Engine: Identifies emotions using Azure Emotion API or Hume AI.

[1732] Database: Use AWS DynamoDB or Firebase to store data.

[1733] Feedback module: Uses a custom message generation algorithm.

[1734] Alert Module: Uses the Twilio API to notify emergency contacts.

[1735] Details of data processing and calculation

[1736] Acquisition of audio and image data

[1737] Device: Records voice data from the user and captures facial expressions and movements with the camera. This data is converted to text using the Google Speech-to-Text API, and image data is analyzed using OpenCV.

[1738] Emotion Recognition in Action

[1739] On the device: The emotion engine analyzes the voice and image data to identify the user's emotion. This information is sent to the server.

[1740] Analyzing data and providing feedback

[1741] Server: Based on the results of voice and image analysis, the user's emotional state is evaluated, and the feedback module generates appropriate advice. Real-time voice and visual feedback is provided to the user.

[1742] Alert Notifications

[1743] Server: If an anomaly is detected, the alert module immediately sends a notification to emergency contacts.

[1744] Specific examples

[1745] If a user comes home late at night and is captured looking nervous:

[1746] Application: The generative AI asks, "Good work. You're feeling a little nervous today, aren't you?"

[1747] User: "Yes, I'm a little nervous."

[1748] Application: The emotion engine detects the user's anxiety, and the alert module notifies emergency contacts in real time.

[1749] Example of input prompt for generative AI model

[1750] Analyze the user's voice and facial expressions to assess their current emotional state. If anxiety or tension is increasing, generate appropriate feedback and alert emergency contacts if necessary.

[1751]

[1752] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1753] Step 1:

[1754] User registration and profile creation

[1755] User: Installs the application on a smartphone and launches it. Creates an account by entering name, email address, password, age, gender, and profile information. Specifically, enters the required information into the application's input form and presses the registration button.

[1756] Input: Name, email address, password, age, gender and profile information.

[1757] Terminal: Sends the entered information to the server. Specifically, it sends data to the server using an HTTP request.

[1758] Output: Registration data sent to the server.

[1759] Server: Stores the received information in a database and sends an account creation completion response to the terminal.

[1760] Output: Account creation complete response.

[1761] Step 2:

[1762] Start a conversation session with generative AI

[1763] User: Launches the application and begins interacting with the generation AI by pressing the start button on the application's main screen.

[1764] Input: Instruction to start a dialogue.

[1765] Terminal: The generated AI asks, "Hello, Name. How are you doing today?"

[1766] Output: Question from the generative AI.

[1767] User: Answers the generated AI verbally about their physical condition and emotions. Specifically, they speak into a microphone.

[1768] Input: User's voice data.

[1769] Step 3:

[1770] Acquisition of audio and image data

[1771] Device: Records voice data from the user and captures facial expressions and movements with a camera. Specifically, the device uses the smartphone's microphone and camera to capture voice and images.

[1772] Input: Audio data, image data.

[1773] Terminal: The voice recognition module converts the recorded voice data into text, and the image data is analyzed by the image analysis module.

[1774] Output: Text data, analysis results.

[1775] Step 4:

[1776] Emotion Recognition in Action

[1777] On the device: The emotion engine analyzes audio and image data to identify the user's emotions. Specifically, emotion analysis is performed using Azure Emotion API and Hume AI.

[1778] Input: Text data, image analysis data.

[1779] Output: Emotion data.

[1780] Terminal: Sends the identified emotion data to the server. Specifically, it sends the emotion data using an HTTP request.

[1781] Output: Emotion data transmission.

[1782] Step 5:

[1783] Analyzing data and providing feedback

[1784] Server: Analyzes data to assess the user's health and emotional state based on the results of the voice recognition and image analysis modules and emotional data. Specifically, it uses a generative AI engine to integrate and evaluate multiple data sources.

[1785] Input: text data, image analysis results, emotion data.

[1786] Output: Health status assessment, emotional status assessment.

[1787] Server: The feedback module generates appropriate feedback to the user, such as advice like "Take a short rest."

[1788] Output: Feedback.

[1789] Terminal: Provides generated feedback to the user in audio and visual form.

[1790] Output: Provide feedback.

[1791] Step 6:

[1792] Alerting and Continuous Monitoring

[1793] Server: If an anomaly is detected, the alert module notifies the user's emergency contacts by sending an alert via SMS or phone call using the Twilio API.

[1794] Input: Emotion data, evaluation results.

[1795] Output: Alert notification.

[1796] Server: Continuously monitors the user's status and accumulates the collected data in a database. Specifically, data is periodically added to the database.

[1797] Output: Database updated.

[1798] Step 7:

[1799] Big data analysis and security plan proposals

[1800] Server: The big data analysis engine analyzes the collected data and generates a security plan optimized for the user. Specifically, it integrates long-term data and analyzes user behavior patterns and emotional tendencies.

[1801] Input: Collected data.

[1802] Output: Optimized security plan.

[1803] Server: Shares the generated security plan with the user and their emergency contacts via email or in-app notification.

[1804] Output: Security plan share.

[1805] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1806] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1807] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1808] [Fourth embodiment]

[1809] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1810] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1811] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1812] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1813] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1814] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1815] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1816] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1817] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1818] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1819] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1820] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1821] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1822] This invention relates to a system that uses generative AI technology to provide personalized care services to individuals. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement appropriate care plans.

[1823] System Configuration

[1824] The system includes the following major components:

[1825] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[1826] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[1827] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[1828] 4. Database: Stores the care recipient's profile, health status, and history data.

[1829] 5. Feedback module: Generates advice and feedback for the care recipient.

[1830] 6. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[1831] 7. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[1832] Program processing

[1833] 1. User registration and profile creation

[1834] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[1835] Terminal: Sends the entered information to the server.

[1836] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[1837] 2. Start a conversation session with the generative AI

[1838] User: Launches the application and begins interacting with the generative AI.

[1839] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[1840] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[1841] 3. Acquisition of audio and image data

[1842] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[1843] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data. The captured image data is passed to the image analysis module, where facial expressions and movements are analyzed.

[1844] 4. Data analysis and feedback

[1845] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[1846] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data and image analysis results.

[1847] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[1848] Device: Provides audio and visual feedback to the care recipient.

[1849] 5. Alert Notifications and Continuous Monitoring

[1850] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[1851] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[1852] 6. Big data analysis and care plan proposals

[1853] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1854] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[1855] Specific examples

[1856] 1. User registration and profile creation

[1857] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[1858] 2. Start a conversation session with the generative AI

[1859] User: Launches the application and begins a dialogue with the AI ​​generator. The AI ​​generator asks, "How are you feeling today?", to which the user responds, "I'm not feeling too well."

[1860] 3. Acquisition of audio and image data

[1861] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[1862] 4. Data analysis and feedback

[1863] Device: The text data and image analysis results are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[1864] 5. Alert Notifications and Continuous Monitoring

[1865] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[1866] 6. Big data analysis and care plan proposals

[1867] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve the plan.

[1868] The above is a description of the mode for carrying out the invention. It is expected that this system will provide individually optimized care and reduce the burden on caregivers.

[1869] The processing flow will be explained below.

[1870] Step 1: User Registration

[1871] User: Installs the application on the tablet device and launches it.

[1872] Device: Displays a registration screen and prompts the user to create an account.

[1873] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[1874] Terminal: Sends the entered information to the server.

[1875] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[1876] Step 2: Create your profile

[1877] Device: Displays a profile creation screen and prompts the user to enter personal information.

[1878] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[1879] Terminal: Sends the entered information to the server.

[1880] Server: The received information is linked to the user account and stored in a database.

[1881] Step 3: Start interacting with generative AI

[1882] User: Launches the application and begins an interactive session with the generative AI.

[1883] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[1884] Step 4: Acquiring audio and visual data

[1885] User: Tells the generated AI about their physical condition, wishes, etc.

[1886] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[1887] Step 5: Convert audio data to text

[1888] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[1889] Step 6: Analyzing the image data

[1890] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[1891] Step 7: Send data to the server

[1892] Terminal: Sends text data generated by voice recognition and image analysis results to the server.

[1893] Step 8: Analysis by generative AI

[1894] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on the text data and image analysis results.

[1895] Server: Stores the evaluation results in a database.

[1896] Step 9: Generate and provide feedback

[1897] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[1898] Server: Sends the generated feedback to the device.

[1899] Device: Provides audio and visual feedback to the care recipient.

[1900] Step 10: Alert Notification

[1901] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[1902] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[1903] Step 11: Continuous monitoring and data collection

[1904] Server: Continuously collects data from the care recipient and stores it as a log.

[1905] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[1906] Step 12: Big data analysis and care plan proposal

[1907] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1908] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[1909] Example 1

[1910] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1911] In an aging society, there is a need to improve the quality of nursing care services, provide personalized care that meets the individual needs of care recipients, and reduce the burden on caregivers and medical institutions. However, conventional systems have difficulty accurately understanding the health and emotional state of care recipients in real time and providing appropriate feedback and care plans. As a result, there has been a lack of efficient means to improve the quality of nursing care services.

[1912] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1913] In this invention, the server includes: means for communicating with the care recipient using AI technology; means for converting acquired voice data into text data; means for analyzing acquired image data; means for transmitting the text data and analysis results; means for issuing an alert when an abnormality is detected; means for a user to input voice and image data; means for transmitting the voice and image data to the server; means for generating feedback for the care recipient using AI technology; means for providing the feedback in audio or visual form; means for generating feedback based on data received by the AI ​​from the server; means for transmitting the feedback from the server to a terminal; means for analyzing big data and proposing an optimized care plan for the care recipient; means for sharing the care plan with a caregiver or a medical institution; means for analyzing the acquired big data; means for generating a care plan based on the analysis results; and means for notifying the caregiver or a medical institution of the care plan from the server. This enables real-time monitoring of the condition of the care recipient and providing appropriate feedback and a personalized care plan.

[1914] "Generative AI technology" is a technology that uses natural language processing and machine learning algorithms to engage in natural dialogue with humans and generate appropriate responses and information.

[1915] "Care recipients" refers to elderly people or individuals with physical disabilities who require nursing or medical assistance.

[1916] "Means of communication" refers to the method by which the system has the functionality to exchange information with the care recipient via voice or text.

[1917] "Audio data" refers to information that has been digitally recorded from the voice of the person receiving care.

[1918] "Text data" refers to information in which voice data is expressed as a string of characters.

[1919] The term "means for converting voice data into text data" refers to a method for converting voice data into corresponding text data using voice recognition technology.

[1920] "Image data" refers to information that digitally records the facial expressions and movements of the person receiving care.

[1921] "Means for analyzing image data" refers to a method of using image processing technology to recognize the facial expressions and movements of the person being cared for and analyze their meaning.

[1922] "Means for transmitting text data and analysis results" refers to the communication function for transmitting the text data and image analysis results acquired by the system to a remote server or other device.

[1923] "Means for sending an alert" refers to a method that has the function of sending a notification when an abnormality is detected and warning caregivers and medical institutions.

[1924] "Means for inputting voice and image data" refers to a method having the function of capturing voice and image data from the care recipient and inputting them into the system.

[1925] "Means by which the generating AI generates feedback based on data received from the server" refers to a method by which the generating AI creates appropriate feedback for the care recipient based on data received from the server.

[1926] "Means for providing audio or visual feedback" refers to a method that has the function of providing the generated feedback to the care recipient as audio and images.

[1927] "Means of analyzing big data" refers to methods that use technology to analyze large amounts of data and identify patterns and trends.

[1928] "Means for proposing an optimized care plan" refers to a method that has the function of creating and proposing an optimal care plan for a care recipient based on the analysis results.

[1929] "Means for sharing a care plan with a caregiver or a medical institution" refers to a method that has a function for sharing a generated care plan with a caregiver or a medical institution through communication.

[1930] A "server" is a computing device that serves as the core of a system and plays a central role in storing, analyzing, and transmitting data.

[1931] "Terminal" refers to a device (e.g., tablet, smartphone) that a user directly operates to communicate.

[1932] These definitions provide a clear understanding of the technical scope and function of the invention and serve as a basis for patent prosecution and enforcement.

[1933] This invention relates to a system that provides personalized care services to individuals using generative AI technology. The system aims to provide necessary information to caregivers and medical institutions through voice and visual communication with the care recipient, and to implement an appropriate care plan. The system includes the following main components:

[1934] System Configuration

[1935] 1. Generative AI Engine: This is the main software component for natural dialogue with the care recipient. The generative AI engine uses natural language processing and machine learning algorithms.

[1936] 2. Speech recognition module: This module recognizes the voice of the care recipient and converts it into text data. Specifically, it uses the Google Cloud Speech-to-Text API.

[1937] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses OpenCV, TensorFlow, etc.

[1938] 4. Database: A database system for storing the care recipient's profile, health status, and past history data. MongoDB or MySQL is used.

[1939] 5. Feedback module: This module generates advice and feedback for the care recipient. It works in conjunction with the generation AI engine to generate appropriate feedback.

[1940] 6. Alert module: This module sends alerts to caregivers and medical institutions when an abnormality is detected. It utilizes Twilio, SMTP, etc.

[1941] 7. Big Data Analysis Engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[1942] Program processing

[1943] 1. User registration and profile creation

[1944] User: Installs the application on a tablet device and launches it. When launching the application for the first time, the user enters the required information (name, email address, password, age, gender, medical history, allergies, current health condition) and creates a profile.

[1945] Terminal: Sends the entered information to the server. HTTPS is used as the communication protocol.

[1946] Server: Stores the received data in a database and notifies the device that account creation is complete.

[1947] 2. Start a conversation session with the generative AI

[1948] User: Launches the application and taps the "Start Interaction" button.

[1949] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[1950] User: Answers the generated AI verbally about their physical condition and wishes.

[1951] 3. Acquisition of audio and image data

[1952] Device: Records the user's voice data and captures facial expressions and movements with a camera.

[1953] Terminal: Sends voice data to the voice recognition module, which converts the voice into text data. Also sends captured image data to the image analysis module, which analyzes facial expressions and movements.

[1954] 4. Data analysis and feedback

[1955] Terminal: Transmits the converted text data and image analysis results to the server.

[1956] Server: The generation AI engine evaluates the health and emotional state of the care recipient based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[1957] Terminal: The generated feedback is converted into speech using a speech synthesis engine, and is played back to the user and also displayed on the screen as a text message.

[1958] 5. Alert Notifications and Continuous Monitoring

[1959] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[1960] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[1961] 6. Big data analysis and care plan proposals

[1962] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[1963] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to an electronic health record (EHR) system.

[1964] Specific examples

[1965] User: The application starts a dialogue, and the generative AI asks, "How is your day going?" The user responds verbally, "I'm feeling a little tired today."

[1966] Device: Records voice data and converts it into "I'm a little tired today" using a voice recognition module. Captures the user's facial expression using a camera, and analyzes the tired facial data using an image analysis module.

[1967] Server: Based on the received data, the generated AI evaluates the level of fatigue and generates feedback such as, "You seem tired. Please take a short rest."

[1968] Device: The feedback is synthesized into speech and played to the user. The same message is also displayed as text on the screen.

[1969] Prompt Sentence Examples

[1970] "Please explain how the generative AI engine assesses the emotional state of the care recipient."

[1971] "Please explain the process for alert notification when the system detects an abnormality."

[1972] These specific embodiments enable the system to provide high-quality, personalized care services to care recipients, thereby reducing the burden on caregivers and medical institutions.

[1973] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1974] Step 1: Register and create your profile

[1975] User: Installs the application on a tablet device and launches it by tapping the icon. On the sign-up screen, the user enters their name, email address, and password, then presses the "Next" button. Next, the user enters profile information such as age, gender, medical history, allergies, and current health condition, and presses the "Submit" button.

[1976] Input: Name, email address, password, age, gender, medical history, allergies, health status, and other information.

[1977] Output: The user's account information is saved in the database.

[1978] Terminal: The information entered by the user is sent to the server using the HTTPS protocol. If the transmission is successful, a confirmation message saying "Your account has been created" is displayed.

[1979] Server: Saves the received user information in the database. After confirming that the data was saved correctly, it sends a response to the device indicating that the account creation is complete.

[1980] Step 2: Start a conversation session with the generative AI

[1981] User: Launches the application and taps the "Start Interaction" button.

[1982] Input: The action to start the interaction.

[1983] Output: A greeting and question from the generated AI are displayed.

[1984] Device: Sends a request to start a conversation to the generation AI engine, and the generation AI engine starts the conversation by saying, "Hello, [Name]. How are you today?"

[1985] User: Answers the generated AI verbally about their physical condition and wishes.

[1986] Step 3: Acquiring audio and image data

[1987] Device: Records the user's voice data through a microphone and captures facial expressions and movements with a camera.

[1988] Input: User's voice and image data.

[1989] Output: Data transfer to speech recognition module and image analysis module.

[1990] Terminal: The recorded voice data is sent to the voice recognition module, which then converts the voice into text. The captured image data is also sent to the image analysis module, which analyzes facial expressions and movements.

[1991] Step 4: Analyze data and provide feedback

[1992] Terminal: Sends the text data converted by the voice recognition module and the results of the image analysis module to the server.

[1993] Input: Text data converted by the speech recognition module and the results of the image analysis module.

[1994] Output: Feedback generation.

[1995] Server: The generation AI engine evaluates the user's health and emotional state based on the received data. The feedback module generates appropriate feedback based on the analysis results.

[1996] Terminal: The generated feedback is converted into speech by a speech synthesis engine and played back to the user. It is also displayed on the screen as a text message.

[1997] Step 5: Alerting and continuous monitoring

[1998] Server: When an abnormality is detected, the alert module is activated and sends a notification to caregivers and medical institutions.

[1999] Input: Data in which an anomaly was detected.

[2000] Output: Alert notification.

[2001] Server: Collects data to continuously monitor the user's status and periodically stores it in a database.

[2002] Step 6: Big data analysis and proposing care plans

[2003] Server: The big data analysis engine analyzes the accumulated data and generates a care plan optimized for the user.

[2004] Input: Accumulated big data.

[2005] Output: Optimized care plan.

[2006] Server: Shares the generated care plan with caregivers and healthcare providers through appropriate platforms, for example, so that the care plan can be automatically added to the Electronic Health Record (EHR) system.

[2007] (Application example 1)

[2008] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2009] In modern society, providing personalized meal plans based on an individual's health status and dietary preferences is extremely important. In particular, food delivery services can provide greater convenience and satisfaction by suggesting optimal meals that take into account the user's health status and dietary restrictions. However, conventional systems have difficulty achieving such high levels of personalization, and there is a lack of technology to provide meal plans tailored to the user's health status.

[2010] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2011] In this invention, the server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and the analysis results and transmitting the plan to a delivery company, and a means for issuing an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[2012] "Generative AI technology" is a technology that uses natural language processing and machine learning models to interact with users and analyze data.

[2013] "Means of communication" refers to devices or programs that use generative AI technology to engage in voice or text dialogue with users.

[2014] The "means for converting acquired voice data into text data" refers to a device or program for converting a user's voice into text format using voice recognition technology.

[2015] "Means for analyzing acquired image data" refers to devices or programs that recognize and analyze the user's facial expressions and movements captured using image analysis technology.

[2016] The "means for proposing a personalized meal plan" is a device or program for generating and proposing an optimal meal plan based on the user's health condition and dietary preferences.

[2017] The "means for communicating to the delivery company" refers to a device or program for notifying the delivery company of the generated meal plan and making meal arrangements.

[2018] The "means for issuing an alert when an abnormality is detected" refers to a device or program for issuing a warning when an abnormality is detected in the user's health condition.

[2019] A "means for generating feedback" is a device or program that automatically generates appropriate advice or suggestions based on the user's input data.

[2020] "Audio or visual providing means" refers to a device or program for providing the generated feedback to the user audio or visually.

[2021] "Means for analyzing big data" refers to devices and programs that collect and analyze large amounts of data and extract useful information based on user behavior and health status.

[2022] The "means for proposing an optimized meal plan" refers to a device or program that uses the results of analyzing large amounts of data to generate and propose an optimal meal plan that meets the individual needs of a user.

[2023] "Means for sharing" refers to a device or program for sharing the generated meal plan and feedback with other parties or systems in collaboration with them.

[2024] This invention relates to a system that uses generative AI technology to propose personalized meal plans and arrange food delivery. The system aims to provide optimal meal plans based on the user's health condition and dietary preferences, and to realize these plans in cooperation with delivery companies.

[2025] System Configuration

[2026] The system includes the following major components:

[2027] 1. Generative AI engine:

[2028] It uses natural language processing and machine learning models (e.g., GPT-4) to interact with users and analyze their needs and health status.

[2029] 2. Speech Recognition Module:

[2030] The user's voice is converted into text data using the Google Speech-to-Text API or similar.

[2031] 3. Image Analysis Module:

[2032] Using OpenCV, TensorFlow, etc., the system analyzes the user's facial expressions and movements to assess their health condition.

[2033] 4. Database:

[2034] It stores user profiles, health status, dietary preferences, and historical data in databases such as MySQL and MongoDB.

[2035] 5. Feedback module:

[2036] Use a chatbot framework (such as Dialogflow) to generate feedback and advice for the user and provide it in audio or visual form.

[2037] 6. Alerts Module:

[2038] If an abnormality is detected, an alert will be sent to prompt appropriate action.

[2039] 7. Big Data Analysis Engine:

[2040] It analyzes the large amount of data collected and proposes an optimized meal plan to the user.

[2041] Program processing

[2042] The server includes a means for communicating based on an individual's health condition and dietary preferences using generative AI technology, a means for converting acquired voice data into text data, a means for analyzing acquired image data, a means for proposing a personalized meal plan based on the text data and analysis results, a means for transmitting the plan to a delivery company, and a means for sending an alert if an abnormality is detected. This makes it possible to provide an optimal meal plan based on the user's health condition and individual needs.

[2043] Specific example explanation

[2044] A user can use their smartphone to start a conversation with the generative AI. For example, if they ask, "What would you recommend for lunch?", the generative AI will respond with, "Hello, Name. You seem a little tired today. I'd like to suggest a healthy salad and soup."

[2045] Example prompt sentence:

[2046] User: What would you recommend for lunch?

[2047] Generative AI: Hello, Name. You seem a little tired today. I suggest a healthy salad and soup.

[2048] The user's voice data is converted to text using the Google Speech-to-Text API, and their facial expressions and movements are analyzed using OpenCV and TensorFlow. This data is sent to a server where a generative AI engine analyzes it and proposes the optimal meal plan for the user. The proposed plan is then shared with the delivery company, and delivery is arranged.

[2049] For example, if a user responds, "I'm not feeling well," the generative AI will respond with, "I'll suggest a porridge that's easy to digest," providing appropriate feedback. If an abnormality is detected, an alert will be sent, prompting the user to take appropriate action.

[2050] This system will enable advanced personalization based on the user's health condition and individual needs, significantly improving the quality of food delivery services.

[2051] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2052] Step 1:

[2053] User registration and profile creation

[2054] The user installs and launches the application on their smartphone. They create an account by entering information such as their name, email address, password, health status, dietary preferences, and allergies. The entered information is sent by the device to the server. The server stores the received information in a database and sends a response to the device indicating that the account has been created.

[2055] Step 2:

[2056] Start a conversation session with generative AI

[2057] The user launches the application and begins a dialogue with the generation AI. The generation AI asks on the device, "Hello, Name. What kind of meal would you like to have today?" The user responds verbally about their meal preferences and physical condition. Input: User's voice data. Output: Text data about their meal preferences.

[2058] Step 3:

[2059] Acquisition of audio and image data

[2060] The device records voice data from the user and captures facial expressions and movements with a camera. The recorded voice data is converted into text data using a voice recognition module (Google Speech-to-Text API). The captured image data is analyzed for the user's facial expressions and movements using an image analysis module (OpenCV, TensorFlow). Input: User's voice data and image data. Output: Text data and analysis results.

[2061] Step 4:

[2062] Data analysis and meal plan suggestions

[2063] The device sends the converted text data and image analysis results to the server. The server analyzes this data using a generative AI engine (GPT-4) to evaluate the user's health condition and preferences. The server generates a personalized meal plan based on the evaluation results. The proposed meal plan is presented by a feedback module (Chatbot framework: Dialogflow) with the message, "For today's meals, we suggest these to suit your physical condition." Input: Text data and image analysis results. Output: Personalized meal plan.

[2064] Step 5:

[2065] Delivery arrangements and alert notifications

[2066] The server arranges an order with a delivery company based on the generated meal plan. The user's meal plan is notified to the delivery company, which then arranges delivery. If an abnormality is detected, the server also sends an alert to caregivers and medical institutions via the alert module. Input: Generated meal plan. Output: Order to delivery company, and alert notification as necessary.

[2067] Step 6:

[2068] Big data analysis and service improvement

[2069] The server analyzes the collected data using a big data analysis engine and improves the algorithm that optimizes each user's meal plan. Furthermore, the service is continuously improved based on user feedback and health status. This makes it possible to propose meal plans that are optimized for each user. Input: Large amounts of analyzed data. Output: Optimized meal plans and improved algorithms.

[2070] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2071] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[2072] System Configuration

[2073] The system includes the following major components:

[2074] 1. Generative AI engine: This is the main component for natural dialogue with the care recipient.

[2075] 2. Speech recognition module: Converts the voice of the care recipient into text data.

[2076] 3. Image analysis module: Analyzes the facial expressions and movements of the care recipient.

[2077] 4. Emotion engine: Recognizes emotions from the care recipient's voice and facial expression data.

[2078] 5. Database: Stores the profile, health status and history data of the care recipient.

[2079] 6. Feedback module: Generates advice and feedback for the care recipient.

[2080] 7. Alert module: Notifies caregivers and medical institutions when an abnormality is detected.

[2081] 8. Big data analysis engine: Analyzes collected data and proposes optimized care plans.

[2082] Program processing

[2083] 1. User registration and profile creation

[2084] User: Installs and launches the application on a tablet device. Creates an account by entering required information (name, email address, password, etc.). Creates a profile by entering name, age, gender, medical history, allergies, and current health condition.

[2085] Terminal: Sends the entered information to the server.

[2086] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[2087] 2. Start a conversation session with the generative AI

[2088] User: Launches the application and begins interacting with the generative AI.

[2089] Terminal: The generated AI greets the care recipient with "Hello, Name. How are you today?" and asks about their health and wishes.

[2090] User: Answers questions about their physical condition and wishes to the generated AI via voice.

[2091] 3. Acquisition of audio and image data

[2092] Device: Records voice data from the care recipient and captures facial expressions and movements with a camera.

[2093] Terminal: The recorded voice data is converted into text by the voice recognition module, and the image data is analyzed by the image analysis module.

[2094] 4. Emotion Recognition

[2095] Terminal: The emotion engine analyzes the voice and image data to identify the emotions of the person receiving care.

[2096] Terminal: Sends the identified emotion data to the server.

[2097] 5. Data analysis and feedback

[2098] Terminal: Sends the text data converted by the voice recognition module, the results of the image analysis module, and the data from the emotion engine to the server.

[2099] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[2100] Server: The feedback module generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a rest.").

[2101] Device: Provides audio and visual feedback to the care recipient.

[2102] 6. Alert Notifications and Continuous Monitoring

[2103] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[2104] Server: Continuously monitors the condition of the care recipient and stores the collected data in a database.

[2105] 7. Big data analysis and care plan proposals

[2106] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[2107] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[2108] Specific examples

[2109] 1. User registration and profile creation

[2110] User: Installs the application and creates an account by entering their name, email address, password, age, gender, medical history, allergies, and current health condition. The device sends this information to the server, which stores it in a database.

[2111] 2. Start a conversation session with the generative AI

[2112] User: Launches the application and begins a conversation with the AI ​​generator. The AI ​​generator asks, "Hello, how are you today?", to which the user responds, "I'm not feeling too well."

[2113] 3. Acquisition of audio and image data

[2114] Terminal: Records voice data and captures the user's facial expressions with a camera. The recorded voice data is converted into text by a voice recognition module, and image data is analyzed by an image analysis module.

[2115] 4. Emotion Recognition

[2116] Terminal: The emotion engine analyzes the voice and image data to identify the user's emotion. The emotion data is sent to the server.

[2117] 5. Data analysis and feedback

[2118] Device: Text data, image analysis results, and emotion data are sent to the server, where the generative AI engine analyzes them. The feedback module generates advice such as "Take a short rest," which is displayed both visually and audibly on the device.

[2119] 6. Alert Notifications and Continuous Monitoring

[2120] Server: If an abnormality is detected, the alert module notifies caregivers and medical institutions and prompts them to take appropriate action.

[2121] 7. Big data analysis and care plan proposals

[2122] Server: Analyzes the collected data and generates the optimal care plan for the care recipient. The generated care plan is shared with caregivers and medical institutions and used to improve services.

[2123] This concludes the description of the embodiment of the invention. It is expected that this system will provide more individually optimized care through emotion recognition and reduce the burden on caregivers.

[2124] The processing flow will be explained below.

[2125] Step 1: User Registration

[2126] User: Installs the application on the tablet device and launches it.

[2127] Device: Displays a registration screen and prompts the user to create an account.

[2128] User: Enter the required information (name, email address, password, etc.) and press the registration button.

[2129] Terminal: Sends the entered information to the server.

[2130] Server: Stores the received information in a database and sends a response to the terminal indicating that account creation is complete.

[2131] Step 2: Create your profile

[2132] Device: Displays a profile creation screen and prompts the user to enter personal information.

[2133] User: Enter name, age, gender, medical history, allergies, current health condition, etc.

[2134] Terminal: Sends the entered information to the server.

[2135] Server: The received information is linked to the user account and stored in a database.

[2136] Step 3: Start interacting with generative AI

[2137] User: Launches the application and begins an interactive session with the generative AI.

[2138] Terminal: The generated AI greets the user with "Hello, Name. How are you today?"

[2139] Step 4: Acquiring audio and visual data

[2140] User: Tells the generated AI about their physical condition, wishes, etc.

[2141] Device: Records audio data in real time and captures facial expressions and movements with a camera.

[2142] Step 5: Convert audio data to text

[2143] Terminal: The recorded voice data is passed to the voice recognition module and converted into text data.

[2144] Step 6: Analyzing the image data

[2145] Terminal: The captured image data is passed to the image analysis module, which analyzes facial expressions and movements.

[2146] Step 7: Emotion Recognition

[2147] Terminal: Passes image analysis and voice recognition data to the emotion engine to recognize the user's emotions.

[2148] Terminal: Sends the recognized emotion data to the server.

[2149] Step 8: Send data to the server

[2150] Terminal: Sends text data generated by voice recognition, image analysis results, and emotion data to the server.

[2151] Step 9: Analysis by generative AI

[2152] Server: The generative AI engine evaluates the health and emotional state of the care recipient based on text data, image analysis results, and emotional data.

[2153] Server: Stores the evaluation results in a database.

[2154] Step 10: Generate and provide feedback

[2155] Server: The generation AI generates appropriate feedback to the care recipient (e.g., "You seem tired. Please take a short rest.").

[2156] Server: Sends the generated feedback to the device.

[2157] Device: Provides audio and visual feedback to the care recipient.

[2158] Step 11: Alert Notification

[2159] Server: If an abnormality is detected, the alert module sends an alert to caregivers and medical institutions.

[2160] Server: Notifies the appropriate device (smartphone, email, etc.) of the alert content.

[2161] Step 12: Continuous monitoring and data collection

[2162] Server: Continuously collects data from the care recipient and stores it as a log.

[2163] Server: Analyzes accumulated log data in real time and takes immediate action if an abnormality is detected.

[2164] Step 13: Big data analysis and care plan proposal

[2165] Server: The big data analysis engine analyzes the large amount of collected data and generates an optimized care plan for the care recipient.

[2166] Server: Shares the generated care plans with caregivers and medical institutions to continuously improve services.

[2167] Example 2

[2168] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2169] In today's nursing care industry, there is a need to accurately grasp the physical and mental state of care recipients and provide appropriate care tailored to their individual needs and emotions. However, conventional systems lack the technology to analyze the emotions and health status of care recipients in real time and provide optimal feedback and care plans, which contributes to the increased burden on caregivers. Furthermore, their ability to effectively analyze large amounts of data and issue appropriate alerts is limited.

[2170] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2171] In this invention, the server includes means for communicating with the care recipient using generative AI technology, means for converting acquired voice data into text data, means for analyzing acquired image data, means for recognizing emotions using an emotion engine, means for transmitting the text data, analysis results, and emotion data to a caregiver or a medical institution, and means for issuing an alert when an abnormality is detected. This makes it possible to grasp the emotional and health state of the care recipient in real time, provide optimal feedback and care plans, and immediately detect and notify abnormalities.

[2172] "Generative AI technology" refers to technology that uses artificial intelligence to generate natural language and engage in dialogue.

[2173] "Communication" refers to the process of engaging in voice and text dialogue with the care recipient through generative AI technology.

[2174] "Audio data" refers to the voice of the care recipient recorded through a microphone and expressed in digital form.

[2175] "Text data" refers to data that has been converted from voice data using a voice recognition module so that it can be handled as text information.

[2176] "Image data" refers to visual information captured through a camera or other image input device.

[2177] "Analysis" is the process of processing collected data using specific algorithms to extract meaningful information.

[2178] An "emotion engine" refers to algorithms and technologies for recognizing the emotional state of a care recipient from voice and image data.

[2179] "Emotion data" refers to information indicating an emotional state output by the emotion engine after analyzing audio and image data.

[2180] "Communication" refers to the process of sending and sharing the generated text data, analysis results, and emotional data with caregivers or medical institutions.

[2181] An "alert" is a notification sent to a caregiver or medical institution as a warning when an abnormality is detected.

[2182] "Feedback" refers to advice and comments that the generative AI technology provides to the care recipient based on analyzed data.

[2183] "Big data" refers to large data sets and the techniques and methods for analyzing them to extract useful information.

[2184] A "care plan" is a plan formulated to provide optimal nursing care services to a care recipient.

[2185] "Sharing" refers to the process of sharing and exchanging information such as generated care plans among stakeholders.

[2186] This invention relates to a system that provides personalized care services to individuals using generative AI technology and an emotion engine. The system recognizes the emotions of care recipients through voice and visual communication, provides necessary information to caregivers and medical institutions, and implements appropriate care plans.

[2187] System Configuration

[2188] The system includes the following major components:

[2189] 1. Generative AI engine: This is the main component for natural conversation with the care recipient. It uses Google Cloud Speech-to-Text API and OpenAI GPT model.

[2190] 2. Speech recognition module: This module converts the voice of the care recipient into text data. It uses the Google Cloud Speech-to-Text API, etc.

[2191] 3. Image analysis module: This module analyzes the facial expressions and movements of the care recipient. It uses TensorFlow, OpenCV, etc.

[2192] 4. Emotion Engine: Algorithms and technologies for recognizing emotions from the voice and facial expression data of the care recipient. This includes tools for extracting emotions from voice and deep learning models for facial expression analysis.

[2193] 5. Database: Data storage for storing the care recipient's profile, health status, and history data. SQL or NoSQL databases are used.

[2194] 6. Feedback module: A program that generates advice and feedback for the care recipient. It works in conjunction with the generative AI engine.

[2195] 7. Alert module: This is a system for notifying caregivers and medical institutions when an abnormality is detected. It includes the function of sending emails and SMS.

[2196] 8. Big data analysis engine: An engine that analyzes collected data and proposes optimized care plans. It uses Apache Hadoop and Spark.

[2197] Program processing

[2198] The program processing of this system is as follows.

[2199] 1. The user installs and launches the application on their tablet device. The user creates an account by entering the required information (such as name, email address, and password), and then enters their profile information (e.g., name, age, gender, medical history, allergies, and current health condition).

[2200] 2. The device sends the entered information to the server, which stores the received information in a database. The server then sends a response to the device indicating that account creation is complete.

[2201] 3. The user launches the application and begins a conversation with the AI. The AI ​​greets the user via voice and text, saying, "Hello,...

Claims

1. A means of communicating with care recipients using generative AI technology; A means for converting the acquired voice data into text data; means for analyzing the acquired image data; a means for transmitting the text data and analysis results to a caregiver or a medical institution; A means of issuing an alert if an anomaly is detected; and A system including:

2. A means for generating feedback for the care recipient using generative AI technology; means for providing said feedback audio or visually; The system of claim 1 further comprising:

3. A means of analyzing big data and proposing optimal care plans for care recipients, a means for sharing said care plan with a caregiver or a medical institution; The system of claim 1 further comprising:

Images