System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a patient's age, aphasia level, and medical chart information; means for receiving and analyzing the patient's information; means for automatically generating a rehabilitation plan based on the analysis result; means for displaying the rehabilitation plan and giving instructions to a user; means for recording video and audio of rehabilitation of the patient; means for receiving and analyzing the recorded data; and means for creating a daily report based on the analysis result and distributing the daily report to a predetermined recipient.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Patients with aphasia or brain damage often have difficulties with language comprehension, expression, and communication skills, which impact their daily lives. While support from speech-language-hearing therapists and rehabilitation physicians is crucial for these patients' rehabilitation, it can be difficult to receive sufficient support due to a lack of personnel and time and labor constraints. This not only delays patients' recovery, but also places a strain on their mental health. Therefore, the purpose of this invention is to solve these issues by utilizing AI technology to automatically generate rehabilitation plans tailored to each patient and provide flexible and sustained support. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by the following means. A system is provided that includes: a means for inputting a patient's age, aphasia level, and medical record information; a means for receiving and analyzing the patient information; a means for automatically generating a rehabilitation plan based on the analysis results; a means for displaying the rehabilitation plan and issuing instructions to the user; a means for recording video and audio data that records the patient's rehabilitation; a means for receiving and analyzing the recorded data; and a means for creating daily reports based on the analysis results and distributing them to specified recipients. This system can also include a means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback, and a means for selecting family information and communication methods and reflecting them in the rehabilitation plan. This allows for the provision of an optimal rehabilitation plan for each patient, achieving efficient and continuous rehabilitation.

[0006] "Patient information" refers to basic data necessary for formulating a rehabilitation plan, including the patient's age, gender, aphasia level, and medical record information.

[0007] "Analysis" is the process of evaluating and diagnosing the patient's current condition and performance based on the received patient information and rehabilitation record data.

[0008] A "rehabilitation plan" is a plan that indicates the specific content and procedures of rehabilitation that are automatically generated based on the analysis results, and includes vocal training, comprehension questions, and communication exercises.

[0009] A "rehabilitation session" is a single rehabilitation process that a patient actually performs according to a plan.

[0010] "Video and audio recording" refers to the act of recording a patient's movements and vocalizations during a rehabilitation session using a camera and microphone.

[0011] "Basic data" refers to the initial information required to formulate a rehabilitation plan, such as the patient's age, gender, level of aphasia, and medical record information.

[0012] The "daily report" is a report that details the results of the rehabilitation session, progress, achievements, and areas for improvement next time, and is sent to the doctor and family.

[0013] "Real-time monitoring" is a technology that monitors and analyzes video and audio data in real time during a rehabilitation session, providing immediate feedback.

[0014] "Feedback" refers to the evaluation, instructions, and suggestions for improvement provided to patients during rehabilitation based on the analysis results.

[0015] "Communication methods" refers to the methods and tools, such as spoken language, written language, and gestures, that patients use during rehabilitation.

[0016] "Selection" is the process of choosing something that meets a specific purpose or condition. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates a series of steps: collecting and analyzing patient information, creating rehabilitation plans, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, we will explain the program processing of this system in natural language and provide detailed examples.

[0039] System configuration and operation

[0040] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[0041] 1. Collecting patient information

[0042] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[0043] 2. Analysis of patient information

[0044] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[0045] 3. Generating a rehabilitation plan

[0046] The server automatically generates an individual rehabilitation plan based on the analysis results. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises. The generated rehabilitation plan is sent to the device and can be viewed by the user.

[0047] 4. Rehabilitation

[0048] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[0049] 5. Rehabilitation progress monitoring

[0050] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides real-time feedback to the user via the device as needed.

[0051] 6. Create and share daily reports

[0052] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[0053] Specific examples

[0054] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the instructions on the device, recording the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[0055] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[0056] The processing flow will be explained below.

[0057] Step 1: Enter patient information

[0058] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[0059] Step 2: Receiving and analyzing patient information

[0060] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[0061] Step 3: Generate a rehabilitation plan

[0062] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[0063] Step 4: Submit your rehabilitation plan

[0064] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[0065] Step 5: Rehabilitation

[0066] The user (patient) begins rehabilitation by following instructions on the device, which uses a camera and microphone to record the rehabilitation session as video and audio.

[0067] Step 6: Real-time data transmission

[0068] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[0069] Step 7: Real-time data analysis and feedback

[0070] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[0071] Step 8: Ending the rehabilitation session and sending data

[0072] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[0073] Step 9: Create a daily report

[0074] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[0075] Step 10: Share the report

[0076] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[0077] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] The present invention aims to maximize the effectiveness of rehabilitation for patients with aphasia or brain damage. Conventional rehabilitation methods have the problem that it takes a great deal of time and effort to create an optimal rehabilitation plan for each individual patient and provide feedback on the implementation status. Furthermore, effective rehabilitation is difficult due to limited means for monitoring the patient's progress in real time. It is desirable to solve these problems.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes means for receiving basic patient information and storing it in a database, means for analyzing the received patient information and generating basic data for evaluating the patient's condition, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the generated rehabilitation plan and issuing instructions to the user, means for recording the patient's rehabilitation with a camera and microphone, means for receiving the recorded data, monitoring it in real time, and generating immediate feedback, and means for analyzing the recorded data, automatically generating daily reports, and distributing them to specified recipients. This makes it possible to quickly provide an optimized rehabilitation plan for each patient, monitor its implementation in real time, and provide immediate feedback.

[0083] "Basic patient information" refers to basic data necessary for the patient's medical rehabilitation, such as the patient's age, gender, aphasia level, and medical record information.

[0084] "Database" refers to a collection of data used by the Server to efficiently store, manage, and retrieve patient information.

[0085] "Baseline data" refers to basic evaluation data generated by analyzing received patient information to evaluate the patient's condition.

[0086] A "rehabilitation plan" refers to a detailed rehabilitation schedule that includes voice training, comprehension exercises, and communication exercises that are optimized for the patient's individual condition.

[0087] "Means for issuing instructions to the user" refers to the system operation and screen display mechanisms for appropriately notifying the user of the generated rehabilitation plan and issuing instructions.

[0088] "Camera and microphone" refers to the video and audio input devices used to record the patient's rehabilitation progress.

[0089] "Means for real-time monitoring" refers to a system that instantly monitors the progress of rehabilitation on the spot and analyzes the data.

[0090] "Immediate feedback" refers to assessments and instructions provided to patients quickly based on real-time monitored data.

[0091] "Daily report" refers to a detailed report summarizing the content and progress of each day's rehabilitation sessions.

[0092] The "predetermined recipient" refers to medical staff or family members of patients who are designated in advance as recipients of the generated daily report.

[0093] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates the collection and analysis of patient information, the creation of rehabilitation plans, the implementation of rehabilitation, progress monitoring, and the creation and sharing of daily reports, providing rehabilitation that is optimized for each individual patient.

[0094] System configuration and operation

[0095] This system mainly consists of three elements: a server, a terminal, and users (patients and medical staff).

[0096] Collecting patient information

[0097] The user (medical staff) enters basic patient information (age, gender, aphasia level, medical record information, etc.) into the terminal. The terminal formats the entered information and sends it to the server. The server stores the patient information received from the terminal in a database.

[0098] Patient information analysis

[0099] The server analyzes the patient information stored in the database and generates basic data to evaluate the patient's condition, which is then used to create an individual rehabilitation plan.

[0100] Generate rehabilitation plans

[0101] The server automatically generates an appropriate rehabilitation plan based on the generated basic data. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises, and the generated plan is sent to the device. The user (medical staff) can then check the rehabilitation plan through the device.

[0102] Rehabilitation

[0103] The user (patient) follows the rehabilitation instructions displayed on the device. The device displays the rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises. The device also records the rehabilitation progress as video and audio.

[0104] Rehabilitation progress monitoring

[0105] The device monitors the progress of rehabilitation in real time and transmits video and audio data to the server, which analyzes the received data and evaluates the rehabilitation progress. If necessary, the server generates real-time feedback and provides it to the user via the device.

[0106] Create and share daily reports

[0107] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server, which then automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated app.

[0108] Specific examples

[0109] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the device's instructions and records the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the doctor in charge and the patient's family.

[0110] By inputting the following prompt sentences into the generative AI model, the system automatically generates an optimal rehabilitation plan for the patient:

[0111] "A 70-year-old male patient suffers from moderate aphasia. Please create an optimal rehabilitation plan for this patient. The patient's basic information is as follows: Age: 70, Gender: Male, Current Status: Moderate aphasia."

[0112] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Processing Steps

[0115] Step 1:

[0116] The user (medical staff) inputs basic information about the patient (age, sex, aphasia level, medical record information, etc.) into the terminal.

[0117] Input: Patient's age, gender, aphasia level, medical record information

[0118] How it works: The user enters patient information into a dedicated application on the device and clicks the send button.

[0119] Output: Information entered into the terminal is formatted and sent to the server.

[0120] Step 2:

[0121] The terminal formats the entered patient information and transmits it to the server.

[0122] Input: Basic patient information entered by the user

[0123] Operation: The device converts the information into a specific format and sends it to the server.

[0124] Output: Formatted patient information

[0125] Step 3:

[0126] The server stores the patient information received from the terminal in a database.

[0127] Input: Formatted patient details

[0128] What happens: The server stores the information in a database

[0129] Output: Patient information stored in the database

[0130] Step 4:

[0131] The server analyzes the patient information stored in the database and generates basic data for evaluating the patient's condition.

[0132] Input: Patient information stored in the database

[0133] How it works: The server analyzes the information using natural language processing algorithms and generates the underlying data.

[0134] Output: The underlying data generated by the analysis

[0135] Step 5:

[0136] The server automatically generates an appropriate rehabilitation plan based on the generated basic data.

[0137] Input: Basic data

[0138] How it works: The server uses the generative AI model to create a rehabilitation plan.

[0139] Output: Automatically generated rehabilitation plan

[0140] Step 6:

[0141] The server transmits the generated rehabilitation plan to the terminal so that the user (medical staff) can check it.

[0142] Input: Rehabilitation Plan

[0143] Operation: The server sends the rehabilitation plan to the device.

[0144] Output: Rehabilitation plan sent to the terminal

[0145] Step 7:

[0146] The user (patient) follows the rehabilitation instructions displayed on the terminal and performs rehabilitation.

[0147] Input: Rehabilitation plan displayed on terminal

[0148] Action: The user uses the camera and microphone to perform rehabilitation.

[0149] Output: Data recording rehabilitation progress

[0150] Step 8:

[0151] The device displays rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises.

[0152] Input: Rehabilitation Plan

[0153] Operation: The device displays rehabilitation instructions to the user and records the rehabilitation progress with video and audio.

[0154] Output: Recorded video and audio data

[0155] Step 9:

[0156] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[0157] Input: Recorded video and audio data

[0158] Action: The device sends data to the server.

[0159] Output: Video and audio data sent to the server

[0160] Step 10:

[0161] The server analyzes the received data and evaluates the degree of rehabilitation achievement.

[0162] Input: Video and audio data

[0163] Action: The server analyzes the data and generates an evaluation result.

[0164] Output: Evaluation results regarding rehabilitation achievement

[0165] Step 11:

[0166] The server generates and provides immediate feedback to the user via the terminal as needed.

[0167] Input: Evaluation results regarding rehabilitation achievement

[0168] What it does: The server generates and sends immediate feedback to the user

[0169] Output: Immediate feedback provided to the user

[0170] Step 12:

[0171] When the rehabilitation session ends, the terminal transmits the rehabilitation data for that day to the server.

[0172] Input: Rehabilitation data for one day

[0173] Operation: The device collects data and sends it to the server

[0174] Output: Rehabilitation data for one day sent to the server

[0175] Step 13:

[0176] The server automatically generates daily reports based on the data received.

[0177] Input: Rehabilitation data for one day

[0178] How it works: The server analyzes the data and generates a daily report.

[0179] Output: Auto-generated daily reports

[0180] Step 14:

[0181] The server distributes the generated report to the doctor in charge and the patient's family.

[0182] Input: Daily Report

[0183] How it works: The server delivers reports via email or a dedicated application.

[0184] Output: Daily reports delivered to treating physicians and patient families

[0185] In this way, this system achieves efficient and sustainable rehabilitation by linking data input, calculation, and output at each step.

[0186] (Application example 1)

[0187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] Current rehabilitation systems struggle to efficiently and continuously monitor and automatically generate personalized rehabilitation plans. Furthermore, they lack the ability to manage rehabilitation progress in real time and provide immediate feedback, making it difficult to achieve effective rehabilitation. Furthermore, there are no systems suitable for use in rehabilitation centers or brick-and-mortar stores, preventing patients from maximizing the benefits of their rehabilitation.

[0189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0190] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical record information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio data for recording the patient's rehabilitation, means for receiving and analyzing the recorded data, means for creating daily reports based on the analysis results and distributing them to specified recipients, and means for providing the rehabilitation plan to a terminal at the rehabilitation center and managing the progress. This enables automatic generation of an individualized rehabilitation plan and real-time monitoring of rehabilitation sessions, and real-time feedback to achieve effective rehabilitation.

[0191] "Patients" refers to those with aphasia, brain damage, etc. who require rehabilitation.

[0192] "Age" is the period indicated by the patient's biological number.

[0193] "Aphasia level" is an index showing the degree of aphasia.

[0194] "Medical record information" refers to medical data including a patient's medical history and treatment records.

[0195] "Means for receiving and analyzing information" refers to the processes and tools for obtaining and analyzing patient-entered information.

[0196] "Means for automatically generating rehabilitation plans" refers to technology that automatically creates individualized rehabilitation programs based on collected patient information.

[0197] "Means for providing instructions" refers to a method for presenting the generated rehabilitation plan to the patient or medical staff.

[0198] "Video and audio recording means" refers to devices and methods for recording rehabilitation sessions using cameras and microphones.

[0199] "Means for receiving and analyzing recorded data" refers to the processes and tools for receiving and analyzing recorded video and audio data.

[0200] "Means for creating and distributing daily reports" refers to a system and method for organizing rehabilitation progress information on a daily basis and distributing it to designated recipients (such as medical staff or family members).

[0201] "Rehabilitation center terminal" refers to computer equipment installed in a rehabilitation facility or clinic that is used to review and implement rehabilitation plans.

[0202] "Progress management means" refers to systems and methods that track a patient's rehabilitation progress and provide appropriate measures and feedback accordingly.

[0203] The system of this invention is designed to support effective rehabilitation for patients with aphasia or brain damage in physical rehabilitation centers or clinics. This system mainly consists of a server, terminals, and users (patients and medical staff). Specific embodiments of the present invention are described in detail below.

[0204] System configuration

[0205] 1. Collecting patient information

[0206] The server receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff) into the terminal. The terminal sends this information to the server, which then analyzes it.

[0207] 2. Analysis of patient information

[0208] The server analyzes the received patient information and generates individual basic data, which is used to prepare an optimal rehabilitation plan for each patient.

[0209] 3. Generating a rehabilitation plan

[0210] The server automatically generates a rehabilitation plan based on the analysis results. The plan is sent to the terminal and confirmed by the user (medical staff and patient). The rehabilitation plan includes vocal exercises, comprehension questions, and communication exercises.

[0211] 4. Rehabilitation

[0212] Patients undergo rehabilitation by following instructions on a device in the facility. The rehabilitation content is displayed on a screen, and patients use a camera and microphone connected to the device to perform voice training and exercises. The device records the rehabilitation session and sends the video and audio data to a server.

[0213] 5. Rehabilitation progress monitoring

[0214] The server monitors the progress of rehabilitation in real time and provides immediate feedback to the device as needed, helping to ensure smooth rehabilitation progress for the patient.

[0215] 6. Create and share daily reports

[0216] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server analyzes the received data and automatically generates a daily report. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated application.

[0217] Specific examples

[0218] For example, if a 70-year-old male patient suffers from moderate aphasia, medical staff enter the patient's basic information into the device and send it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results and sends it to the device. The patient performs rehabilitation according to the device's instructions, recording the process with a camera and microphone. The device then sends the recorded data to the server in real time, which analyzes the data and provides immediate feedback. After the rehabilitation session ends, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[0219] Hardware and software used

[0220] Smartphone: Equipped with a camera and microphone, it is used as a device to record the patient's rehabilitation progress.

[0221] Server: Analyzes patient information, generates rehabilitation plans, performs real-time monitoring, and creates daily reports.

[0222] Software: Python program, server communication via HTTP protocol.

[0223] Prompt Sentence Examples

[0224] "Please provide a specific use case for a smartphone application to help patients with their rehabilitation. Please explain in detail how it collects patient information, captures rehabilitation plans, records rehabilitation sessions, and generates and shares daily reports."

[0225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0226] System program processing steps

[0227] Step 1:

[0228] The terminal receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff). This basic information is formatted by the terminal and sent to the server. The input data includes the patient's biometric number and medical records related to rehabilitation.

[0229] Step 2:

[0230] The server receives and analyzes the patient information sent from the device. Specifically, it uses each data item as material for identifying the disease and developing a rehabilitation plan to generate basic patient data based on the received data. This basic data includes information to clarify the patient's condition and rehabilitation goals.

[0231] Step 3:

[0232] The server automatically generates a rehabilitation plan based on the analysis results. The plan includes voice training, comprehension questions, and communication exercises tailored to each patient. The automatically generated plan is sent to the device and provided to medical staff and the patient.

[0233] Step 4:

[0234] The user (patient) performs rehabilitation by following instructions on the device. The device records each rehabilitation session with a camera and microphone and transmits the data in real time to a server. The recorded data includes rehabilitation progress and specific exercises.

[0235] Step 5:

[0236] The server analyzes the recorded rehabilitation data received in real time and generates feedback on the spot, returning it to the device. The feedback includes vocal errors and exercise improvements, allowing patients to receive immediate corrections as needed during rehabilitation.

[0237] Step 6:

[0238] After the rehabilitation session, the device sends the day's worth of rehabilitation data to the server, which then automatically generates a daily report based on the received data, detailing rehabilitation progress, achievements, and recommended next exercises.

[0239] Step 7:

[0240] Finally, the server delivers the generated daily report to the designated recipient (physician or family member) via email or a dedicated application. The delivered report serves as a reference for the recipient to check the patient's rehabilitation progress and plan future treatment plans.

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] This invention is a system that uses AI technology to support the rehabilitation of patients with aphasia or brain damage, and by combining it with an emotion engine that recognizes the user's emotions, it implements rehabilitation plans more adaptively and effectively. This system automates a series of steps: collecting and analyzing patient information, generating rehabilitation plans, recognizing emotions, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[0243] System configuration and operation

[0244] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[0245] 1. Collecting patient information

[0246] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[0247] 2. Analysis of patient information

[0248] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[0249] 3. Generating a rehabilitation plan

[0250] The server automatically generates an individual rehabilitation plan based on the analysis results. This rehabilitation plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user.

[0251] 4. Emotion Recognition with Emotion Engine

[0252] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session and transmits this information to the emotion engine, which analyzes this information and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.).

[0253] 5. Adaptive adjustment of rehabilitation plans

[0254] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it can add relaxation exercises.

[0255] 6. Rehabilitation

[0256] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[0257] 7. Rehabilitation progress monitoring

[0258] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides immediate feedback to the user via the device as needed.

[0259] 8. Create and share daily reports

[0260] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[0261] Specific examples

[0262] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic patient data. Next, based on the analysis results, the server automatically generates an individual rehabilitation plan including speech exercises and comprehension questions and sends it to the device.

[0263] The user (patient) performs rehabilitation according to instructions on the device, recording the process with a camera and microphone. The device then sends the recorded data to the emotion engine, which analyzes the user's facial expressions and vocal tone to recognize their emotional state. The server adaptively adjusts the rehabilitation content based on the recognized emotion results and provides it to the user. For example, if the user is tired, it can add relaxation exercises.

[0264] After the rehabilitation session, the server creates a daily report based on the data from the device and distributes it to the patient's doctor and family. In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, achieving efficient and sustainable rehabilitation.

[0265] The processing flow will be explained below.

[0266] Step 1: Enter patient information

[0267] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[0268] Step 2: Receiving and analyzing patient information

[0269] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[0270] Step 3: Generate a rehabilitation plan

[0271] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[0272] Step 4: Submit your rehabilitation plan

[0273] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[0274] Step 5: Emotion Recognition with the Emotion Engine

[0275] The device uses a camera and microphone to record the user's (patient's) facial expressions, voice tone, and word choice during the rehabilitation session. The device then sends this recorded data to the emotion engine, which analyzes the recorded data and recognizes the user's emotional state.

[0276] Step 6: Adaptively adjusting the rehabilitation plan

[0277] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it adds relaxation exercises.

[0278] Step 7: Rehabilitation

[0279] The user performs rehabilitation by following instructions from the device, which uses a camera and microphone to record video and audio of the rehabilitation session.

[0280] Step 8: Real-time data transmission

[0281] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[0282] Step 9: Real-time data analysis and feedback

[0283] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[0284] Step 10: Ending the rehabilitation session and sending data

[0285] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[0286] Step 11: Create a daily report

[0287] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[0288] Step 12: Share the report

[0289] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[0290] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[0291] Example 2

[0292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] Currently, rehabilitation for patients with aphasia and brain damage requires flexible rehabilitation plans tailored to each individual, as the effectiveness of these plans varies from patient to patient. However, existing rehabilitation systems lack the ability to appropriately grasp a patient's emotional state and dynamically adjust the rehabilitation plan. They also lack the ability to monitor the progress of rehabilitation in real time and provide immediate feedback. Therefore, there is a need for a system that can generate rehabilitation plans tailored to each patient's individual needs and recognize and adapt to their emotional state during the implementation process.

[0294] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input the patient's age, disability level, and medical information; a means for receiving and analyzing the patient information; a means for automatically generating a rehabilitation plan based on the analysis results; a means for collecting data on the patient's facial expressions, tone of voice, and word choice during the rehabilitation session to recognize the patient's emotional state; a means for adaptively adjusting the rehabilitation plan based on the recognized emotional state; a means for receiving and analyzing the recorded data; a means for creating a daily report based on the analysis results and distributing it to a predetermined recipient; and a means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback. This enables the creation and provision of an optimal rehabilitation plan tailored to the individual patient's condition. Furthermore, by accurately understanding the patient's emotional state during the rehabilitation process and making adaptive adjustments, the effectiveness of the rehabilitation can be maximized.

[0295] "User" is the medical professional or person who operates the system, inputs patient data, and reviews and adjusts rehabilitation plans.

[0296] "Patients" refer to people who need rehabilitation for conditions such as aphasia or brain damage.

[0297] A "server" is a computer system that analyzes patient information, generates rehabilitation plans, and stores and processes data.

[0298] A "terminal" is an electronic device such as a computer, tablet, or smartphone that is directly operated by a user or patient.

[0299] "Patient information" refers to basic data necessary for generating and adjusting a rehabilitation plan, such as age, disability level, and medical information.

[0300] A "rehabilitation plan" is a plan that includes specific rehabilitation content and steps and is automatically generated based on the patient's condition.

[0301] "Emotion engine" refers to algorithms and machine learning models that analyze a patient's facial expressions, tone of voice, and word choice data to recognize their emotional state.

[0302] The "Daily Report" is a report that summarizes the daily activity record, including the progress, achievements, and areas for improvement of the rehabilitation session.

[0303] "Real-time monitoring" is the process of observing rehabilitation progress in real time and collecting and analyzing data.

[0304] "Immediate feedback" refers to advice or instructions provided immediately to a user or patient based on the results of data analysis.

[0305] "Adaptive adjustment" is the process of dynamically changing or adjusting a rehabilitation plan based on the patient's emotional state and progress.

[0306] This invention provides a system for supporting rehabilitation for patients with aphasia or brain damage, recognizing the patient's emotional state and adaptively adjusting the rehabilitation plan based on that state. The system automates the collection and analysis of patient information, generation of rehabilitation plans, emotion recognition, rehabilitation implementation, progress monitoring, and creation and sharing of daily reports.

[0307] The main elements of this system are the server, terminals, and users (patients and medical staff). Below we will explain in detail how each element works together.

[0308] Hardware and Software Used

[0309] Hardware

[0310] 1. Devices: Personal computers, tablets, smartphones, etc. used by users and patients.

[0311] 2. Camera and microphone: Devices for collecting the patient's facial expressions and voice.

[0312] 3. Server: A high-performance computer for analyzing data and generating rehabilitation plans.

[0313] software

[0314] 1. Emotion engine: An AI model for analyzing patient emotions (e.g., Google® Cloud Natural Language API or Azure® Emotion API).

[0315] 2. Analysis algorithm: Software for analyzing patient information and generating rehabilitation plans.

[0316] 3. Database system: A system for storing patient information and rehabilitation progress data (e.g., MySQL (registered trademark), PostgreSQL).

[0317] 4. Rehabilitation support app: A dedicated application to help patients carry out rehabilitation.

[0318] System example and operation

[0319] Collecting patient information

[0320] The user (medical staff) uses a terminal to input basic information such as the patient's age, disability level, medical information, etc. The terminal then transmits the input information to the server.

[0321] Patient information analysis

[0322] The server receives the patient information sent from the terminal, cleanses the information using an analysis program, and generates basic data.

[0323] Generate rehabilitation plans

[0324] The server automatically generates a rehabilitation plan based on the generated basic data. This rehabilitation plan includes vocal training, comprehension questions, and communication exercises. A prompt sentence is input into the generative AI model to generate the rehabilitation content. An example of a prompt sentence is, "Please generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[0325] Emotion recognition with emotion engine

[0326] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session, and the recorded data is sent to the emotion engine on the server, which analyzes it and recognizes the patient's emotional state.

[0327] Adaptive adjustment of rehabilitation plans

[0328] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine, for example adding relaxation exercises if the patient is feeling stressed.

[0329] Rehabilitation

[0330] The user (patient) follows instructions from the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation process as video and audio.

[0331] Rehabilitation progress monitoring

[0332] The device transmits real-time information about the rehabilitation progress to a server, which analyzes the data and provides immediate feedback to the user as needed.

[0333] Create and share daily reports

[0334] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The report is then sent to the patient's doctor and family via email or a dedicated application.

[0335] As described above, the present invention is a system that provides rehabilitation plans tailored to the individual patient's condition, and enables real-time monitoring and immediate feedback, thereby achieving efficient and effective rehabilitation.

[0336] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0337] Step 1: Enter and submit patient information

[0338] The user (medical staff) inputs basic information such as the patient's age, disability level, and medical information into the terminal.

[0339] Input: Patient's age, disability level, medical information

[0340] Action: Fill out the form and click the "Submit" button.

[0341] Output: The entered patient information is sent from the terminal to the server.

[0342] Step 2: Receiving and analyzing patient information

[0343] The server receives the patient information transmitted from the terminal.

[0344] Input: Patient information sent from the terminal

[0345] Operation: Receive data via the receiving API.

[0346] Output: Pass the received patient information to the analysis program.

[0347] Step 3: Data cleansing and basic data generation

[0348] The server analyzes the received information and cleanses the data to generate the base data.

[0349] Input: Received patient information

[0350] How it works: The analysis program cleanses the data, extracting and organizing only the necessary items.

[0351] Output: Cleansed underlying data

[0352] Step 4: Generate a rehabilitation plan

[0353] The server automatically generates a rehabilitation plan based on the basic data.

[0354] Input: Cleansed underlying data

[0355] How it works: A prompt sentence is input into the generative AI model, which generates rehabilitation content (vocal training, comprehension questions, and communication exercises).

[0356] Example prompt: "Generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[0357] Output: Automatically generated rehabilitation plan

[0358] Step 5: Submit and view your rehabilitation plan

[0359] The server transmits the generated rehabilitation plan to the terminal, and the terminal displays the rehabilitation plan to the user.

[0360] Input: Auto-generated rehabilitation plan

[0361] Operation: The server sends planning data to the terminal, which receives the data and displays it on the screen.

[0362] Output: User-viewable rehabilitation plan display

[0363] Step 6: Recording your rehabilitation sessions

[0364] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session.

[0365] Input: Patient behavior and voice during rehabilitation session

[0366] How it works: Cameras and microphones collect and record data in real time.

[0367] Output: Recorded video and audio data

[0368] Step 7: Sending Emotion Data and Emotion Recognition

[0369] The terminal sends the recorded data to the emotion engine, and the server recognizes the emotional state.

[0370] Input: Recorded video and audio data

[0371] How it works: Data is sent to the server's emotion engine API, where it is analyzed by emotion recognition algorithms.

[0372] Output: Recognized emotional state data

[0373] Step 8: Adaptively adjusting the rehabilitation plan

[0374] The server adaptively adjusts the rehabilitation plan based on the results of the emotion engine.

[0375] Input: Recognized emotional state data

[0376] Action: Re-analyze the rehabilitation plan and add relaxation exercises and content as needed.

[0377] Output: Coordinated rehabilitation plan

[0378] Step 9: Implement and record rehabilitation

[0379] The user (patient) performs rehabilitation by following the instructions on the terminal, and the terminal records the progress.

[0380] Input: Coordinated rehabilitation plan

[0381] How it works: Follow the instructions and use the camera and microphone to record your vocal exercises and workouts in real time.

[0382] Output: Newly recorded video and audio of the rehabilitation session

[0383] Step 10: Real-time analysis of rehabilitation progress and provision of feedback

[0384] The device monitors the progress of rehabilitation in real time, and the server provides immediate feedback based on the analysis results.

[0385] Input: Real-time rehabilitation progress data

[0386] What it does: Sends data to a server that analyzes progress and forwards instant advice and encouragement to the device.

[0387] Output: Immediate feedback provided to the user

[0388] Step 11: Create and share daily reports

[0389] The terminal transmits the rehabilitation data for one day to the server, which automatically generates a daily report.

[0390] Input: One day of rehabilitation data

[0391] What it does: Aggregate data and generate reports that show progress, achievements, and areas for improvement.

[0392] Output: Auto-generated daily report

[0393] Step 12: Daily Report Delivery

[0394] The server distributes the generated daily reports to the patient's doctor and family.

[0395] Input: Auto-generated daily report

[0396] What it does: Reports are delivered via email or a dedicated application.

[0397] Output: Daily reports sent to your doctor and family

[0398] (Application example 2)

[0399] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0400] Patients undergoing rehabilitation often experience significant mental stress and anxiety. Conventional rehabilitation systems struggle to recognize a patient's emotional and psychological state in real time and provide appropriate support. This can result in insufficient rehabilitation benefits and further problems. While there is a need to understand a patient's mental crisis in real time and respond immediately in emergencies, current technology is unable to adequately address this need.

[0401] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0402] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio for recording the patient's rehabilitation, means for receiving the recorded data and analyzing the emotional state, means for creating a daily report based on the analysis results and distributing it to a predetermined recipient, and means for making an emergency call if the user's psychological state is judged to be dangerous. This makes it possible to analyze the patient's emotional state in real time and make adaptive adjustments to the rehabilitation plan, as well as to make an emergency call immediately in the event of an emergency.

[0403] "Patient" refers to a person with a health condition requiring rehabilitation.

[0404] "Age" is information that indicates the period of time that has elapsed since the patient's date of birth until the present.

[0405] "Aphasia level" refers to a scale or information that indicates the degree of impairment of a patient's language function.

[0406] "Medical information" refers to various information related to medical treatment, such as patient charts and medical records.

[0407] "Input means" refers to a device or method for inputting data such as the patient's age, aphasia level, and medical information into the system.

[0408] "Receiving means" refers to a device or method by which the server receives patient information transmitted from the terminal.

[0409] The term "analysis means" refers to a method or device for analyzing received patient information and generating data necessary for formulating a rehabilitation plan.

[0410] "Rehabilitation plan" refers to a program that includes rehabilitation activities appropriate for a patient.

[0411] "Display means" refers to a device or method for visually showing the generated rehabilitation plan to the user.

[0412] "Instruction means" refers to a device or method for encouraging a user to take specific actions or practice based on a rehabilitation plan.

[0413] "Recording Means" refers to any device or method for capturing video and audio recordings of a patient's rehabilitation session.

[0414] "Emotional state" refers to a state that indicates the patient's psychological feelings.

[0415] "Emergency notification means" refers to a device or method for immediately reporting when the user's psychological state is judged to be dangerous.

[0416] "Daily report" refers to a report summarizing the results of daily rehabilitation.

[0417] "Delivery means" refers to a device or method for transmitting the generated daily report to a predetermined recipient.

[0418] "Real-time" refers to processing and response occurring immediately, without any time delay.

[0419] MODE FOR CARRYING OUT THE INVENTION

[0420] This invention is a technology for monitoring a patient's emotional state in real time in a rehabilitation system and providing an adaptive rehabilitation plan. At the same time, it also has a function to make an emergency call if the user's psychological state is judged to be dangerous. This system mainly consists of three elements: a server, a terminal, and the user.

[0421] System configuration and operation

[0422] 1. Collection and analysis of patient information

[0423] The server receives the patient's age, aphasia level, medical information, etc. sent from the terminal and analyzes this information. The user (medical staff) inputs the necessary data through the terminal and sends it to the server. This process generates basic data for understanding the patient's condition.

[0424] 2. Generating a rehabilitation plan

[0425] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user (patient).

[0426] 3. Emotion recognition and adaptive regulation by the emotion engine

[0427] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session, and sends this information to the emotion engine. The emotion engine analyzes this information and recognizes the user's emotional state. The server adaptively adjusts the rehabilitation plan based on the emotion recognition results.

[0428] 4. Rehabilitation implementation and progress monitoring

[0429] The user (patient) follows instructions on the device to perform rehabilitation. The rehabilitation session is recorded with video and audio, and the device sends this data to a server. The server analyzes the data and provides real-time feedback to the user via the device as needed.

[0430] 5. Emergency call

[0431] If the user's psychological state is judged to be dangerous, for example, if they show fear or anger, the device will immediately make an emergency call, which may include automatically notifying designated emergency contacts and contacting public authorities such as the police.

[0432] 6. Create and share daily reports

[0433] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family. The daily report includes rehabilitation progress, achievements, and next steps for improvement.

[0434] Specific examples

[0435] As a concrete example, consider a case where a user is undergoing rehabilitation using a smartphone application. During the rehabilitation session, the device's camera and microphone constantly monitor the user's facial expressions and voice. An emotion engine analyzes the user's emotions, and if the user expresses fear or anger, the device immediately initiates an emergency call. This call includes contacting designated emergency contacts and public authorities.

[0436] Prompt Sentence Examples

[0437] For example, you can use the following prompts for a generative AI model to achieve emotion recognition and emergency calling capabilities:

[0438] Implement an application that recognizes emotions from the user's facial and voice data, and makes an emergency call when danger is detected.

[0439] Facial expression data is captured in real time from the smartphone camera and analyzed using emotion recognition models.

[0440] · It also analyzes voice data to determine emotional state.

[0441] If the user expresses fear or anger, it will notify your emergency contacts.

[0442] The system allows for real-time analysis of a patient's emotional state, providing adaptive rehabilitation plans, as well as immediate response to dangerous situations.

[0443] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0444] Step 1:

[0445] The server receives the patient's age, aphasia level, and medical information sent from the device. Based on this input data, it prepares to analyze the patient's information. As a result of the analysis, basic data for understanding the patient's condition is generated.

[0446] Step 2:

[0447] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal so that the user (patient or medical staff) can review it.

[0448] Step 3:

[0449] The device uses a camera and microphone to record the user's facial expressions, voice tone, and word choice during the rehabilitation session. This information is then sent to an emotion engine to analyze the user's emotional state. The input to this process is video and audio data, and the output is emotion recognition results.

[0450] Step 4:

[0451] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.). The model performs data calculations based on the input video and audio data, and obtains the emotional state as output.

[0452] Step 5:

[0453] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results. For example, if the user is feeling stressed, it adds relaxation exercises. Based on the emotion recognition results as input, the server generates a revised and adjusted rehabilitation plan as output.

[0454] Step 6:

[0455] The user (patient) follows the instructions on the device to perform rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[0456] Step 7:

[0457] The device monitors the progress of rehabilitation in real time and transmits the video and audio data to the server. The server analyzes this data and provides real-time feedback to the user via the device as needed. The input is real-time video and audio data, and the output is feedback information.

[0458] Step 8:

[0459] The device continuously monitors the user's psychological state based on the data recorded during the rehabilitation session. If the user's psychological state is judged to be dangerous, the device immediately makes an emergency call. The input is the analysis result of the psychological state, and the output is the call action.

[0460] Step 9:

[0461] When the rehabilitation session ends, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family via email or a dedicated application. The input is the day's rehabilitation data, and the output is the daily report.

[0462] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0464] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0465] [Second embodiment]

[0466] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0467] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0468] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0469] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0470] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0471] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0472] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0473] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0474] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0475] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0476] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0477] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0478] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates a series of steps: collecting and analyzing patient information, creating rehabilitation plans, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, we will explain the program processing of this system in natural language and provide detailed examples.

[0479] System configuration and operation

[0480] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[0481] 1. Collecting patient information

[0482] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[0483] 2. Analysis of patient information

[0484] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[0485] 3. Generating a rehabilitation plan

[0486] The server automatically generates an individual rehabilitation plan based on the analysis results. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises. The generated rehabilitation plan is sent to the device and can be viewed by the user.

[0487] 4. Rehabilitation

[0488] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[0489] 5. Rehabilitation progress monitoring

[0490] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides real-time feedback to the user via the device as needed.

[0491] 6. Create and share daily reports

[0492] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[0493] Specific examples

[0494] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the instructions on the device, recording the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[0495] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[0496] The processing flow will be explained below.

[0497] Step 1: Enter patient information

[0498] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[0499] Step 2: Receiving and analyzing patient information

[0500] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[0501] Step 3: Generate a rehabilitation plan

[0502] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[0503] Step 4: Submit your rehabilitation plan

[0504] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[0505] Step 5: Rehabilitation

[0506] The user (patient) begins rehabilitation by following instructions on the device, which uses a camera and microphone to record the rehabilitation session as video and audio.

[0507] Step 6: Real-time data transmission

[0508] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[0509] Step 7: Real-time data analysis and feedback

[0510] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[0511] Step 8: Ending the rehabilitation session and sending data

[0512] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[0513] Step 9: Create a daily report

[0514] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[0515] Step 10: Share the report

[0516] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[0517] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[0518] Example 1

[0519] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0520] The present invention aims to maximize the effectiveness of rehabilitation for patients with aphasia or brain damage. Conventional rehabilitation methods have the problem that it takes a great deal of time and effort to create an optimal rehabilitation plan for each individual patient and provide feedback on the implementation status. Furthermore, effective rehabilitation is difficult due to limited means for monitoring the patient's progress in real time. It is desirable to solve these problems.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0522] In this invention, the server includes means for receiving basic patient information and storing it in a database, means for analyzing the received patient information and generating basic data for evaluating the patient's condition, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the generated rehabilitation plan and issuing instructions to the user, means for recording the patient's rehabilitation with a camera and microphone, means for receiving the recorded data, monitoring it in real time, and generating immediate feedback, and means for analyzing the recorded data, automatically generating daily reports, and distributing them to specified recipients. This makes it possible to quickly provide an optimized rehabilitation plan for each patient, monitor its implementation in real time, and provide immediate feedback.

[0523] "Basic patient information" refers to basic data necessary for the patient's medical rehabilitation, such as the patient's age, gender, aphasia level, and medical record information.

[0524] "Database" refers to a collection of data used by the Server to efficiently store, manage, and retrieve patient information.

[0525] "Baseline data" refers to basic evaluation data generated by analyzing received patient information to evaluate the patient's condition.

[0526] A "rehabilitation plan" refers to a detailed rehabilitation schedule that includes voice training, comprehension exercises, and communication exercises that are optimized for the patient's individual condition.

[0527] "Means for issuing instructions to the user" refers to the system operation and screen display mechanisms for appropriately notifying the user of the generated rehabilitation plan and issuing instructions.

[0528] "Camera and microphone" refers to the video and audio input devices used to record the patient's rehabilitation progress.

[0529] "Means for real-time monitoring" refers to a system that instantly monitors the progress of rehabilitation on the spot and analyzes the data.

[0530] "Immediate feedback" refers to assessments and instructions provided to patients quickly based on real-time monitored data.

[0531] "Daily report" refers to a detailed report summarizing the content and progress of each day's rehabilitation sessions.

[0532] The "predetermined recipient" refers to medical staff or family members of patients who are designated in advance as recipients of the generated daily report.

[0533] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates the collection and analysis of patient information, the creation of rehabilitation plans, the implementation of rehabilitation, progress monitoring, and the creation and sharing of daily reports, providing rehabilitation that is optimized for each individual patient.

[0534] System configuration and operation

[0535] This system mainly consists of three elements: a server, a terminal, and users (patients and medical staff).

[0536] Collecting patient information

[0537] The user (medical staff) enters basic patient information (age, gender, aphasia level, medical record information, etc.) into the terminal. The terminal formats the entered information and sends it to the server. The server stores the patient information received from the terminal in a database.

[0538] Patient information analysis

[0539] The server analyzes the patient information stored in the database and generates basic data to evaluate the patient's condition, which is then used to create an individual rehabilitation plan.

[0540] Generate rehabilitation plans

[0541] The server automatically generates an appropriate rehabilitation plan based on the generated basic data. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises, and the generated plan is sent to the device. The user (medical staff) can then check the rehabilitation plan through the device.

[0542] Rehabilitation

[0543] The user (patient) follows the rehabilitation instructions displayed on the device. The device displays the rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises. The device also records the rehabilitation progress as video and audio.

[0544] Rehabilitation progress monitoring

[0545] The device monitors the progress of rehabilitation in real time and transmits video and audio data to the server, which analyzes the received data and evaluates the rehabilitation progress. If necessary, the server generates real-time feedback and provides it to the user via the device.

[0546] Create and share daily reports

[0547] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server, which then automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated app.

[0548] Specific examples

[0549] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the device's instructions and records the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the doctor in charge and the patient's family.

[0550] By inputting the following prompt sentences into the generative AI model, the system automatically generates an optimal rehabilitation plan for the patient:

[0551] "A 70-year-old male patient suffers from moderate aphasia. Please create an optimal rehabilitation plan for this patient. The patient's basic information is as follows: Age: 70, Gender: Male, Current Status: Moderate aphasia."

[0552] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[0553] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0554] Processing Steps

[0555] Step 1:

[0556] The user (medical staff) inputs basic information about the patient (age, sex, aphasia level, medical record information, etc.) into the terminal.

[0557] Input: Patient's age, gender, aphasia level, medical record information

[0558] How it works: The user enters patient information into a dedicated application on the device and clicks the send button.

[0559] Output: Information entered into the terminal is formatted and sent to the server.

[0560] Step 2:

[0561] The terminal formats the entered patient information and transmits it to the server.

[0562] Input: Basic patient information entered by the user

[0563] Operation: The device converts the information into a specific format and sends it to the server.

[0564] Output: Formatted patient information

[0565] Step 3:

[0566] The server stores the patient information received from the terminal in a database.

[0567] Input: Formatted patient details

[0568] What happens: The server stores the information in a database

[0569] Output: Patient information stored in the database

[0570] Step 4:

[0571] The server analyzes the patient information stored in the database and generates basic data for evaluating the patient's condition.

[0572] Input: Patient information stored in the database

[0573] How it works: The server analyzes the information using natural language processing algorithms and generates the underlying data.

[0574] Output: The underlying data generated by the analysis

[0575] Step 5:

[0576] The server automatically generates an appropriate rehabilitation plan based on the generated basic data.

[0577] Input: Basic data

[0578] How it works: The server uses the generative AI model to create a rehabilitation plan.

[0579] Output: Automatically generated rehabilitation plan

[0580] Step 6:

[0581] The server transmits the generated rehabilitation plan to the terminal so that the user (medical staff) can check it.

[0582] Input: Rehabilitation Plan

[0583] Operation: The server sends the rehabilitation plan to the device.

[0584] Output: Rehabilitation plan sent to the terminal

[0585] Step 7:

[0586] The user (patient) follows the rehabilitation instructions displayed on the terminal and performs rehabilitation.

[0587] Input: Rehabilitation plan displayed on terminal

[0588] Action: The user uses the camera and microphone to perform rehabilitation.

[0589] Output: Data recording rehabilitation progress

[0590] Step 8:

[0591] The device displays rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises.

[0592] Input: Rehabilitation Plan

[0593] Operation: The device displays rehabilitation instructions to the user and records the rehabilitation progress with video and audio.

[0594] Output: Recorded video and audio data

[0595] Step 9:

[0596] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[0597] Input: Recorded video and audio data

[0598] Action: The device sends data to the server.

[0599] Output: Video and audio data sent to the server

[0600] Step 10:

[0601] The server analyzes the received data and evaluates the degree of rehabilitation achievement.

[0602] Input: Video and audio data

[0603] Action: The server analyzes the data and generates an evaluation result.

[0604] Output: Evaluation results regarding rehabilitation achievement

[0605] Step 11:

[0606] The server generates and provides immediate feedback to the user via the terminal as needed.

[0607] Input: Evaluation results regarding rehabilitation achievement

[0608] What it does: The server generates and sends immediate feedback to the user

[0609] Output: Immediate feedback provided to the user

[0610] Step 12:

[0611] When the rehabilitation session ends, the terminal transmits the rehabilitation data for that day to the server.

[0612] Input: Rehabilitation data for one day

[0613] Operation: The device collects data and sends it to the server

[0614] Output: Rehabilitation data for one day sent to the server

[0615] Step 13:

[0616] The server automatically generates daily reports based on the data received.

[0617] Input: Rehabilitation data for one day

[0618] How it works: The server analyzes the data and generates a daily report.

[0619] Output: Auto-generated daily reports

[0620] Step 14:

[0621] The server distributes the generated report to the doctor in charge and the patient's family.

[0622] Input: Daily Report

[0623] How it works: The server delivers reports via email or a dedicated application.

[0624] Output: Daily reports delivered to treating physicians and patient families

[0625] In this way, this system achieves efficient and sustainable rehabilitation by linking data input, calculation, and output at each step.

[0626] (Application example 1)

[0627] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0628] Current rehabilitation systems struggle to efficiently and continuously monitor and automatically generate personalized rehabilitation plans. Furthermore, they lack the ability to manage rehabilitation progress in real time and provide immediate feedback, making it difficult to achieve effective rehabilitation. Furthermore, there are no systems suitable for use in rehabilitation centers or brick-and-mortar stores, preventing patients from maximizing the benefits of their rehabilitation.

[0629] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0630] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical record information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio data for recording the patient's rehabilitation, means for receiving and analyzing the recorded data, means for creating daily reports based on the analysis results and distributing them to specified recipients, and means for providing the rehabilitation plan to a terminal at the rehabilitation center and managing the progress. This enables automatic generation of an individualized rehabilitation plan and real-time monitoring of rehabilitation sessions, and real-time feedback to achieve effective rehabilitation.

[0631] "Patients" refers to those with aphasia, brain damage, etc. who require rehabilitation.

[0632] "Age" is the period indicated by the patient's biological number.

[0633] "Aphasia level" is an index showing the degree of aphasia.

[0634] "Medical record information" refers to medical data including a patient's medical history and treatment records.

[0635] "Means for receiving and analyzing information" refers to the processes and tools for obtaining and analyzing patient-entered information.

[0636] "Means for automatically generating rehabilitation plans" refers to technology that automatically creates individualized rehabilitation programs based on collected patient information.

[0637] "Means for providing instructions" refers to a method for presenting the generated rehabilitation plan to the patient or medical staff.

[0638] "Video and audio recording means" refers to devices and methods for recording rehabilitation sessions using cameras and microphones.

[0639] "Means for receiving and analyzing recorded data" refers to the processes and tools for receiving and analyzing recorded video and audio data.

[0640] "Means for creating and distributing daily reports" refers to a system and method for organizing rehabilitation progress information on a daily basis and distributing it to designated recipients (such as medical staff or family members).

[0641] "Rehabilitation center terminal" refers to computer equipment installed in a rehabilitation facility or clinic that is used to review and implement rehabilitation plans.

[0642] "Progress management means" refers to systems and methods that track a patient's rehabilitation progress and provide appropriate measures and feedback accordingly.

[0643] The system of this invention is designed to support effective rehabilitation for patients with aphasia or brain damage in physical rehabilitation centers or clinics. This system mainly consists of a server, terminals, and users (patients and medical staff). Specific embodiments of the present invention are described in detail below.

[0644] System configuration

[0645] 1. Collecting patient information

[0646] The server receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff) into the terminal. The terminal sends this information to the server, which then analyzes it.

[0647] 2. Analysis of patient information

[0648] The server analyzes the received patient information and generates individual basic data, which is used to prepare an optimal rehabilitation plan for each patient.

[0649] 3. Generating a rehabilitation plan

[0650] The server automatically generates a rehabilitation plan based on the analysis results. The plan is sent to the terminal and confirmed by the user (medical staff and patient). The rehabilitation plan includes vocal exercises, comprehension questions, and communication exercises.

[0651] 4. Rehabilitation

[0652] Patients undergo rehabilitation by following instructions on a device in the facility. The rehabilitation content is displayed on a screen, and patients use a camera and microphone connected to the device to perform voice training and exercises. The device records the rehabilitation session and sends the video and audio data to a server.

[0653] 5. Rehabilitation progress monitoring

[0654] The server monitors the progress of rehabilitation in real time and provides immediate feedback to the device as needed, helping to ensure smooth rehabilitation progress for the patient.

[0655] 6. Create and share daily reports

[0656] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server analyzes the received data and automatically generates a daily report. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated application.

[0657] Specific examples

[0658] For example, if a 70-year-old male patient suffers from moderate aphasia, medical staff enter the patient's basic information into the device and send it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results and sends it to the device. The patient performs rehabilitation according to the device's instructions, recording the process with a camera and microphone. The device then sends the recorded data to the server in real time, which analyzes the data and provides immediate feedback. After the rehabilitation session ends, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[0659] Hardware and software used

[0660] Smartphone: Equipped with a camera and microphone, it is used as a device to record the patient's rehabilitation progress.

[0661] Server: Analyzes patient information, generates rehabilitation plans, performs real-time monitoring, and creates daily reports.

[0662] Software: Python program, server communication via HTTP protocol.

[0663] Prompt Sentence Examples

[0664] "Please provide a specific use case for a smartphone application to help patients with their rehabilitation. Please explain in detail how it collects patient information, captures rehabilitation plans, records rehabilitation sessions, and generates and shares daily reports."

[0665] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0666] System program processing steps

[0667] Step 1:

[0668] The terminal receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff). This basic information is formatted by the terminal and sent to the server. The input data includes the patient's biometric number and medical records related to rehabilitation.

[0669] Step 2:

[0670] The server receives and analyzes the patient information sent from the device. Specifically, it uses each data item as material for identifying the disease and developing a rehabilitation plan to generate basic patient data based on the received data. This basic data includes information to clarify the patient's condition and rehabilitation goals.

[0671] Step 3:

[0672] The server automatically generates a rehabilitation plan based on the analysis results. The plan includes voice training, comprehension questions, and communication exercises tailored to each patient. The automatically generated plan is sent to the device and provided to medical staff and the patient.

[0673] Step 4:

[0674] The user (patient) performs rehabilitation by following instructions on the device. The device records each rehabilitation session with a camera and microphone and transmits the data in real time to a server. The recorded data includes rehabilitation progress and specific exercises.

[0675] Step 5:

[0676] The server analyzes the recorded rehabilitation data received in real time and generates feedback on the spot, returning it to the device. The feedback includes vocal errors and exercise improvements, allowing patients to receive immediate corrections as needed during rehabilitation.

[0677] Step 6:

[0678] After the rehabilitation session, the device sends the day's worth of rehabilitation data to the server, which then automatically generates a daily report based on the received data, detailing rehabilitation progress, achievements, and recommended next exercises.

[0679] Step 7:

[0680] Finally, the server delivers the generated daily report to the designated recipient (physician or family member) via email or a dedicated application. The delivered report serves as a reference for the recipient to check the patient's rehabilitation progress and plan future treatment plans.

[0681] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0682] This invention is a system that uses AI technology to support the rehabilitation of patients with aphasia or brain damage, and by combining it with an emotion engine that recognizes the user's emotions, it implements rehabilitation plans more adaptively and effectively. This system automates a series of steps: collecting and analyzing patient information, generating rehabilitation plans, recognizing emotions, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[0683] System configuration and operation

[0684] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[0685] 1. Collecting patient information

[0686] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[0687] 2. Analysis of patient information

[0688] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[0689] 3. Generating a rehabilitation plan

[0690] The server automatically generates an individual rehabilitation plan based on the analysis results. This rehabilitation plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user.

[0691] 4. Emotion Recognition with Emotion Engine

[0692] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session and transmits this information to the emotion engine, which analyzes this information and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.).

[0693] 5. Adaptive adjustment of rehabilitation plans

[0694] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it can add relaxation exercises.

[0695] 6. Rehabilitation

[0696] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[0697] 7. Rehabilitation progress monitoring

[0698] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides immediate feedback to the user via the device as needed.

[0699] 8. Create and share daily reports

[0700] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[0701] Specific examples

[0702] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic patient data. Next, based on the analysis results, the server automatically generates an individual rehabilitation plan including speech exercises and comprehension questions and sends it to the device.

[0703] The user (patient) performs rehabilitation according to instructions on the device, recording the process with a camera and microphone. The device then sends the recorded data to the emotion engine, which analyzes the user's facial expressions and vocal tone to recognize their emotional state. The server adaptively adjusts the rehabilitation content based on the recognized emotion results and provides it to the user. For example, if the user is tired, it can add relaxation exercises.

[0704] After the rehabilitation session, the server creates a daily report based on the data from the device and distributes it to the patient's doctor and family. In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, achieving efficient and sustainable rehabilitation.

[0705] The processing flow will be explained below.

[0706] Step 1: Enter patient information

[0707] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[0708] Step 2: Receiving and analyzing patient information

[0709] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[0710] Step 3: Generate a rehabilitation plan

[0711] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[0712] Step 4: Submit your rehabilitation plan

[0713] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[0714] Step 5: Emotion Recognition with the Emotion Engine

[0715] The device uses a camera and microphone to record the user's (patient's) facial expressions, voice tone, and word choice during the rehabilitation session. The device then sends this recorded data to the emotion engine, which analyzes the recorded data and recognizes the user's emotional state.

[0716] Step 6: Adaptively adjusting the rehabilitation plan

[0717] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it adds relaxation exercises.

[0718] Step 7: Rehabilitation

[0719] The user performs rehabilitation by following instructions from the device, which uses a camera and microphone to record video and audio of the rehabilitation session.

[0720] Step 8: Real-time data transmission

[0721] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[0722] Step 9: Real-time data analysis and feedback

[0723] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[0724] Step 10: Ending the rehabilitation session and sending data

[0725] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[0726] Step 11: Create a daily report

[0727] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[0728] Step 12: Share the report

[0729] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[0730] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[0731] Example 2

[0732] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0733] Currently, rehabilitation for patients with aphasia and brain damage requires flexible rehabilitation plans tailored to each individual, as the effectiveness of these plans varies from patient to patient. However, existing rehabilitation systems lack the ability to appropriately grasp a patient's emotional state and dynamically adjust the rehabilitation plan. They also lack the ability to monitor the progress of rehabilitation in real time and provide immediate feedback. Therefore, there is a need for a system that can generate rehabilitation plans tailored to each patient's individual needs and recognize and adapt to their emotional state during the implementation process.

[0734] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input the patient's age, disability level, and medical information; a means for receiving and analyzing the patient information; a means for automatically generating a rehabilitation plan based on the analysis results; a means for collecting data on the patient's facial expressions, tone of voice, and word choice during the rehabilitation session to recognize the patient's emotional state; a means for adaptively adjusting the rehabilitation plan based on the recognized emotional state; a means for receiving and analyzing the recorded data; a means for creating a daily report based on the analysis results and distributing it to a predetermined recipient; and a means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback. This enables the creation and provision of an optimal rehabilitation plan tailored to the individual patient's condition. Furthermore, by accurately understanding the patient's emotional state during the rehabilitation process and making adaptive adjustments, the effectiveness of the rehabilitation can be maximized.

[0735] "User" is the medical professional or person who operates the system, inputs patient data, and reviews and adjusts rehabilitation plans.

[0736] "Patients" refer to people who need rehabilitation for conditions such as aphasia or brain damage.

[0737] A "server" is a computer system that analyzes patient information, generates rehabilitation plans, and stores and processes data.

[0738] A "terminal" is an electronic device such as a computer, tablet, or smartphone that is directly operated by a user or patient.

[0739] "Patient information" refers to basic data necessary for generating and adjusting a rehabilitation plan, such as age, disability level, and medical information.

[0740] A "rehabilitation plan" is a plan that includes specific rehabilitation content and steps and is automatically generated based on the patient's condition.

[0741] "Emotion engine" refers to algorithms and machine learning models that analyze a patient's facial expressions, tone of voice, and word choice data to recognize their emotional state.

[0742] The "Daily Report" is a report that summarizes the daily activity record, including the progress, achievements, and areas for improvement of the rehabilitation session.

[0743] "Real-time monitoring" is the process of observing rehabilitation progress in real time and collecting and analyzing data.

[0744] "Immediate feedback" refers to advice or instructions provided immediately to a user or patient based on the results of data analysis.

[0745] "Adaptive adjustment" is the process of dynamically changing or adjusting a rehabilitation plan based on the patient's emotional state and progress.

[0746] This invention provides a system for supporting rehabilitation for patients with aphasia or brain damage, recognizing the patient's emotional state and adaptively adjusting the rehabilitation plan based on that state. The system automates the collection and analysis of patient information, generation of rehabilitation plans, emotion recognition, rehabilitation implementation, progress monitoring, and creation and sharing of daily reports.

[0747] The main elements of this system are the server, terminals, and users (patients and medical staff). Below we will explain in detail how each element works together.

[0748] Hardware and Software Used

[0749] Hardware

[0750] 1. Devices: Personal computers, tablets, smartphones, etc. used by users and patients.

[0751] 2. Camera and microphone: Devices for collecting the patient's facial expressions and voice.

[0752] 3. Server: A high-performance computer for analyzing data and generating rehabilitation plans.

[0753] software

[0754] 1. Emotion engine: An AI model to analyze patient emotions (e.g., Google Cloud's Natural Language API or Azure Emotion API).

[0755] 2. Analysis algorithm: Software for analyzing patient information and generating rehabilitation plans.

[0756] 3. Database system: A system for storing patient information and rehabilitation progress data (e.g., MySQL, PostgreSQL).

[0757] 4. Rehabilitation support app: A dedicated application to help patients carry out rehabilitation.

[0758] System example and operation

[0759] Collecting patient information

[0760] The user (medical staff) uses a terminal to input basic information such as the patient's age, disability level, medical information, etc. The terminal then transmits the input information to the server.

[0761] Patient information analysis

[0762] The server receives the patient information sent from the terminal, cleanses the information using an analysis program, and generates basic data.

[0763] Generate rehabilitation plans

[0764] The server automatically generates a rehabilitation plan based on the generated basic data. This rehabilitation plan includes vocal training, comprehension questions, and communication exercises. A prompt sentence is input into the generative AI model to generate the rehabilitation content. An example of a prompt sentence is, "Please generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[0765] Emotion recognition with emotion engine

[0766] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session, and the recorded data is sent to the emotion engine on the server, which analyzes it and recognizes the patient's emotional state.

[0767] Adaptive adjustment of rehabilitation plans

[0768] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine, for example adding relaxation exercises if the patient is feeling stressed.

[0769] Rehabilitation

[0770] The user (patient) follows instructions from the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation process as video and audio.

[0771] Rehabilitation progress monitoring

[0772] The device transmits real-time information about the rehabilitation progress to a server, which analyzes the data and provides immediate feedback to the user as needed.

[0773] Create and share daily reports

[0774] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The report is then sent to the patient's doctor and family via email or a dedicated application.

[0775] As described above, the present invention is a system that provides rehabilitation plans tailored to the individual patient's condition, and enables real-time monitoring and immediate feedback, thereby achieving efficient and effective rehabilitation.

[0776] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0777] Step 1: Enter and submit patient information

[0778] The user (medical staff) inputs basic information such as the patient's age, disability level, and medical information into the terminal.

[0779] Input: Patient's age, disability level, medical information

[0780] Action: Fill out the form and click the "Submit" button.

[0781] Output: The entered patient information is sent from the terminal to the server.

[0782] Step 2: Receiving and analyzing patient information

[0783] The server receives the patient information transmitted from the terminal.

[0784] Input: Patient information sent from the terminal

[0785] Operation: Receive data via the receiving API.

[0786] Output: Pass the received patient information to the analysis program.

[0787] Step 3: Data cleansing and basic data generation

[0788] The server analyzes the received information and cleanses the data to generate the base data.

[0789] Input: Received patient information

[0790] How it works: The analysis program cleanses the data, extracting and organizing only the necessary items.

[0791] Output: Cleansed underlying data

[0792] Step 4: Generate a rehabilitation plan

[0793] The server automatically generates a rehabilitation plan based on the basic data.

[0794] Input: Cleansed underlying data

[0795] How it works: A prompt sentence is input into the generative AI model, which generates rehabilitation content (vocal training, comprehension questions, and communication exercises).

[0796] Example prompt: "Generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[0797] Output: Automatically generated rehabilitation plan

[0798] Step 5: Submit and view your rehabilitation plan

[0799] The server transmits the generated rehabilitation plan to the terminal, and the terminal displays the rehabilitation plan to the user.

[0800] Input: Auto-generated rehabilitation plan

[0801] Operation: The server sends planning data to the terminal, which receives the data and displays it on the screen.

[0802] Output: User-viewable rehabilitation plan display

[0803] Step 6: Recording your rehabilitation sessions

[0804] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session.

[0805] Input: Patient behavior and voice during rehabilitation session

[0806] How it works: Cameras and microphones collect and record data in real time.

[0807] Output: Recorded video and audio data

[0808] Step 7: Sending Emotion Data and Emotion Recognition

[0809] The terminal sends the recorded data to the emotion engine, and the server recognizes the emotional state.

[0810] Input: Recorded video and audio data

[0811] How it works: Data is sent to the server's emotion engine API, where it is analyzed by emotion recognition algorithms.

[0812] Output: Recognized emotional state data

[0813] Step 8: Adaptively adjusting the rehabilitation plan

[0814] The server adaptively adjusts the rehabilitation plan based on the results of the emotion engine.

[0815] Input: Recognized emotional state data

[0816] Action: Re-analyze the rehabilitation plan and add relaxation exercises and content as needed.

[0817] Output: Coordinated rehabilitation plan

[0818] Step 9: Implement and record rehabilitation

[0819] The user (patient) performs rehabilitation by following the instructions on the terminal, and the terminal records the progress.

[0820] Input: Coordinated rehabilitation plan

[0821] How it works: Follow the instructions and use the camera and microphone to record your vocal exercises and workouts in real time.

[0822] Output: Newly recorded video and audio of the rehabilitation session

[0823] Step 10: Real-time analysis of rehabilitation progress and provision of feedback

[0824] The device monitors the progress of rehabilitation in real time, and the server provides immediate feedback based on the analysis results.

[0825] Input: Real-time rehabilitation progress data

[0826] What it does: Sends data to a server that analyzes progress and forwards instant advice and encouragement to the device.

[0827] Output: Immediate feedback provided to the user

[0828] Step 11: Create and share daily reports

[0829] The terminal transmits the rehabilitation data for one day to the server, which automatically generates a daily report.

[0830] Input: One day of rehabilitation data

[0831] What it does: Aggregate data and generate reports that show progress, achievements, and areas for improvement.

[0832] Output: Auto-generated daily report

[0833] Step 12: Daily Report Delivery

[0834] The server distributes the generated daily reports to the patient's doctor and family.

[0835] Input: Auto-generated daily report

[0836] What it does: Reports are delivered via email or a dedicated application.

[0837] Output: Daily reports sent to your doctor and family

[0838] (Application example 2)

[0839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] Patients undergoing rehabilitation often experience significant mental stress and anxiety. Conventional rehabilitation systems struggle to recognize a patient's emotional and psychological state in real time and provide appropriate support. This can result in insufficient rehabilitation benefits and further problems. While there is a need to understand a patient's mental crisis in real time and respond immediately in emergencies, current technology is unable to adequately address this need.

[0841] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0842] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio for recording the patient's rehabilitation, means for receiving the recorded data and analyzing the emotional state, means for creating a daily report based on the analysis results and distributing it to a predetermined recipient, and means for making an emergency call if the user's psychological state is judged to be dangerous. This makes it possible to analyze the patient's emotional state in real time and make adaptive adjustments to the rehabilitation plan, as well as to make an emergency call immediately in the event of an emergency.

[0843] "Patient" refers to a person with a health condition requiring rehabilitation.

[0844] "Age" is information that indicates the period of time that has elapsed since the patient's date of birth until the present.

[0845] "Aphasia level" refers to a scale or information that indicates the degree of impairment of a patient's language function.

[0846] "Medical information" refers to various information related to medical treatment, such as patient charts and medical records.

[0847] "Input means" refers to a device or method for inputting data such as the patient's age, aphasia level, and medical information into the system.

[0848] "Receiving means" refers to a device or method by which the server receives patient information transmitted from the terminal.

[0849] The term "analysis means" refers to a method or device for analyzing received patient information and generating data necessary for formulating a rehabilitation plan.

[0850] "Rehabilitation plan" refers to a program that includes rehabilitation activities appropriate for a patient.

[0851] "Display means" refers to a device or method for visually showing the generated rehabilitation plan to the user.

[0852] "Instruction means" refers to a device or method for encouraging a user to take specific actions or practice based on a rehabilitation plan.

[0853] "Recording Means" refers to any device or method for capturing video and audio recordings of a patient's rehabilitation session.

[0854] "Emotional state" refers to a state that indicates the patient's psychological feelings.

[0855] "Emergency notification means" refers to a device or method for immediately reporting when the user's psychological state is judged to be dangerous.

[0856] "Daily report" refers to a report summarizing the results of daily rehabilitation.

[0857] "Delivery means" refers to a device or method for transmitting the generated daily report to a predetermined recipient.

[0858] "Real-time" refers to processing and response occurring immediately, without any time delay.

[0859] MODE FOR CARRYING OUT THE INVENTION

[0860] This invention is a technology for monitoring a patient's emotional state in real time in a rehabilitation system and providing an adaptive rehabilitation plan. At the same time, it also has a function to make an emergency call if the user's psychological state is judged to be dangerous. This system mainly consists of three elements: a server, a terminal, and the user.

[0861] System configuration and operation

[0862] 1. Collection and analysis of patient information

[0863] The server receives the patient's age, aphasia level, medical information, etc. sent from the terminal and analyzes this information. The user (medical staff) inputs the necessary data through the terminal and sends it to the server. This process generates basic data for understanding the patient's condition.

[0864] 2. Generating a rehabilitation plan

[0865] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user (patient).

[0866] 3. Emotion recognition and adaptive regulation by the emotion engine

[0867] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session, and sends this information to the emotion engine. The emotion engine analyzes this information and recognizes the user's emotional state. The server adaptively adjusts the rehabilitation plan based on the emotion recognition results.

[0868] 4. Rehabilitation implementation and progress monitoring

[0869] The user (patient) follows instructions on the device to perform rehabilitation. The rehabilitation session is recorded with video and audio, and the device sends this data to a server. The server analyzes the data and provides real-time feedback to the user via the device as needed.

[0870] 5. Emergency call

[0871] If the user's psychological state is judged to be dangerous, for example, if they show fear or anger, the device will immediately make an emergency call, which may include automatically notifying designated emergency contacts and contacting public authorities such as the police.

[0872] 6. Create and share daily reports

[0873] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family. The daily report includes rehabilitation progress, achievements, and next steps for improvement.

[0874] Specific examples

[0875] As a concrete example, consider a case where a user is undergoing rehabilitation using a smartphone application. During the rehabilitation session, the device's camera and microphone constantly monitor the user's facial expressions and voice. An emotion engine analyzes the user's emotions, and if the user expresses fear or anger, the device immediately initiates an emergency call. This call includes contacting designated emergency contacts and public authorities.

[0876] Prompt Sentence Examples

[0877] For example, you can use the following prompts for a generative AI model to achieve emotion recognition and emergency calling capabilities:

[0878] Implement an application that recognizes emotions from the user's facial and voice data, and makes an emergency call when danger is detected.

[0879] Facial expression data is captured in real time from the smartphone camera and analyzed using emotion recognition models.

[0880] · It also analyzes voice data to determine emotional state.

[0881] If the user expresses fear or anger, it will notify your emergency contacts.

[0882] The system allows for real-time analysis of a patient's emotional state, providing adaptive rehabilitation plans, as well as immediate response to dangerous situations.

[0883] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0884] Step 1:

[0885] The server receives the patient's age, aphasia level, and medical information sent from the device. Based on this input data, it prepares to analyze the patient's information. As a result of the analysis, basic data for understanding the patient's condition is generated.

[0886] Step 2:

[0887] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal so that the user (patient or medical staff) can review it.

[0888] Step 3:

[0889] The device uses a camera and microphone to record the user's facial expressions, voice tone, and word choice during the rehabilitation session. This information is then sent to an emotion engine to analyze the user's emotional state. The input to this process is video and audio data, and the output is emotion recognition results.

[0890] Step 4:

[0891] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.). The model performs data calculations based on the input video and audio data, and obtains the emotional state as output.

[0892] Step 5:

[0893] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results. For example, if the user is feeling stressed, it adds relaxation exercises. Based on the emotion recognition results as input, the server generates a revised and adjusted rehabilitation plan as output.

[0894] Step 6:

[0895] The user (patient) follows the instructions on the device to perform rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[0896] Step 7:

[0897] The device monitors the progress of rehabilitation in real time and transmits the video and audio data to the server. The server analyzes this data and provides real-time feedback to the user via the device as needed. The input is real-time video and audio data, and the output is feedback information.

[0898] Step 8:

[0899] The device continuously monitors the user's psychological state based on the data recorded during the rehabilitation session. If the user's psychological state is judged to be dangerous, the device immediately makes an emergency call. The input is the analysis result of the psychological state, and the output is the call action.

[0900] Step 9:

[0901] When the rehabilitation session ends, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family via email or a dedicated application. The input is the day's rehabilitation data, and the output is the daily report.

[0902] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0903] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0904] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0905] [Third embodiment]

[0906] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0907] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0908] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0909] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0910] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0911] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0912] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0913] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0914] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0915] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0916] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0917] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0918] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates a series of steps: collecting and analyzing patient information, creating rehabilitation plans, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, we will explain the program processing of this system in natural language and provide detailed examples.

[0919] System configuration and operation

[0920] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[0921] 1. Collecting patient information

[0922] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[0923] 2. Analysis of patient information

[0924] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[0925] 3. Generating a rehabilitation plan

[0926] The server automatically generates an individual rehabilitation plan based on the analysis results. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises. The generated rehabilitation plan is sent to the device and can be viewed by the user.

[0927] 4. Rehabilitation

[0928] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[0929] 5. Rehabilitation progress monitoring

[0930] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides real-time feedback to the user via the device as needed.

[0931] 6. Create and share daily reports

[0932] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[0933] Specific examples

[0934] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the instructions on the device, recording the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[0935] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[0936] The processing flow will be explained below.

[0937] Step 1: Enter patient information

[0938] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[0939] Step 2: Receiving and analyzing patient information

[0940] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[0941] Step 3: Generate a rehabilitation plan

[0942] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[0943] Step 4: Submit your rehabilitation plan

[0944] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[0945] Step 5: Rehabilitation

[0946] The user (patient) begins rehabilitation by following instructions on the device, which uses a camera and microphone to record the rehabilitation session as video and audio.

[0947] Step 6: Real-time data transmission

[0948] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[0949] Step 7: Real-time data analysis and feedback

[0950] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[0951] Step 8: Ending the rehabilitation session and sending data

[0952] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[0953] Step 9: Create a daily report

[0954] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[0955] Step 10: Share the report

[0956] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[0957] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[0958] Example 1

[0959] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0960] The present invention aims to maximize the effectiveness of rehabilitation for patients with aphasia or brain damage. Conventional rehabilitation methods have the problem that it takes a great deal of time and effort to create an optimal rehabilitation plan for each individual patient and provide feedback on the implementation status. Furthermore, effective rehabilitation is difficult due to limited means for monitoring the patient's progress in real time. It is desirable to solve these problems.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0962] In this invention, the server includes means for receiving basic patient information and storing it in a database, means for analyzing the received patient information and generating basic data for evaluating the patient's condition, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the generated rehabilitation plan and issuing instructions to the user, means for recording the patient's rehabilitation with a camera and microphone, means for receiving the recorded data, monitoring it in real time, and generating immediate feedback, and means for analyzing the recorded data, automatically generating daily reports, and distributing them to specified recipients. This makes it possible to quickly provide an optimized rehabilitation plan for each patient, monitor its implementation in real time, and provide immediate feedback.

[0963] "Basic patient information" refers to basic data necessary for the patient's medical rehabilitation, such as the patient's age, gender, aphasia level, and medical record information.

[0964] "Database" refers to a collection of data used by the Server to efficiently store, manage, and retrieve patient information.

[0965] "Baseline data" refers to basic evaluation data generated by analyzing received patient information to evaluate the patient's condition.

[0966] A "rehabilitation plan" refers to a detailed rehabilitation schedule that includes voice training, comprehension exercises, and communication exercises that are optimized for the patient's individual condition.

[0967] "Means for issuing instructions to the user" refers to the system operation and screen display mechanisms for appropriately notifying the user of the generated rehabilitation plan and issuing instructions.

[0968] "Camera and microphone" refers to the video and audio input devices used to record the patient's rehabilitation progress.

[0969] "Means for real-time monitoring" refers to a system that instantly monitors the progress of rehabilitation on the spot and analyzes the data.

[0970] "Immediate feedback" refers to assessments and instructions provided to patients quickly based on real-time monitored data.

[0971] "Daily report" refers to a detailed report summarizing the content and progress of each day's rehabilitation sessions.

[0972] The "predetermined recipient" refers to medical staff or family members of patients who are designated in advance as recipients of the generated daily report.

[0973] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates the collection and analysis of patient information, the creation of rehabilitation plans, the implementation of rehabilitation, progress monitoring, and the creation and sharing of daily reports, providing rehabilitation that is optimized for each individual patient.

[0974] System configuration and operation

[0975] This system mainly consists of three elements: a server, a terminal, and users (patients and medical staff).

[0976] Collecting patient information

[0977] The user (medical staff) enters basic patient information (age, gender, aphasia level, medical record information, etc.) into the terminal. The terminal formats the entered information and sends it to the server. The server stores the patient information received from the terminal in a database.

[0978] Patient information analysis

[0979] The server analyzes the patient information stored in the database and generates basic data to evaluate the patient's condition, which is then used to create an individual rehabilitation plan.

[0980] Generate rehabilitation plans

[0981] The server automatically generates an appropriate rehabilitation plan based on the generated basic data. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises, and the generated plan is sent to the device. The user (medical staff) can then check the rehabilitation plan through the device.

[0982] Rehabilitation

[0983] The user (patient) follows the rehabilitation instructions displayed on the device. The device displays the rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises. The device also records the rehabilitation progress as video and audio.

[0984] Rehabilitation progress monitoring

[0985] The device monitors the progress of rehabilitation in real time and transmits video and audio data to the server, which analyzes the received data and evaluates the rehabilitation progress. If necessary, the server generates real-time feedback and provides it to the user via the device.

[0986] Create and share daily reports

[0987] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server, which then automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated app.

[0988] Specific examples

[0989] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the device's instructions and records the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the doctor in charge and the patient's family.

[0990] By inputting the following prompt sentences into the generative AI model, the system automatically generates an optimal rehabilitation plan for the patient:

[0991] "A 70-year-old male patient suffers from moderate aphasia. Please create an optimal rehabilitation plan for this patient. The patient's basic information is as follows: Age: 70, Gender: Male, Current Status: Moderate aphasia."

[0992] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[0993] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0994] Processing Steps

[0995] Step 1:

[0996] The user (medical staff) inputs basic information about the patient (age, sex, aphasia level, medical record information, etc.) into the terminal.

[0997] Input: Patient's age, gender, aphasia level, medical record information

[0998] How it works: The user enters patient information into a dedicated application on the device and clicks the send button.

[0999] Output: Information entered into the terminal is formatted and sent to the server.

[1000] Step 2:

[1001] The terminal formats the entered patient information and transmits it to the server.

[1002] Input: Basic patient information entered by the user

[1003] Operation: The device converts the information into a specific format and sends it to the server.

[1004] Output: Formatted patient information

[1005] Step 3:

[1006] The server stores the patient information received from the terminal in a database.

[1007] Input: Formatted patient details

[1008] What happens: The server stores the information in a database

[1009] Output: Patient information stored in the database

[1010] Step 4:

[1011] The server analyzes the patient information stored in the database and generates basic data for evaluating the patient's condition.

[1012] Input: Patient information stored in the database

[1013] How it works: The server analyzes the information using natural language processing algorithms and generates the underlying data.

[1014] Output: The underlying data generated by the analysis

[1015] Step 5:

[1016] The server automatically generates an appropriate rehabilitation plan based on the generated basic data.

[1017] Input: Basic data

[1018] How it works: The server uses the generative AI model to create a rehabilitation plan.

[1019] Output: Automatically generated rehabilitation plan

[1020] Step 6:

[1021] The server transmits the generated rehabilitation plan to the terminal so that the user (medical staff) can check it.

[1022] Input: Rehabilitation Plan

[1023] Operation: The server sends the rehabilitation plan to the device.

[1024] Output: Rehabilitation plan sent to the terminal

[1025] Step 7:

[1026] The user (patient) follows the rehabilitation instructions displayed on the terminal and performs rehabilitation.

[1027] Input: Rehabilitation plan displayed on terminal

[1028] Action: The user uses the camera and microphone to perform rehabilitation.

[1029] Output: Data recording rehabilitation progress

[1030] Step 8:

[1031] The device displays rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises.

[1032] Input: Rehabilitation Plan

[1033] Operation: The device displays rehabilitation instructions to the user and records the rehabilitation progress with video and audio.

[1034] Output: Recorded video and audio data

[1035] Step 9:

[1036] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[1037] Input: Recorded video and audio data

[1038] Action: The device sends data to the server.

[1039] Output: Video and audio data sent to the server

[1040] Step 10:

[1041] The server analyzes the received data and evaluates the degree of rehabilitation achievement.

[1042] Input: Video and audio data

[1043] Action: The server analyzes the data and generates an evaluation result.

[1044] Output: Evaluation results regarding rehabilitation achievement

[1045] Step 11:

[1046] The server generates and provides immediate feedback to the user via the terminal as needed.

[1047] Input: Evaluation results regarding rehabilitation achievement

[1048] What it does: The server generates and sends immediate feedback to the user

[1049] Output: Immediate feedback provided to the user

[1050] Step 12:

[1051] When the rehabilitation session ends, the terminal transmits the rehabilitation data for that day to the server.

[1052] Input: Rehabilitation data for one day

[1053] Operation: The device collects data and sends it to the server

[1054] Output: Rehabilitation data for one day sent to the server

[1055] Step 13:

[1056] The server automatically generates daily reports based on the data received.

[1057] Input: Rehabilitation data for one day

[1058] How it works: The server analyzes the data and generates a daily report.

[1059] Output: Auto-generated daily reports

[1060] Step 14:

[1061] The server distributes the generated report to the doctor in charge and the patient's family.

[1062] Input: Daily Report

[1063] How it works: The server delivers reports via email or a dedicated application.

[1064] Output: Daily reports delivered to treating physicians and patient families

[1065] In this way, this system achieves efficient and sustainable rehabilitation by linking data input, calculation, and output at each step.

[1066] (Application example 1)

[1067] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1068] Current rehabilitation systems struggle to efficiently and continuously monitor and automatically generate personalized rehabilitation plans. Furthermore, they lack the ability to manage rehabilitation progress in real time and provide immediate feedback, making it difficult to achieve effective rehabilitation. Furthermore, there are no systems suitable for use in rehabilitation centers or brick-and-mortar stores, preventing patients from maximizing the benefits of their rehabilitation.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1070] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical record information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio data for recording the patient's rehabilitation, means for receiving and analyzing the recorded data, means for creating daily reports based on the analysis results and distributing them to specified recipients, and means for providing the rehabilitation plan to a terminal at the rehabilitation center and managing the progress. This enables automatic generation of an individualized rehabilitation plan and real-time monitoring of rehabilitation sessions, and real-time feedback to achieve effective rehabilitation.

[1071] "Patients" refers to those with aphasia, brain damage, etc. who require rehabilitation.

[1072] "Age" is the period indicated by the patient's biological number.

[1073] "Aphasia level" is an index showing the degree of aphasia.

[1074] "Medical record information" refers to medical data including a patient's medical history and treatment records.

[1075] "Means for receiving and analyzing information" refers to the processes and tools for obtaining and analyzing patient-entered information.

[1076] "Means for automatically generating rehabilitation plans" refers to technology that automatically creates individualized rehabilitation programs based on collected patient information.

[1077] "Means for providing instructions" refers to a method for presenting the generated rehabilitation plan to the patient or medical staff.

[1078] "Video and audio recording means" refers to devices and methods for recording rehabilitation sessions using cameras and microphones.

[1079] "Means for receiving and analyzing recorded data" refers to the processes and tools for receiving and analyzing recorded video and audio data.

[1080] "Means for creating and distributing daily reports" refers to a system and method for organizing rehabilitation progress information on a daily basis and distributing it to designated recipients (such as medical staff or family members).

[1081] "Rehabilitation center terminal" refers to computer equipment installed in a rehabilitation facility or clinic that is used to review and implement rehabilitation plans.

[1082] "Progress management means" refers to systems and methods that track a patient's rehabilitation progress and provide appropriate measures and feedback accordingly.

[1083] The system of this invention is designed to support effective rehabilitation for patients with aphasia or brain damage in physical rehabilitation centers or clinics. This system mainly consists of a server, terminals, and users (patients and medical staff). Specific embodiments of the present invention are described in detail below.

[1084] System configuration

[1085] 1. Collecting patient information

[1086] The server receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff) into the terminal. The terminal sends this information to the server, which then analyzes it.

[1087] 2. Analysis of patient information

[1088] The server analyzes the received patient information and generates individual basic data, which is used to prepare an optimal rehabilitation plan for each patient.

[1089] 3. Generating a rehabilitation plan

[1090] The server automatically generates a rehabilitation plan based on the analysis results. The plan is sent to the terminal and confirmed by the user (medical staff and patient). The rehabilitation plan includes vocal exercises, comprehension questions, and communication exercises.

[1091] 4. Rehabilitation

[1092] Patients undergo rehabilitation by following instructions on a device in the facility. The rehabilitation content is displayed on a screen, and patients use a camera and microphone connected to the device to perform voice training and exercises. The device records the rehabilitation session and sends the video and audio data to a server.

[1093] 5. Rehabilitation progress monitoring

[1094] The server monitors the progress of rehabilitation in real time and provides immediate feedback to the device as needed, helping to ensure smooth rehabilitation progress for the patient.

[1095] 6. Create and share daily reports

[1096] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server analyzes the received data and automatically generates a daily report. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated application.

[1097] Specific examples

[1098] For example, if a 70-year-old male patient suffers from moderate aphasia, medical staff enter the patient's basic information into the device and send it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results and sends it to the device. The patient performs rehabilitation according to the device's instructions, recording the process with a camera and microphone. The device then sends the recorded data to the server in real time, which analyzes the data and provides immediate feedback. After the rehabilitation session ends, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[1099] Hardware and software used

[1100] Smartphone: Equipped with a camera and microphone, it is used as a device to record the patient's rehabilitation progress.

[1101] Server: Analyzes patient information, generates rehabilitation plans, performs real-time monitoring, and creates daily reports.

[1102] Software: Python program, server communication via HTTP protocol.

[1103] Prompt Sentence Examples

[1104] "Please provide a specific use case for a smartphone application to help patients with their rehabilitation. Please explain in detail how it collects patient information, captures rehabilitation plans, records rehabilitation sessions, and generates and shares daily reports."

[1105] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1106] System program processing steps

[1107] Step 1:

[1108] The terminal receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff). This basic information is formatted by the terminal and sent to the server. The input data includes the patient's biometric number and medical records related to rehabilitation.

[1109] Step 2:

[1110] The server receives and analyzes the patient information sent from the device. Specifically, it uses each data item as material for identifying the disease and developing a rehabilitation plan to generate basic patient data based on the received data. This basic data includes information to clarify the patient's condition and rehabilitation goals.

[1111] Step 3:

[1112] The server automatically generates a rehabilitation plan based on the analysis results. The plan includes voice training, comprehension questions, and communication exercises tailored to each patient. The automatically generated plan is sent to the device and provided to medical staff and the patient.

[1113] Step 4:

[1114] The user (patient) performs rehabilitation by following instructions on the device. The device records each rehabilitation session with a camera and microphone and transmits the data in real time to a server. The recorded data includes rehabilitation progress and specific exercises.

[1115] Step 5:

[1116] The server analyzes the recorded rehabilitation data received in real time and generates feedback on the spot, returning it to the device. The feedback includes vocal errors and exercise improvements, allowing patients to receive immediate corrections as needed during rehabilitation.

[1117] Step 6:

[1118] After the rehabilitation session, the device sends the day's worth of rehabilitation data to the server, which then automatically generates a daily report based on the received data, detailing rehabilitation progress, achievements, and recommended next exercises.

[1119] Step 7:

[1120] Finally, the server delivers the generated daily report to the designated recipient (physician or family member) via email or a dedicated application. The delivered report serves as a reference for the recipient to check the patient's rehabilitation progress and plan future treatment plans.

[1121] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1122] This invention is a system that uses AI technology to support the rehabilitation of patients with aphasia or brain damage, and by combining it with an emotion engine that recognizes the user's emotions, it implements rehabilitation plans more adaptively and effectively. This system automates a series of steps: collecting and analyzing patient information, generating rehabilitation plans, recognizing emotions, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[1123] System configuration and operation

[1124] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[1125] 1. Collecting patient information

[1126] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[1127] 2. Analysis of patient information

[1128] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[1129] 3. Generating a rehabilitation plan

[1130] The server automatically generates an individual rehabilitation plan based on the analysis results. This rehabilitation plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user.

[1131] 4. Emotion Recognition with Emotion Engine

[1132] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session and transmits this information to the emotion engine, which analyzes this information and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.).

[1133] 5. Adaptive adjustment of rehabilitation plans

[1134] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it can add relaxation exercises.

[1135] 6. Rehabilitation

[1136] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[1137] 7. Rehabilitation progress monitoring

[1138] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides immediate feedback to the user via the device as needed.

[1139] 8. Create and share daily reports

[1140] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[1141] Specific examples

[1142] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic patient data. Next, based on the analysis results, the server automatically generates an individual rehabilitation plan including speech exercises and comprehension questions and sends it to the device.

[1143] The user (patient) performs rehabilitation according to instructions on the device, recording the process with a camera and microphone. The device then sends the recorded data to the emotion engine, which analyzes the user's facial expressions and vocal tone to recognize their emotional state. The server adaptively adjusts the rehabilitation content based on the recognized emotion results and provides it to the user. For example, if the user is tired, it can add relaxation exercises.

[1144] After the rehabilitation session, the server creates a daily report based on the data from the device and distributes it to the patient's doctor and family. In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, achieving efficient and sustainable rehabilitation.

[1145] The processing flow will be explained below.

[1146] Step 1: Enter patient information

[1147] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[1148] Step 2: Receiving and analyzing patient information

[1149] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[1150] Step 3: Generate a rehabilitation plan

[1151] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[1152] Step 4: Submit your rehabilitation plan

[1153] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[1154] Step 5: Emotion Recognition with the Emotion Engine

[1155] The device uses a camera and microphone to record the user's (patient's) facial expressions, voice tone, and word choice during the rehabilitation session. The device then sends this recorded data to the emotion engine, which analyzes the recorded data and recognizes the user's emotional state.

[1156] Step 6: Adaptively adjusting the rehabilitation plan

[1157] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it adds relaxation exercises.

[1158] Step 7: Rehabilitation

[1159] The user performs rehabilitation by following instructions from the device, which uses a camera and microphone to record video and audio of the rehabilitation session.

[1160] Step 8: Real-time data transmission

[1161] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[1162] Step 9: Real-time data analysis and feedback

[1163] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[1164] Step 10: Ending the rehabilitation session and sending data

[1165] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[1166] Step 11: Create a daily report

[1167] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[1168] Step 12: Share the report

[1169] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[1170] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[1171] Example 2

[1172] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1173] Currently, rehabilitation for patients with aphasia and brain damage requires flexible rehabilitation plans tailored to each individual, as the effectiveness of these plans varies from patient to patient. However, existing rehabilitation systems lack the ability to appropriately grasp a patient's emotional state and dynamically adjust the rehabilitation plan. They also lack the ability to monitor the progress of rehabilitation in real time and provide immediate feedback. Therefore, there is a need for a system that can generate rehabilitation plans tailored to each patient's individual needs and recognize and adapt to their emotional state during the implementation process.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input the patient's age, disability level, and medical information; a means for receiving and analyzing the patient information; a means for automatically generating a rehabilitation plan based on the analysis results; a means for collecting data on the patient's facial expressions, tone of voice, and word choice during the rehabilitation session to recognize the patient's emotional state; a means for adaptively adjusting the rehabilitation plan based on the recognized emotional state; a means for receiving and analyzing the recorded data; a means for creating a daily report based on the analysis results and distributing it to a predetermined recipient; and a means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback. This enables the creation and provision of an optimal rehabilitation plan tailored to the individual patient's condition. Furthermore, by accurately understanding the patient's emotional state during the rehabilitation process and making adaptive adjustments, the effectiveness of the rehabilitation can be maximized.

[1175] "User" is the medical professional or person who operates the system, inputs patient data, and reviews and adjusts rehabilitation plans.

[1176] "Patients" refer to people who need rehabilitation for conditions such as aphasia or brain damage.

[1177] A "server" is a computer system that analyzes patient information, generates rehabilitation plans, and stores and processes data.

[1178] A "terminal" is an electronic device such as a computer, tablet, or smartphone that is directly operated by a user or patient.

[1179] "Patient information" refers to basic data necessary for generating and adjusting a rehabilitation plan, such as age, disability level, and medical information.

[1180] A "rehabilitation plan" is a plan that includes specific rehabilitation content and steps and is automatically generated based on the patient's condition.

[1181] "Emotion engine" refers to algorithms and machine learning models that analyze a patient's facial expressions, tone of voice, and word choice data to recognize their emotional state.

[1182] The "Daily Report" is a report that summarizes the daily activity record, including the progress, achievements, and areas for improvement of the rehabilitation session.

[1183] "Real-time monitoring" is the process of observing rehabilitation progress in real time and collecting and analyzing data.

[1184] "Immediate feedback" refers to advice or instructions provided immediately to a user or patient based on the results of data analysis.

[1185] "Adaptive adjustment" is the process of dynamically changing or adjusting a rehabilitation plan based on the patient's emotional state and progress.

[1186] This invention provides a system for supporting rehabilitation for patients with aphasia or brain damage, recognizing the patient's emotional state and adaptively adjusting the rehabilitation plan based on that state. The system automates the collection and analysis of patient information, generation of rehabilitation plans, emotion recognition, rehabilitation implementation, progress monitoring, and creation and sharing of daily reports.

[1187] The main elements of this system are the server, terminals, and users (patients and medical staff). Below we will explain in detail how each element works together.

[1188] Hardware and Software Used

[1189] Hardware

[1190] 1. Devices: Personal computers, tablets, smartphones, etc. used by users and patients.

[1191] 2. Camera and microphone: Devices for collecting the patient's facial expressions and voice.

[1192] 3. Server: A high-performance computer for analyzing data and generating rehabilitation plans.

[1193] software

[1194] 1. Emotion engine: An AI model to analyze patient emotions (e.g., Google Cloud's Natural Language API or Azure Emotion API).

[1195] 2. Analysis algorithm: Software for analyzing patient information and generating rehabilitation plans.

[1196] 3. Database system: A system for storing patient information and rehabilitation progress data (e.g., MySQL, PostgreSQL).

[1197] 4. Rehabilitation support app: A dedicated application to help patients carry out rehabilitation.

[1198] System example and operation

[1199] Collecting patient information

[1200] The user (medical staff) uses a terminal to input basic information such as the patient's age, disability level, medical information, etc. The terminal then transmits the input information to the server.

[1201] Patient information analysis

[1202] The server receives the patient information sent from the terminal, cleanses the information using an analysis program, and generates basic data.

[1203] Generate rehabilitation plans

[1204] The server automatically generates a rehabilitation plan based on the generated basic data. This rehabilitation plan includes vocal training, comprehension questions, and communication exercises. A prompt sentence is input into the generative AI model to generate the rehabilitation content. An example of a prompt sentence is, "Please generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[1205] Emotion recognition with emotion engine

[1206] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session, and the recorded data is sent to the emotion engine on the server, which analyzes it and recognizes the patient's emotional state.

[1207] Adaptive adjustment of rehabilitation plans

[1208] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine, for example adding relaxation exercises if the patient is feeling stressed.

[1209] Rehabilitation

[1210] The user (patient) follows instructions from the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation process as video and audio.

[1211] Rehabilitation progress monitoring

[1212] The device transmits real-time information about the rehabilitation progress to a server, which analyzes the data and provides immediate feedback to the user as needed.

[1213] Create and share daily reports

[1214] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The report is then sent to the patient's doctor and family via email or a dedicated application.

[1215] As described above, the present invention is a system that provides rehabilitation plans tailored to the individual patient's condition, and enables real-time monitoring and immediate feedback, thereby achieving efficient and effective rehabilitation.

[1216] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1217] Step 1: Enter and submit patient information

[1218] The user (medical staff) inputs basic information such as the patient's age, disability level, and medical information into the terminal.

[1219] Input: Patient's age, disability level, medical information

[1220] Action: Fill out the form and click the "Submit" button.

[1221] Output: The entered patient information is sent from the terminal to the server.

[1222] Step 2: Receiving and analyzing patient information

[1223] The server receives the patient information transmitted from the terminal.

[1224] Input: Patient information sent from the terminal

[1225] Operation: Receive data via the receiving API.

[1226] Output: Pass the received patient information to the analysis program.

[1227] Step 3: Data cleansing and basic data generation

[1228] The server analyzes the received information and cleanses the data to generate the base data.

[1229] Input: Received patient information

[1230] How it works: The analysis program cleanses the data, extracting and organizing only the necessary items.

[1231] Output: Cleansed underlying data

[1232] Step 4: Generate a rehabilitation plan

[1233] The server automatically generates a rehabilitation plan based on the basic data.

[1234] Input: Cleansed underlying data

[1235] How it works: A prompt sentence is input into the generative AI model, which generates rehabilitation content (vocal training, comprehension questions, and communication exercises).

[1236] Example prompt: "Generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[1237] Output: Automatically generated rehabilitation plan

[1238] Step 5: Submit and view your rehabilitation plan

[1239] The server transmits the generated rehabilitation plan to the terminal, and the terminal displays the rehabilitation plan to the user.

[1240] Input: Auto-generated rehabilitation plan

[1241] Operation: The server sends planning data to the terminal, which receives the data and displays it on the screen.

[1242] Output: User-viewable rehabilitation plan display

[1243] Step 6: Recording your rehabilitation sessions

[1244] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session.

[1245] Input: Patient behavior and voice during rehabilitation session

[1246] How it works: Cameras and microphones collect and record data in real time.

[1247] Output: Recorded video and audio data

[1248] Step 7: Sending Emotion Data and Emotion Recognition

[1249] The terminal sends the recorded data to the emotion engine, and the server recognizes the emotional state.

[1250] Input: Recorded video and audio data

[1251] How it works: Data is sent to the server's emotion engine API, where it is analyzed by emotion recognition algorithms.

[1252] Output: Recognized emotional state data

[1253] Step 8: Adaptively adjusting the rehabilitation plan

[1254] The server adaptively adjusts the rehabilitation plan based on the results of the emotion engine.

[1255] Input: Recognized emotional state data

[1256] Action: Re-analyze the rehabilitation plan and add relaxation exercises and content as needed.

[1257] Output: Coordinated rehabilitation plan

[1258] Step 9: Implement and record rehabilitation

[1259] The user (patient) performs rehabilitation by following the instructions on the terminal, and the terminal records the progress.

[1260] Input: Coordinated rehabilitation plan

[1261] How it works: Follow the instructions and use the camera and microphone to record your vocal exercises and workouts in real time.

[1262] Output: Newly recorded video and audio of the rehabilitation session

[1263] Step 10: Real-time analysis of rehabilitation progress and provision of feedback

[1264] The device monitors the progress of rehabilitation in real time, and the server provides immediate feedback based on the analysis results.

[1265] Input: Real-time rehabilitation progress data

[1266] What it does: Sends data to a server that analyzes progress and forwards instant advice and encouragement to the device.

[1267] Output: Immediate feedback provided to the user

[1268] Step 11: Create and share daily reports

[1269] The terminal transmits the rehabilitation data for one day to the server, which automatically generates a daily report.

[1270] Input: One day of rehabilitation data

[1271] What it does: Aggregate data and generate reports that show progress, achievements, and areas for improvement.

[1272] Output: Auto-generated daily report

[1273] Step 12: Daily Report Delivery

[1274] The server distributes the generated daily reports to the patient's doctor and family.

[1275] Input: Auto-generated daily report

[1276] What it does: Reports are delivered via email or a dedicated application.

[1277] Output: Daily reports sent to your doctor and family

[1278] (Application example 2)

[1279] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1280] Patients undergoing rehabilitation often experience significant mental stress and anxiety. Conventional rehabilitation systems struggle to recognize a patient's emotional and psychological state in real time and provide appropriate support. This can result in insufficient rehabilitation benefits and further problems. While there is a need to understand a patient's mental crisis in real time and respond immediately in emergencies, current technology is unable to adequately address this need.

[1281] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1282] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio for recording the patient's rehabilitation, means for receiving the recorded data and analyzing the emotional state, means for creating a daily report based on the analysis results and distributing it to a predetermined recipient, and means for making an emergency call if the user's psychological state is judged to be dangerous. This makes it possible to analyze the patient's emotional state in real time and make adaptive adjustments to the rehabilitation plan, as well as to make an emergency call immediately in the event of an emergency.

[1283] "Patient" refers to a person with a health condition requiring rehabilitation.

[1284] "Age" is information that indicates the period of time that has elapsed since the patient's date of birth until the present.

[1285] "Aphasia level" refers to a scale or information that indicates the degree of impairment of a patient's language function.

[1286] "Medical information" refers to various information related to medical treatment, such as patient charts and medical records.

[1287] "Input means" refers to a device or method for inputting data such as the patient's age, aphasia level, and medical information into the system.

[1288] "Receiving means" refers to a device or method by which the server receives patient information transmitted from the terminal.

[1289] The term "analysis means" refers to a method or device for analyzing received patient information and generating data necessary for formulating a rehabilitation plan.

[1290] "Rehabilitation plan" refers to a program that includes rehabilitation activities appropriate for a patient.

[1291] "Display means" refers to a device or method for visually showing the generated rehabilitation plan to the user.

[1292] "Instruction means" refers to a device or method for encouraging a user to take specific actions or practice based on a rehabilitation plan.

[1293] "Recording Means" refers to any device or method for capturing video and audio recordings of a patient's rehabilitation session.

[1294] "Emotional state" refers to a state that indicates the patient's psychological feelings.

[1295] "Emergency notification means" refers to a device or method for immediately reporting when the user's psychological state is judged to be dangerous.

[1296] "Daily report" refers to a report summarizing the results of daily rehabilitation.

[1297] "Delivery means" refers to a device or method for transmitting the generated daily report to a predetermined recipient.

[1298] "Real-time" refers to processing and response occurring immediately, without any time delay.

[1299] MODE FOR CARRYING OUT THE INVENTION

[1300] This invention is a technology for monitoring a patient's emotional state in real time in a rehabilitation system and providing an adaptive rehabilitation plan. At the same time, it also has a function to make an emergency call if the user's psychological state is judged to be dangerous. This system mainly consists of three elements: a server, a terminal, and the user.

[1301] System configuration and operation

[1302] 1. Collection and analysis of patient information

[1303] The server receives the patient's age, aphasia level, medical information, etc. sent from the terminal and analyzes this information. The user (medical staff) inputs the necessary data through the terminal and sends it to the server. This process generates basic data for understanding the patient's condition.

[1304] 2. Generating a rehabilitation plan

[1305] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user (patient).

[1306] 3. Emotion recognition and adaptive regulation by the emotion engine

[1307] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session, and sends this information to the emotion engine. The emotion engine analyzes this information and recognizes the user's emotional state. The server adaptively adjusts the rehabilitation plan based on the emotion recognition results.

[1308] 4. Rehabilitation implementation and progress monitoring

[1309] The user (patient) follows instructions on the device to perform rehabilitation. The rehabilitation session is recorded with video and audio, and the device sends this data to a server. The server analyzes the data and provides real-time feedback to the user via the device as needed.

[1310] 5. Emergency call

[1311] If the user's psychological state is judged to be dangerous, for example, if they show fear or anger, the device will immediately make an emergency call, which may include automatically notifying designated emergency contacts and contacting public authorities such as the police.

[1312] 6. Create and share daily reports

[1313] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family. The daily report includes rehabilitation progress, achievements, and next steps for improvement.

[1314] Specific examples

[1315] As a concrete example, consider a case where a user is undergoing rehabilitation using a smartphone application. During the rehabilitation session, the device's camera and microphone constantly monitor the user's facial expressions and voice. An emotion engine analyzes the user's emotions, and if the user expresses fear or anger, the device immediately initiates an emergency call. This call includes contacting designated emergency contacts and public authorities.

[1316] Prompt Sentence Examples

[1317] For example, you can use the following prompts for a generative AI model to achieve emotion recognition and emergency calling capabilities:

[1318] Implement an application that recognizes emotions from the user's facial and voice data, and makes an emergency call when danger is detected.

[1319] Facial expression data is captured in real time from the smartphone camera and analyzed using emotion recognition models.

[1320] · It also analyzes voice data to determine emotional state.

[1321] If the user expresses fear or anger, it will notify your emergency contacts.

[1322] The system allows for real-time analysis of a patient's emotional state, providing adaptive rehabilitation plans, as well as immediate response to dangerous situations.

[1323] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1324] Step 1:

[1325] The server receives the patient's age, aphasia level, and medical information sent from the device. Based on this input data, it prepares to analyze the patient's information. As a result of the analysis, basic data for understanding the patient's condition is generated.

[1326] Step 2:

[1327] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal so that the user (patient or medical staff) can review it.

[1328] Step 3:

[1329] The device uses a camera and microphone to record the user's facial expressions, voice tone, and word choice during the rehabilitation session. This information is then sent to an emotion engine to analyze the user's emotional state. The input to this process is video and audio data, and the output is emotion recognition results.

[1330] Step 4:

[1331] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.). The model performs data calculations based on the input video and audio data, and obtains the emotional state as output.

[1332] Step 5:

[1333] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results. For example, if the user is feeling stressed, it adds relaxation exercises. Based on the emotion recognition results as input, the server generates a revised and adjusted rehabilitation plan as output.

[1334] Step 6:

[1335] The user (patient) follows the instructions on the device to perform rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[1336] Step 7:

[1337] The device monitors the progress of rehabilitation in real time and transmits the video and audio data to the server. The server analyzes this data and provides real-time feedback to the user via the device as needed. The input is real-time video and audio data, and the output is feedback information.

[1338] Step 8:

[1339] The device continuously monitors the user's psychological state based on the data recorded during the rehabilitation session. If the user's psychological state is judged to be dangerous, the device immediately makes an emergency call. The input is the analysis result of the psychological state, and the output is the call action.

[1340] Step 9:

[1341] When the rehabilitation session ends, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family via email or a dedicated application. The input is the day's rehabilitation data, and the output is the daily report.

[1342] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1343] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1344] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1345] [Fourth embodiment]

[1346] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1347] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1348] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1349] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1350] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1351] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1352] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1353] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1354] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1355] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1356] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1357] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1358] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1359] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates a series of steps: collecting and analyzing patient information, creating rehabilitation plans, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, we will explain the program processing of this system in natural language and provide detailed examples.

[1360] System configuration and operation

[1361] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[1362] 1. Collecting patient information

[1363] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[1364] 2. Analysis of patient information

[1365] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[1366] 3. Generating a rehabilitation plan

[1367] The server automatically generates an individual rehabilitation plan based on the analysis results. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises. The generated rehabilitation plan is sent to the device and can be viewed by the user.

[1368] 4. Rehabilitation

[1369] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[1370] 5. Rehabilitation progress monitoring

[1371] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides real-time feedback to the user via the device as needed.

[1372] 6. Create and share daily reports

[1373] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[1374] Specific examples

[1375] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the instructions on the device, recording the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[1376] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[1377] The processing flow will be explained below.

[1378] Step 1: Enter patient information

[1379] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[1380] Step 2: Receiving and analyzing patient information

[1381] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[1382] Step 3: Generate a rehabilitation plan

[1383] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[1384] Step 4: Submit your rehabilitation plan

[1385] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[1386] Step 5: Rehabilitation

[1387] The user (patient) begins rehabilitation by following instructions on the device, which uses a camera and microphone to record the rehabilitation session as video and audio.

[1388] Step 6: Real-time data transmission

[1389] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[1390] Step 7: Real-time data analysis and feedback

[1391] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[1392] Step 8: Ending the rehabilitation session and sending data

[1393] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[1394] Step 9: Create a daily report

[1395] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[1396] Step 10: Share the report

[1397] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[1398] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[1399] Example 1

[1400] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1401] The present invention aims to maximize the effectiveness of rehabilitation for patients with aphasia or brain damage. Conventional rehabilitation methods have the problem that it takes a great deal of time and effort to create an optimal rehabilitation plan for each individual patient and provide feedback on the implementation status. Furthermore, effective rehabilitation is difficult due to limited means for monitoring the patient's progress in real time. It is desirable to solve these problems.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1403] In this invention, the server includes means for receiving basic patient information and storing it in a database, means for analyzing the received patient information and generating basic data for evaluating the patient's condition, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the generated rehabilitation plan and issuing instructions to the user, means for recording the patient's rehabilitation with a camera and microphone, means for receiving the recorded data, monitoring it in real time, and generating immediate feedback, and means for analyzing the recorded data, automatically generating daily reports, and distributing them to specified recipients. This makes it possible to quickly provide an optimized rehabilitation plan for each patient, monitor its implementation in real time, and provide immediate feedback.

[1404] "Basic patient information" refers to basic data necessary for the patient's medical rehabilitation, such as the patient's age, gender, aphasia level, and medical record information.

[1405] "Database" refers to a collection of data used by the Server to efficiently store, manage, and retrieve patient information.

[1406] "Baseline data" refers to basic evaluation data generated by analyzing received patient information to evaluate the patient's condition.

[1407] A "rehabilitation plan" refers to a detailed rehabilitation schedule that includes voice training, comprehension exercises, and communication exercises that are optimized for the patient's individual condition.

[1408] "Means for issuing instructions to the user" refers to the system operation and screen display mechanisms for appropriately notifying the user of the generated rehabilitation plan and issuing instructions.

[1409] "Camera and microphone" refers to the video and audio input devices used to record the patient's rehabilitation progress.

[1410] "Means for real-time monitoring" refers to a system that instantly monitors the progress of rehabilitation on the spot and analyzes the data.

[1411] "Immediate feedback" refers to assessments and instructions provided to patients quickly based on real-time monitored data.

[1412] "Daily report" refers to a detailed report summarizing the content and progress of each day's rehabilitation sessions.

[1413] The "predetermined recipient" refers to medical staff or family members of patients who are designated in advance as recipients of the generated daily report.

[1414] This invention is a system that uses AI technology to support rehabilitation for patients with aphasia and brain damage. This system automates the collection and analysis of patient information, the creation of rehabilitation plans, the implementation of rehabilitation, progress monitoring, and the creation and sharing of daily reports, providing rehabilitation that is optimized for each individual patient.

[1415] System configuration and operation

[1416] This system mainly consists of three elements: a server, a terminal, and users (patients and medical staff).

[1417] Collecting patient information

[1418] The user (medical staff) enters basic patient information (age, gender, aphasia level, medical record information, etc.) into the terminal. The terminal formats the entered information and sends it to the server. The server stores the patient information received from the terminal in a database.

[1419] Patient information analysis

[1420] The server analyzes the patient information stored in the database and generates basic data to evaluate the patient's condition, which is then used to create an individual rehabilitation plan.

[1421] Generate rehabilitation plans

[1422] The server automatically generates an appropriate rehabilitation plan based on the generated basic data. The rehabilitation plan includes vocal training, comprehension questions, and communication exercises, and the generated plan is sent to the device. The user (medical staff) can then check the rehabilitation plan through the device.

[1423] Rehabilitation

[1424] The user (patient) follows the rehabilitation instructions displayed on the device. The device displays the rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises. The device also records the rehabilitation progress as video and audio.

[1425] Rehabilitation progress monitoring

[1426] The device monitors the progress of rehabilitation in real time and transmits video and audio data to the server, which analyzes the received data and evaluates the rehabilitation progress. If necessary, the server generates real-time feedback and provides it to the user via the device.

[1427] Create and share daily reports

[1428] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server, which then automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated app.

[1429] Specific examples

[1430] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results, including speech exercises and comprehension questions, and sends it to the device. The user (patient) performs rehabilitation according to the device's instructions and records the process with a camera and microphone. The device sends the recorded data to the server in real time, and the server analyzes the data and provides immediate feedback. When the rehabilitation session is over, the server creates a daily report based on the day's data and distributes it to the doctor in charge and the patient's family.

[1431] By inputting the following prompt sentences into the generative AI model, the system automatically generates an optimal rehabilitation plan for the patient:

[1432] "A 70-year-old male patient suffers from moderate aphasia. Please create an optimal rehabilitation plan for this patient. The patient's basic information is as follows: Age: 70, Gender: Male, Current Status: Moderate aphasia."

[1433] In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, enabling efficient and sustainable rehabilitation.

[1434] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1435] Processing Steps

[1436] Step 1:

[1437] The user (medical staff) inputs basic information about the patient (age, sex, aphasia level, medical record information, etc.) into the terminal.

[1438] Input: Patient's age, gender, aphasia level, medical record information

[1439] How it works: The user enters patient information into a dedicated application on the device and clicks the send button.

[1440] Output: Information entered into the terminal is formatted and sent to the server.

[1441] Step 2:

[1442] The terminal formats the entered patient information and transmits it to the server.

[1443] Input: Basic patient information entered by the user

[1444] Operation: The device converts the information into a specific format and sends it to the server.

[1445] Output: Formatted patient information

[1446] Step 3:

[1447] The server stores the patient information received from the terminal in a database.

[1448] Input: Formatted patient details

[1449] What happens: The server stores the information in a database

[1450] Output: Patient information stored in the database

[1451] Step 4:

[1452] The server analyzes the patient information stored in the database and generates basic data for evaluating the patient's condition.

[1453] Input: Patient information stored in the database

[1454] How it works: The server analyzes the information using natural language processing algorithms and generates the underlying data.

[1455] Output: The underlying data generated by the analysis

[1456] Step 5:

[1457] The server automatically generates an appropriate rehabilitation plan based on the generated basic data.

[1458] Input: Basic data

[1459] How it works: The server uses the generative AI model to create a rehabilitation plan.

[1460] Output: Automatically generated rehabilitation plan

[1461] Step 6:

[1462] The server transmits the generated rehabilitation plan to the terminal so that the user (medical staff) can check it.

[1463] Input: Rehabilitation Plan

[1464] Operation: The server sends the rehabilitation plan to the device.

[1465] Output: Rehabilitation plan sent to the terminal

[1466] Step 7:

[1467] The user (patient) follows the rehabilitation instructions displayed on the terminal and performs rehabilitation.

[1468] Input: Rehabilitation plan displayed on terminal

[1469] Action: The user uses the camera and microphone to perform rehabilitation.

[1470] Output: Data recording rehabilitation progress

[1471] Step 8:

[1472] The device displays rehabilitation content on the screen and uses a camera and microphone to perform vocal training and exercises.

[1473] Input: Rehabilitation Plan

[1474] Operation: The device displays rehabilitation instructions to the user and records the rehabilitation progress with video and audio.

[1475] Output: Recorded video and audio data

[1476] Step 9:

[1477] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[1478] Input: Recorded video and audio data

[1479] Action: The device sends data to the server.

[1480] Output: Video and audio data sent to the server

[1481] Step 10:

[1482] The server analyzes the received data and evaluates the degree of rehabilitation achievement.

[1483] Input: Video and audio data

[1484] Action: The server analyzes the data and generates an evaluation result.

[1485] Output: Evaluation results regarding rehabilitation achievement

[1486] Step 11:

[1487] The server generates and provides immediate feedback to the user via the terminal as needed.

[1488] Input: Evaluation results regarding rehabilitation achievement

[1489] What it does: The server generates and sends immediate feedback to the user

[1490] Output: Immediate feedback provided to the user

[1491] Step 12:

[1492] When the rehabilitation session ends, the terminal transmits the rehabilitation data for that day to the server.

[1493] Input: Rehabilitation data for one day

[1494] Operation: The device collects data and sends it to the server

[1495] Output: Rehabilitation data for one day sent to the server

[1496] Step 13:

[1497] The server automatically generates daily reports based on the data received.

[1498] Input: Rehabilitation data for one day

[1499] How it works: The server analyzes the data and generates a daily report.

[1500] Output: Auto-generated daily reports

[1501] Step 14:

[1502] The server distributes the generated report to the doctor in charge and the patient's family.

[1503] Input: Daily Report

[1504] How it works: The server delivers reports via email or a dedicated application.

[1505] Output: Daily reports delivered to treating physicians and patient families

[1506] In this way, this system achieves efficient and sustainable rehabilitation by linking data input, calculation, and output at each step.

[1507] (Application example 1)

[1508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] Current rehabilitation systems struggle to efficiently and continuously monitor and automatically generate personalized rehabilitation plans. Furthermore, they lack the ability to manage rehabilitation progress in real time and provide immediate feedback, making it difficult to achieve effective rehabilitation. Furthermore, there are no systems suitable for use in rehabilitation centers or brick-and-mortar stores, preventing patients from maximizing the benefits of their rehabilitation.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1511] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical record information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio data for recording the patient's rehabilitation, means for receiving and analyzing the recorded data, means for creating daily reports based on the analysis results and distributing them to specified recipients, and means for providing the rehabilitation plan to a terminal at the rehabilitation center and managing the progress. This enables automatic generation of an individualized rehabilitation plan and real-time monitoring of rehabilitation sessions, and real-time feedback to achieve effective rehabilitation.

[1512] "Patients" refers to those with aphasia, brain damage, etc. who require rehabilitation.

[1513] "Age" is the period indicated by the patient's biological number.

[1514] "Aphasia level" is an index showing the degree of aphasia.

[1515] "Medical record information" refers to medical data including a patient's medical history and treatment records.

[1516] "Means for receiving and analyzing information" refers to the processes and tools for obtaining and analyzing patient-entered information.

[1517] "Means for automatically generating rehabilitation plans" refers to technology that automatically creates individualized rehabilitation programs based on collected patient information.

[1518] "Means for providing instructions" refers to a method for presenting the generated rehabilitation plan to the patient or medical staff.

[1519] "Video and audio recording means" refers to devices and methods for recording rehabilitation sessions using cameras and microphones.

[1520] "Means for receiving and analyzing recorded data" refers to the processes and tools for receiving and analyzing recorded video and audio data.

[1521] "Means for creating and distributing daily reports" refers to a system and method for organizing rehabilitation progress information on a daily basis and distributing it to designated recipients (such as medical staff or family members).

[1522] "Rehabilitation center terminal" refers to computer equipment installed in a rehabilitation facility or clinic that is used to review and implement rehabilitation plans.

[1523] "Progress management means" refers to systems and methods that track a patient's rehabilitation progress and provide appropriate measures and feedback accordingly.

[1524] The system of this invention is designed to support effective rehabilitation for patients with aphasia or brain damage in physical rehabilitation centers or clinics. This system mainly consists of a server, terminals, and users (patients and medical staff). Specific embodiments of the present invention are described in detail below.

[1525] System configuration

[1526] 1. Collecting patient information

[1527] The server receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff) into the terminal. The terminal sends this information to the server, which then analyzes it.

[1528] 2. Analysis of patient information

[1529] The server analyzes the received patient information and generates individual basic data, which is used to prepare an optimal rehabilitation plan for each patient.

[1530] 3. Generating a rehabilitation plan

[1531] The server automatically generates a rehabilitation plan based on the analysis results. The plan is sent to the terminal and confirmed by the user (medical staff and patient). The rehabilitation plan includes vocal exercises, comprehension questions, and communication exercises.

[1532] 4. Rehabilitation

[1533] Patients undergo rehabilitation by following instructions on a device in the facility. The rehabilitation content is displayed on a screen, and patients use a camera and microphone connected to the device to perform voice training and exercises. The device records the rehabilitation session and sends the video and audio data to a server.

[1534] 5. Rehabilitation progress monitoring

[1535] The server monitors the progress of rehabilitation in real time and provides immediate feedback to the device as needed, helping to ensure smooth rehabilitation progress for the patient.

[1536] 6. Create and share daily reports

[1537] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server analyzes the received data and automatically generates a daily report. This report details the rehabilitation progress, achievements, and next steps for improvement. The daily report is then sent to the patient's doctor and family via email or a dedicated application.

[1538] Specific examples

[1539] For example, if a 70-year-old male patient suffers from moderate aphasia, medical staff enter the patient's basic information into the device and send it to the server. The server analyzes the received information and generates basic data on the patient. The server then automatically generates an individual rehabilitation plan based on the analysis results and sends it to the device. The patient performs rehabilitation according to the device's instructions, recording the process with a camera and microphone. The device then sends the recorded data to the server in real time, which analyzes the data and provides immediate feedback. After the rehabilitation session ends, the server creates a daily report based on the day's data and distributes it to the patient's doctor and family.

[1540] Hardware and software used

[1541] Smartphone: Equipped with a camera and microphone, it is used as a device to record the patient's rehabilitation progress.

[1542] Server: Analyzes patient information, generates rehabilitation plans, performs real-time monitoring, and creates daily reports.

[1543] Software: Python program, server communication via HTTP protocol.

[1544] Prompt Sentence Examples

[1545] "Please provide a specific use case for a smartphone application to help patients with their rehabilitation. Please explain in detail how it collects patient information, captures rehabilitation plans, records rehabilitation sessions, and generates and shares daily reports."

[1546] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1547] System program processing steps

[1548] Step 1:

[1549] The terminal receives information such as the patient's age, aphasia level, and medical record information entered by the user (medical staff). This basic information is formatted by the terminal and sent to the server. The input data includes the patient's biometric number and medical records related to rehabilitation.

[1550] Step 2:

[1551] The server receives and analyzes the patient information sent from the device. Specifically, it uses each data item as material for identifying the disease and developing a rehabilitation plan to generate basic patient data based on the received data. This basic data includes information to clarify the patient's condition and rehabilitation goals.

[1552] Step 3:

[1553] The server automatically generates a rehabilitation plan based on the analysis results. The plan includes voice training, comprehension questions, and communication exercises tailored to each patient. The automatically generated plan is sent to the device and provided to medical staff and the patient.

[1554] Step 4:

[1555] The user (patient) performs rehabilitation by following instructions on the device. The device records each rehabilitation session with a camera and microphone and transmits the data in real time to a server. The recorded data includes rehabilitation progress and specific exercises.

[1556] Step 5:

[1557] The server analyzes the recorded rehabilitation data received in real time and generates feedback on the spot, returning it to the device. The feedback includes vocal errors and exercise improvements, allowing patients to receive immediate corrections as needed during rehabilitation.

[1558] Step 6:

[1559] After the rehabilitation session, the device sends the day's worth of rehabilitation data to the server, which then automatically generates a daily report based on the received data, detailing rehabilitation progress, achievements, and recommended next exercises.

[1560] Step 7:

[1561] Finally, the server delivers the generated daily report to the designated recipient (physician or family member) via email or a dedicated application. The delivered report serves as a reference for the recipient to check the patient's rehabilitation progress and plan future treatment plans.

[1562] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1563] This invention is a system that uses AI technology to support the rehabilitation of patients with aphasia or brain damage, and by combining it with an emotion engine that recognizes the user's emotions, it implements rehabilitation plans more adaptively and effectively. This system automates a series of steps: collecting and analyzing patient information, generating rehabilitation plans, recognizing emotions, implementing rehabilitation, monitoring progress, and creating and sharing daily reports. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[1564] System configuration and operation

[1565] This system mainly consists of three elements: a server, a terminal, and a user (patient).

[1566] 1. Collecting patient information

[1567] The server receives basic patient information (age, gender, aphasia level, medical record information, etc.) sent from the terminal. The user (medical staff) inputs the necessary information into the terminal and sends it to the server, whereupon patient information is collected.

[1568] 2. Analysis of patient information

[1569] The server analyzes the received information and generates basic data to understand the patient's condition, which prepares the basis for creating an individual rehabilitation plan.

[1570] 3. Generating a rehabilitation plan

[1571] The server automatically generates an individual rehabilitation plan based on the analysis results. This rehabilitation plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user.

[1572] 4. Emotion Recognition with Emotion Engine

[1573] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session and transmits this information to the emotion engine, which analyzes this information and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.).

[1574] 5. Adaptive adjustment of rehabilitation plans

[1575] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it can add relaxation exercises.

[1576] 6. Rehabilitation

[1577] The user (patient) follows the instructions on the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[1578] 7. Rehabilitation progress monitoring

[1579] The device monitors the progress of rehabilitation in real time and transmits video and audio data to a server, which analyzes the data and provides immediate feedback to the user via the device as needed.

[1580] 8. Create and share daily reports

[1581] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. The daily report details the rehabilitation progress, achievements, and next steps for improvement. This report is then sent to the patient's doctor and family via email or a dedicated application.

[1582] Specific examples

[1583] For example, if a 70-year-old male patient suffers from moderate aphasia, the user (medical staff) enters the patient's basic information into the device and sends it to the server. The server analyzes the received information and generates basic patient data. Next, based on the analysis results, the server automatically generates an individual rehabilitation plan including speech exercises and comprehension questions and sends it to the device.

[1584] The user (patient) performs rehabilitation according to instructions on the device, recording the process with a camera and microphone. The device then sends the recorded data to the emotion engine, which analyzes the user's facial expressions and vocal tone to recognize their emotional state. The server adaptively adjusts the rehabilitation content based on the recognized emotion results and provides it to the user. For example, if the user is tired, it can add relaxation exercises.

[1585] After the rehabilitation session, the server creates a daily report based on the data from the device and distributes it to the patient's doctor and family. In this way, the system continuously monitors the patient's condition and provides an optimal rehabilitation plan, achieving efficient and sustainable rehabilitation.

[1586] The processing flow will be explained below.

[1587] Step 1: Enter patient information

[1588] The user (medical staff) inputs the patient's age, gender, aphasia level, and medical record information into the terminal, which then transmits this information to the server.

[1589] Step 2: Receiving and analyzing patient information

[1590] The server receives the patient information sent from the terminal and analyzes the received information to generate basic patient data.

[1591] Step 3: Generate a rehabilitation plan

[1592] Based on the analysis results, the server automatically generates an individual rehabilitation plan for each patient, which includes vocal training, comprehension questions, and communication exercises.

[1593] Step 4: Submit your rehabilitation plan

[1594] The server transmits the generated rehabilitation plan to the terminal, which then displays the received rehabilitation plan to the user.

[1595] Step 5: Emotion Recognition with the Emotion Engine

[1596] The device uses a camera and microphone to record the user's (patient's) facial expressions, voice tone, and word choice during the rehabilitation session. The device then sends this recorded data to the emotion engine, which analyzes the recorded data and recognizes the user's emotional state.

[1597] Step 6: Adaptively adjusting the rehabilitation plan

[1598] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine. For example, if the user is feeling stressed, it adds relaxation exercises.

[1599] Step 7: Rehabilitation

[1600] The user performs rehabilitation by following instructions from the device, which uses a camera and microphone to record video and audio of the rehabilitation session.

[1601] Step 8: Real-time data transmission

[1602] The terminal monitors the progress of rehabilitation in real time and transmits video and audio data to a server.

[1603] Step 9: Real-time data analysis and feedback

[1604] The server analyzes the data received in real time and provides immediate feedback to the user via the terminal as needed.

[1605] Step 10: Ending the rehabilitation session and sending data

[1606] When the user finishes rehabilitation, the terminal transmits the rehabilitation data for that day to the server.

[1607] Step 11: Create a daily report

[1608] The server automatically generates daily reports based on the received rehabilitation data, including rehabilitation progress, achievements, and next steps for improvement.

[1609] Step 12: Share the report

[1610] The server delivers the generated daily reports to the attending physician and family via email or a dedicated application.

[1611] In this way, through the specific actions taken at each step, the system can provide the patient with an individual rehabilitation plan, continuously monitor their progress, and provide optimal support.

[1612] Example 2

[1613] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1614] Currently, rehabilitation for patients with aphasia and brain damage requires flexible rehabilitation plans tailored to each individual, as the effectiveness of these plans varies from patient to patient. However, existing rehabilitation systems lack the ability to appropriately grasp a patient's emotional state and dynamically adjust the rehabilitation plan. They also lack the ability to monitor the progress of rehabilitation in real time and provide immediate feedback. Therefore, there is a need for a system that can generate rehabilitation plans tailored to each patient's individual needs and recognize and adapt to their emotional state during the implementation process.

[1615] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input the patient's age, disability level, and medical information; a means for receiving and analyzing the patient information; a means for automatically generating a rehabilitation plan based on the analysis results; a means for collecting data on the patient's facial expressions, tone of voice, and word choice during the rehabilitation session to recognize the patient's emotional state; a means for adaptively adjusting the rehabilitation plan based on the recognized emotional state; a means for receiving and analyzing the recorded data; a means for creating a daily report based on the analysis results and distributing it to a predetermined recipient; and a means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback. This enables the creation and provision of an optimal rehabilitation plan tailored to the individual patient's condition. Furthermore, by accurately understanding the patient's emotional state during the rehabilitation process and making adaptive adjustments, the effectiveness of the rehabilitation can be maximized.

[1616] "User" is the medical professional or person who operates the system, inputs patient data, and reviews and adjusts rehabilitation plans.

[1617] "Patients" refer to people who need rehabilitation for conditions such as aphasia or brain damage.

[1618] A "server" is a computer system that analyzes patient information, generates rehabilitation plans, and stores and processes data.

[1619] A "terminal" is an electronic device such as a computer, tablet, or smartphone that is directly operated by a user or patient.

[1620] "Patient information" refers to basic data necessary for generating and adjusting a rehabilitation plan, such as age, disability level, and medical information.

[1621] A "rehabilitation plan" is a plan that includes specific rehabilitation content and steps and is automatically generated based on the patient's condition.

[1622] "Emotion engine" refers to algorithms and machine learning models that analyze a patient's facial expressions, tone of voice, and word choice data to recognize their emotional state.

[1623] The "Daily Report" is a report that summarizes the daily activity record, including the progress, achievements, and areas for improvement of the rehabilitation session.

[1624] "Real-time monitoring" is the process of observing rehabilitation progress in real time and collecting and analyzing data.

[1625] "Immediate feedback" refers to advice or instructions provided immediately to a user or patient based on the results of data analysis.

[1626] "Adaptive adjustment" is the process of dynamically changing or adjusting a rehabilitation plan based on the patient's emotional state and progress.

[1627] This invention provides a system for supporting rehabilitation for patients with aphasia or brain damage, recognizing the patient's emotional state and adaptively adjusting the rehabilitation plan based on that state. The system automates the collection and analysis of patient information, generation of rehabilitation plans, emotion recognition, rehabilitation implementation, progress monitoring, and creation and sharing of daily reports.

[1628] The main elements of this system are the server, terminals, and users (patients and medical staff). Below we will explain in detail how each element works together.

[1629] Hardware and Software Used

[1630] Hardware

[1631] 1. Devices: Personal computers, tablets, smartphones, etc. used by users and patients.

[1632] 2. Camera and microphone: Devices for collecting the patient's facial expressions and voice.

[1633] 3. Server: A high-performance computer for analyzing data and generating rehabilitation plans.

[1634] software

[1635] 1. Emotion engine: An AI model to analyze patient emotions (e.g., Google Cloud's Natural Language API or Azure Emotion API).

[1636] 2. Analysis algorithm: Software for analyzing patient information and generating rehabilitation plans.

[1637] 3. Database system: A system for storing patient information and rehabilitation progress data (e.g., MySQL, PostgreSQL).

[1638] 4. Rehabilitation support app: A dedicated application to help patients carry out rehabilitation.

[1639] System example and operation

[1640] Collecting patient information

[1641] The user (medical staff) uses a terminal to input basic information such as the patient's age, disability level, medical information, etc. The terminal then transmits the input information to the server.

[1642] Patient information analysis

[1643] The server receives the patient information sent from the terminal, cleanses the information using an analysis program, and generates basic data.

[1644] Generate rehabilitation plans

[1645] The server automatically generates a rehabilitation plan based on the generated basic data. This rehabilitation plan includes vocal training, comprehension questions, and communication exercises. A prompt sentence is input into the generative AI model to generate the rehabilitation content. An example of a prompt sentence is, "Please generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[1646] Emotion recognition with emotion engine

[1647] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session, and the recorded data is sent to the emotion engine on the server, which analyzes it and recognizes the patient's emotional state.

[1648] Adaptive adjustment of rehabilitation plans

[1649] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results from the emotion engine, for example adding relaxation exercises if the patient is feeling stressed.

[1650] Rehabilitation

[1651] The user (patient) follows instructions from the device to carry out rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation process as video and audio.

[1652] Rehabilitation progress monitoring

[1653] The device transmits real-time information about the rehabilitation progress to a server, which analyzes the data and provides immediate feedback to the user as needed.

[1654] Create and share daily reports

[1655] Once the rehabilitation session is over, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data. This report details the rehabilitation progress, achievements, and next steps for improvement. The report is then sent to the patient's doctor and family via email or a dedicated application.

[1656] As described above, the present invention is a system that provides rehabilitation plans tailored to the individual patient's condition, and enables real-time monitoring and immediate feedback, thereby achieving efficient and effective rehabilitation.

[1657] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1658] Step 1: Enter and submit patient information

[1659] The user (medical staff) inputs basic information such as the patient's age, disability level, and medical information into the terminal.

[1660] Input: Patient's age, disability level, medical information

[1661] Action: Fill out the form and click the "Submit" button.

[1662] Output: The entered patient information is sent from the terminal to the server.

[1663] Step 2: Receiving and analyzing patient information

[1664] The server receives the patient information transmitted from the terminal.

[1665] Input: Patient information sent from the terminal

[1666] Operation: Receive data via the receiving API.

[1667] Output: Pass the received patient information to the analysis program.

[1668] Step 3: Data cleansing and basic data generation

[1669] The server analyzes the received information and cleanses the data to generate the base data.

[1670] Input: Received patient information

[1671] How it works: The analysis program cleanses the data, extracting and organizing only the necessary items.

[1672] Output: Cleansed underlying data

[1673] Step 4: Generate a rehabilitation plan

[1674] The server automatically generates a rehabilitation plan based on the basic data.

[1675] Input: Cleansed underlying data

[1676] How it works: A prompt sentence is input into the generative AI model, which generates rehabilitation content (vocal training, comprehension questions, and communication exercises).

[1677] Example prompt: "Generate a rehabilitation plan for a 70-year-old male with moderate aphasia."

[1678] Output: Automatically generated rehabilitation plan

[1679] Step 5: Submit and view your rehabilitation plan

[1680] The server transmits the generated rehabilitation plan to the terminal, and the terminal displays the rehabilitation plan to the user.

[1681] Input: Auto-generated rehabilitation plan

[1682] Operation: The server sends planning data to the terminal, which receives the data and displays it on the screen.

[1683] Output: User-viewable rehabilitation plan display

[1684] Step 6: Recording your rehabilitation sessions

[1685] The device uses a camera and microphone to record the patient's facial expressions and vocal tone during the rehabilitation session.

[1686] Input: Patient behavior and voice during rehabilitation session

[1687] How it works: Cameras and microphones collect and record data in real time.

[1688] Output: Recorded video and audio data

[1689] Step 7: Sending Emotion Data and Emotion Recognition

[1690] The terminal sends the recorded data to the emotion engine, and the server recognizes the emotional state.

[1691] Input: Recorded video and audio data

[1692] How it works: Data is sent to the server's emotion engine API, where it is analyzed by emotion recognition algorithms.

[1693] Output: Recognized emotional state data

[1694] Step 8: Adaptively adjusting the rehabilitation plan

[1695] The server adaptively adjusts the rehabilitation plan based on the results of the emotion engine.

[1696] Input: Recognized emotional state data

[1697] Action: Re-analyze the rehabilitation plan and add relaxation exercises and content as needed.

[1698] Output: Coordinated rehabilitation plan

[1699] Step 9: Implement and record rehabilitation

[1700] The user (patient) performs rehabilitation by following the instructions on the terminal, and the terminal records the progress.

[1701] Input: Coordinated rehabilitation plan

[1702] How it works: Follow the instructions and use the camera and microphone to record your vocal exercises and workouts in real time.

[1703] Output: Newly recorded video and audio of the rehabilitation session

[1704] Step 10: Real-time analysis of rehabilitation progress and provision of feedback

[1705] The device monitors the progress of rehabilitation in real time, and the server provides immediate feedback based on the analysis results.

[1706] Input: Real-time rehabilitation progress data

[1707] What it does: Sends data to a server that analyzes progress and forwards instant advice and encouragement to the device.

[1708] Output: Immediate feedback provided to the user

[1709] Step 11: Create and share daily reports

[1710] The terminal transmits the rehabilitation data for one day to the server, which automatically generates a daily report.

[1711] Input: One day of rehabilitation data

[1712] What it does: Aggregate data and generate reports that show progress, achievements, and areas for improvement.

[1713] Output: Auto-generated daily report

[1714] Step 12: Daily Report Delivery

[1715] The server distributes the generated daily reports to the patient's doctor and family.

[1716] Input: Auto-generated daily report

[1717] What it does: Reports are delivered via email or a dedicated application.

[1718] Output: Daily reports sent to your doctor and family

[1719] (Application example 2)

[1720] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1721] Patients undergoing rehabilitation often experience significant mental stress and anxiety. Conventional rehabilitation systems struggle to recognize a patient's emotional and psychological state in real time and provide appropriate support. This can result in insufficient rehabilitation benefits and further problems. While there is a need to understand a patient's mental crisis in real time and respond immediately in emergencies, current technology is unable to adequately address this need.

[1722] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1723] In this invention, the server includes means for inputting the patient's age, aphasia level, and medical information, means for receiving and analyzing the patient information, means for automatically generating a rehabilitation plan based on the analysis results, means for displaying the rehabilitation plan and issuing instructions to the user, means for recording video and audio for recording the patient's rehabilitation, means for receiving the recorded data and analyzing the emotional state, means for creating a daily report based on the analysis results and distributing it to a predetermined recipient, and means for making an emergency call if the user's psychological state is judged to be dangerous. This makes it possible to analyze the patient's emotional state in real time and make adaptive adjustments to the rehabilitation plan, as well as to make an emergency call immediately in the event of an emergency.

[1724] "Patient" refers to a person with a health condition requiring rehabilitation.

[1725] "Age" is information that indicates the period of time that has elapsed since the patient's date of birth until the present.

[1726] "Aphasia level" refers to a scale or information that indicates the degree of impairment of a patient's language function.

[1727] "Medical information" refers to various information related to medical treatment, such as patient charts and medical records.

[1728] "Input means" refers to a device or method for inputting data such as the patient's age, aphasia level, and medical information into the system.

[1729] "Receiving means" refers to a device or method by which the server receives patient information transmitted from the terminal.

[1730] The term "analysis means" refers to a method or device for analyzing received patient information and generating data necessary for formulating a rehabilitation plan.

[1731] "Rehabilitation plan" refers to a program that includes rehabilitation activities appropriate for a patient.

[1732] "Display means" refers to a device or method for visually showing the generated rehabilitation plan to the user.

[1733] "Instruction means" refers to a device or method for encouraging a user to take specific actions or practice based on a rehabilitation plan.

[1734] "Recording Means" refers to any device or method for capturing video and audio recordings of a patient's rehabilitation session.

[1735] "Emotional state" refers to a state that indicates the patient's psychological feelings.

[1736] "Emergency notification means" refers to a device or method for immediately reporting when the user's psychological state is judged to be dangerous.

[1737] "Daily report" refers to a report summarizing the results of daily rehabilitation.

[1738] "Delivery means" refers to a device or method for transmitting the generated daily report to a predetermined recipient.

[1739] "Real-time" refers to processing and response occurring immediately, without any time delay.

[1740] MODE FOR CARRYING OUT THE INVENTION

[1741] This invention is a technology for monitoring a patient's emotional state in real time in a rehabilitation system and providing an adaptive rehabilitation plan. At the same time, it also has a function to make an emergency call if the user's psychological state is judged to be dangerous. This system mainly consists of three elements: a server, a terminal, and the user.

[1742] System configuration and operation

[1743] 1. Collection and analysis of patient information

[1744] The server receives the patient's age, aphasia level, medical information, etc. sent from the terminal and analyzes this information. The user (medical staff) inputs the necessary data through the terminal and sends it to the server. This process generates basic data for understanding the patient's condition.

[1745] 2. Generating a rehabilitation plan

[1746] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal and can be viewed by the user (patient).

[1747] 3. Emotion recognition and adaptive regulation by the emotion engine

[1748] The device uses a camera and microphone to record the patient's facial expressions, voice tone, and word choice during the rehabilitation session, and sends this information to the emotion engine. The emotion engine analyzes this information and recognizes the user's emotional state. The server adaptively adjusts the rehabilitation plan based on the emotion recognition results.

[1749] 4. Rehabilitation implementation and progress monitoring

[1750] The user (patient) follows instructions on the device to perform rehabilitation. The rehabilitation session is recorded with video and audio, and the device sends this data to a server. The server analyzes the data and provides real-time feedback to the user via the device as needed.

[1751] 5. Emergency call

[1752] If the user's psychological state is judged to be dangerous, for example, if they show fear or anger, the device will immediately make an emergency call, which may include automatically notifying designated emergency contacts and contacting public authorities such as the police.

[1753] 6. Create and share daily reports

[1754] After the rehabilitation session, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family. The daily report includes rehabilitation progress, achievements, and next steps for improvement.

[1755] Specific examples

[1756] As a concrete example, consider a case where a user is undergoing rehabilitation using a smartphone application. During the rehabilitation session, the device's camera and microphone constantly monitor the user's facial expressions and voice. An emotion engine analyzes the user's emotions, and if the user expresses fear or anger, the device immediately initiates an emergency call. This call includes contacting designated emergency contacts and public authorities.

[1757] Prompt Sentence Examples

[1758] For example, you can use the following prompts for a generative AI model to achieve emotion recognition and emergency calling capabilities:

[1759] Implement an application that recognizes emotions from the user's facial and voice data, and makes an emergency call when danger is detected.

[1760] Facial expression data is captured in real time from the smartphone camera and analyzed using emotion recognition models.

[1761] · It also analyzes voice data to determine emotional state.

[1762] If the user expresses fear or anger, it will notify your emergency contacts.

[1763] The system allows for real-time analysis of a patient's emotional state, providing adaptive rehabilitation plans, as well as immediate response to dangerous situations.

[1764] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1765] Step 1:

[1766] The server receives the patient's age, aphasia level, and medical information sent from the device. Based on this input data, it prepares to analyze the patient's information. As a result of the analysis, basic data for understanding the patient's condition is generated.

[1767] Step 2:

[1768] The server automatically generates a rehabilitation plan based on the analysis results. This plan includes vocal training, comprehension questions, communication exercises, etc. The generated rehabilitation plan is sent to the terminal so that the user (patient or medical staff) can review it.

[1769] Step 3:

[1770] The device uses a camera and microphone to record the user's facial expressions, voice tone, and word choice during the rehabilitation session. This information is then sent to an emotion engine to analyze the user's emotional state. The input to this process is video and audio data, and the output is emotion recognition results.

[1771] Step 4:

[1772] The emotion engine analyzes the received data and recognizes the user's emotional state (e.g., joy, sadness, anxiety, anger, etc.). The model performs data calculations based on the input video and audio data, and obtains the emotional state as output.

[1773] Step 5:

[1774] The server adaptively adjusts the rehabilitation plan based on the emotion recognition results. For example, if the user is feeling stressed, it adds relaxation exercises. Based on the emotion recognition results as input, the server generates a revised and adjusted rehabilitation plan as output.

[1775] Step 6:

[1776] The user (patient) follows the instructions on the device to perform rehabilitation. The rehabilitation content is displayed on the screen, and the patient uses a camera and microphone to practice vocalization and exercises. The device records the rehabilitation session as video and audio.

[1777] Step 7:

[1778] The device monitors the progress of rehabilitation in real time and transmits the video and audio data to the server. The server analyzes this data and provides real-time feedback to the user via the device as needed. The input is real-time video and audio data, and the output is feedback information.

[1779] Step 8:

[1780] The device continuously monitors the user's psychological state based on the data recorded during the rehabilitation session. If the user's psychological state is judged to be dangerous, the device immediately makes an emergency call. The input is the analysis result of the psychological state, and the output is the call action.

[1781] Step 9:

[1782] When the rehabilitation session ends, the device sends the day's rehabilitation data to the server. The server automatically generates a daily report based on the received data and distributes it to the patient's doctor and family via email or a dedicated application. The input is the day's rehabilitation data, and the output is the daily report.

[1783] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1784] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1785] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1786] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1787] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1788] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1789] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1790] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1791] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1792] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1793] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1794] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1795] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1796] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1797] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1798] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1799] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1800] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1801] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1802] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1803] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1804] The following is further disclosed regarding the above embodiment.

[1805] (Claim 1)

[1806] A means of inputting the patient's age, aphasia level, and medical record information;

[1807] means for receiving and analyzing said patient information;

[1808] means for automatically generating a rehabilitation plan based on the analysis results;

[1809] means for displaying the rehabilitation plan and providing instructions to a user;

[1810] a video and audio recording means for recording the rehabilitation of the patient;

[1811] means for receiving and analyzing the recorded data;

[1812] means for generating and distributing daily reports based on the analysis results to predetermined recipients;

[1813] A system including:

[1814] (Claim 2)

[1815] 10. The system of claim 1, further comprising means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback.

[1816] (Claim 3)

[1817] 10. The system of claim 1, further comprising means for selecting and incorporating patient family information and communication means into the rehabilitation plan.

[1818] "Example 1"

[1819] (Claim 1)

[1820] A means of inputting the patient's age, aphasia level, and medical record information;

[1821] means for receiving and storing basic information of the patient in a database;

[1822] means for analyzing the received patient information and generating basic data for evaluating the patient's condition;

[1823] means for automatically generating a rehabilitation plan based on the analysis results;

[1824] means for displaying the rehabilitation plan and providing instructions to a user;

[1825] A means of recording the patient's rehabilitation with a camera and microphone;

[1826] means for receiving the recorded data and monitoring it in real time to generate instant feedback;

[1827] means for analyzing the recorded data, automatically generating daily reports, and distributing the reports to predetermined recipients;

[1828] A system including:

[1829] (Claim 2)

[1830] 10. The system of claim 1, further comprising means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback.

[1831] (Claim 3)

[1832] 10. The system of claim 1, further comprising means for selecting and incorporating patient family information and communication means into the rehabilitation plan.

[1833] "Application Example 1"

[1834] (Claim 1)

[1835] A means of inputting the patient's age, aphasia level, and medical record information;

[1836] means for receiving and analyzing said patient information;

[1837] means for automatically generating a rehabilitation plan based on the analysis results;

[1838] means for displaying the rehabilitation plan and providing instructions to a user;

[1839] a video and audio recording means for recording the rehabilitation of the patient;

[1840] means for receiving and analyzing the recorded data;

[1841] means for generating and distributing daily reports based on the analysis results to predetermined recipients;

[1842] A means for providing rehabilitation plans to rehabilitation center terminals and managing progress;

[1843] A system including:

[1844] (Claim 2)

[1845] 10. The system of claim 1, further comprising means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback.

[1846] (Claim 3)

[1847] 10. The system of claim 1, further comprising means for selecting and incorporating patient family information and communication means into the rehabilitation plan.

[1848] "Example 2: Combining Emotion Engines"

[1849] (Claim 1)

[1850] a means for a user to input the patient's age, disability level, and medical information;

[1851] means for receiving and analyzing said patient information;

[1852] means for automatically generating a rehabilitation plan based on the analysis results;

[1853] means for displaying the rehabilitation plan and providing instructions to a user;

[1854] a video and audio recording means for recording the rehabilitation of the patient;

[1855] a means of collecting data on the patient's facial expressions, voice tone, and word choice during the rehabilitation session to recognize their emotional state;

[1856] a means for adaptively adjusting the content of a rehabilitation plan based on the perceived emotional state;

[1857] means for receiving and analyzing the recorded data;

[1858] means for generating and distributing daily reports based on the analysis results to predetermined recipients;

[1859] A system including:

[1860] (Claim 2)

[1861] 10. The system of claim 1, further comprising means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback.

[1862] (Claim 3)

[1863] 10. The system of claim 1, further comprising means for selecting the patient's family information and communication means and reflecting them in the rehab...

Claims

1. A means of inputting the patient's age, aphasia level, and medical record information; means for receiving and analyzing said patient information; means for automatically generating a rehabilitation plan based on the analysis results; means for displaying the rehabilitation plan and providing instructions to a user; a video and audio recording means for recording the rehabilitation of the patient; means for receiving and analyzing the recorded data; means for generating and distributing daily reports based on the analysis results to predetermined recipients; A system including:

2. 10. The system of claim 1, further comprising means for monitoring the patient's rehabilitation progress in real time and providing immediate feedback.

3. The system of claim 1 further comprising means for selecting and incorporating patient family information and communication means into the rehabilitation plan.

Images