System

Abstract

A system is provided.SOLUTION: The system includes a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means for assisting a medical action, a sensor means for monitoring the state of a patient in real time, a remote assistance means for connecting to a medical expert in real time, and a means for performing encryption and access control of patient data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In emergency medical situations, ordinary citizens without specialized medical knowledge or skills are often forced to respond immediately. In particular, when cardiac arrest or other acute symptoms occur, the inability to provide prompt, appropriate treatment can be a direct threat to the patient's life. Despite the widespread availability of medical equipment such as AEDs, many users remain unsure of how to use them, resulting in a low usage rate. Furthermore, with the risk of sudden death increasing in an aging society, there is a growing demand for appropriate initial medical treatment. To solve these problems, there is an urgent need to develop a system that allows appropriate emergency medical treatment without specialized knowledge. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means: a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means for assisting medical procedures, a sensor means for monitoring the patient's condition in real time, a remote assistance means for connecting with a medical professional in real time, and a means for encrypting and controlling access to patient data.

[0006] This system monitors the patient's condition in real time and immediately issues appropriate medical instructions if an abnormality is detected. Users can perform initial medical procedures, such as cardiopulmonary resuscitation and wound care, following audio and visual guidance. It also uses image recognition technology to diagnose the patient's condition and provide appropriate instructions based on the results. Furthermore, it uses natural language processing to smoothly communicate with the user, collect detailed information, and provide appropriate medical guidance. It also features a remote assistance function that allows users to connect with medical professionals in real time for guidance, enabling the management of acute symptoms. It also incorporates security and privacy protection measures, such as encryption of patient data, access control, and automatic application of security patches, allowing for safe use. In this way, the present invention aims to reduce the risk of sudden death by enabling appropriate responses at emergency medical sites without specialized knowledge.

[0007] "Natural language processing means" is a technology that uses artificial intelligence to communicate with users in natural language.

[0008] "Image recognition means" refers to technology that analyzes image data acquired using cameras or other sensors and recognizes specific patterns or anomalies.

[0009] "Voice guidance means" refers to a technique for providing instructions and information to a user through voice.

[0010] "Artificial intelligence means" refers to a system that has the knowledge and judgment capabilities to assist medical procedures using machine learning and deep learning.

[0011] "Sensor means" refers to devices or technologies for monitoring the patient's physical condition in real time, and is used to obtain information such as temperature, pulse, and respiration.

[0012] "Remote assistance methods" are technologies that allow patients to connect with and receive guidance from medical professionals in remote locations in real time.

[0013] "Encryption and access control measures" refers to technologies that encrypt data and control access rights to ensure the security of patient data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element works together to assist in emergency medical treatment. Specific embodiments for carrying out the present invention will be described below.

[0036] Natural Language Processing Implementation

[0037] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question and generate a response such as, "The patient's heart rate is normal." The response is conveyed to the user via the terminal in voice or text.

[0038] Image Recognition Implementation

[0039] Image data acquired from cameras and sensors is analyzed using image recognition techniques to diagnose a patient's condition. For example, a device takes an image of a patient, and the server analyzes the image using image recognition technology to predict the patient's condition from the patient's facial expression and appearance. The diagnosis result is notified to the user visually and audibly.

[0040] Implementing voice guidance

[0041] The voice guidance means guides the user through emergency medical procedures by voice. For example, if cardiopulmonary resuscitation is required, the server issues specific instructions such as "Please press on the chest in a steady rhythm" through the voice guidance program and conveys this to the user via the terminal.

[0042] Implementation of artificial intelligence for medical procedure assistance

[0043] AI tools utilize machine learning and deep learning to have the knowledge and judgment necessary to assist medical procedures. The server collects patient data in real time and uses AI models to determine the optimal treatment method. The device then instructs the user on the treatment method and encourages appropriate action.

[0044] Sensor Implementation

[0045] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the collected data and immediately generates appropriate instructions if an abnormality is detected.

[0046] Implementing Remote Assistance

[0047] The remote assistance method allows a user to connect with a medical professional in real time and receive guidance. For example, a device can establish a connection and allow the medical professional to check the patient's condition in real time. The server receives instructions from the professional and relays them to the user via the device. This allows for the management of acute symptoms.

[0048] Implementing encryption and access control

[0049] Encryption and access control are technologies used to ensure the safety of patient data. Patient data collected by the device is encrypted before being sent to the server. The server securely stores the encrypted data and ensures that only authorized users can access the data based on the access control policy. Security patches are also regularly applied to maintain the safety of the system.

[0050] Specific examples

[0051] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[0052] 1. Sensor detection

[0053] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[0054] 2. CPR instructions

[0055] Server: Uses image recognition technology to check the patient's condition and determine that they have undergone cardiac arrest.

[0056] Terminal: Based on instructions from the server, the terminal provides voice and on-screen guidance on cardiopulmonary resuscitation procedures to the user.

[0057] User: Begin CPR as instructed.

[0058] 3. Providing remote assistance

[0059] Device: Connect with medical professionals and receive real-time guidance.

[0060] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[0061] User: Perform additional medical procedures under the guidance of a professional.

[0062] 4. Ensuring safety

[0063] Terminal: Patient data is encrypted and securely transmitted to the server.

[0064] Server: Encrypts and stores data and applies access controls.

[0065] In this way, the present invention aims to support the implementation of appropriate medical procedures in emergencies even without specialized knowledge, thereby reducing the risk of sudden death.

[0066] The processing flow will be explained below.

[0067] Natural Language Processing Implementation

[0068] Step 1:

[0069] Terminal: The user types a question into the microphone, for example, "How is the patient doing?"

[0070] Step 2:

[0071] Terminal: Converts voice data into text and sends it to the server.

[0072] Step 3:

[0073] Server: The received text data is input into a natural language processing model and the question content is analyzed.

[0074] Step 4:

[0075] Server: Generates appropriate responses to questions and sends them to the terminal in text format.

[0076] Step 5:

[0077] Terminal: Converts the received response into speech and provides it to the user. The speech response is, "The patient's heart rate is normal."

[0078] Image Recognition Implementation

[0079] Step 1:

[0080] Terminal: Captures images of the patient through a camera and sends the image data to the server.

[0081] Step 2:

[0082] Server: Inputs the received image data into an image recognition model to predict the patient's condition.

[0083] Step 3:

[0084] Server: Generates a diagnosis of the medical condition and determines the appropriate course of treatment.

[0085] Step 4:

[0086] Server: Sends instructions on how to proceed to the terminal.

[0087] Step 5:

[0088] Terminal: The user is notified of the diagnosis and treatment method visually and audibly. "The patient is unconscious. Please begin CPR." is displayed and audible.

[0089] Implementing voice guidance

[0090] Step 1:

[0091] Server: Generates appropriate medical procedures based on the patient's condition.

[0092] Step 2:

[0093] Server: Sends the generated procedure information to the terminal.

[0094] Step 3:

[0095] Device: Provides instructions to the user through a voice guidance system, such as "Compress the chest about 5 cm at a rate of 100 times per minute."

[0096] Step 4:

[0097] User: Follow the audio prompts to perform CPR.

[0098] Implementation of artificial intelligence for medical procedure assistance

[0099] Step 1:

[0100] Terminal: Collects the patient's vital data (e.g., heart rate, blood pressure) from sensors and sends it to the server.

[0101] Step 2:

[0102] Server: Inputs collected vital data into an artificial intelligence model to determine appropriate treatment.

[0103] Step 3:

[0104] Server: Based on the judgment results, generates appropriate treatment procedures and sends them to the terminal.

[0105] Step 4:

[0106] Terminal: Provides the user with visual and audio instructions for appropriate treatment. "Patient's heart rate is dropping. Prepare oxygen for inhalation."

[0107] Sensor Implementation

[0108] Step 1:

[0109] Device: Uses sensors to capture real-time vital data such as the patient's heart rate, temperature, and respiratory rate.

[0110] Step 2:

[0111] Terminal: Sends the acquired data to the server.

[0112] Step 3:

[0113] Server: Analyzes the received data and determines whether there are any abnormalities.

[0114] Step 4:

[0115] Server: If an abnormality is detected, it determines the appropriate response method and sends instruction information to the terminal.

[0116] Step 5:

[0117] Device: Notifies the user of the abnormality and how to respond. "An abnormal heart rate has been detected. Please begin CPR immediately."

[0118] Implementing Remote Assistance

[0119] Step 1:

[0120] Terminal: When an abnormality in the patient is detected, the remote assistance function is activated and sends a request to the server to establish a connection with a medical professional.

[0121] Step 2:

[0122] Server: Establishes connection with medical professional services and transmits patient status in real time.

[0123] Step 3:

[0124] Server: Receives instructions from medical professionals and sends them to the device.

[0125] Step 4:

[0126] Terminal: Provides audio and visual instructions to the user from a specialist. "Please put an oxygen mask on the patient. Detailed instructions are below."

[0127] Step 5:

[0128] User: Follow the instructions of a professional and take appropriate measures.

[0129] Implementing encryption and access control

[0130] Step 1:

[0131] Terminal: Collects patient data (e.g., vital data, diagnostic results) and encrypts the data using an encryption algorithm.

[0132] Step 2:

[0133] Device: Sends encrypted data to the server.

[0134] Step 3:

[0135] Server: Receives the encrypted data and stores it in a secure database.

[0136] Step 4:

[0137] Server: Authenticates users accessing data and enforces access control policies.

[0138] Step 5:

[0139] Server: Regularly apply security patches to keep your systems secure.

[0140] The above is the flow of specific processing steps for each function, which will realize a system that supports appropriate responses at the scene of emergency medical care, even without specialized knowledge.

[0141] Example 1

[0142] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] In modern society, rapid and accurate responses are required in emergency medical situations, but appropriate treatment is often not provided due to the absence of experts or lack of knowledge. Furthermore, in order to accurately grasp the patient's condition and provide the necessary treatment, a wide range of technologies and knowledge must be integrated. In particular, the lack of real-time monitoring of patient conditions, accurate diagnosis of illness, and appropriate medical guidance is a problem.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0145] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for analyzing user questions and generating appropriate responses, means for analyzing image data acquired by a camera and predicting the patient's condition, means for providing audio and visual guidance for medical procedure procedures, and means for encrypting and storing collected patient data, thereby enabling prompt and appropriate responses in emergency medical situations.

[0146] "Natural language processing means" refers to means for communicating with users in natural language and for understanding and analyzing questions and instructions.

[0147] "Image recognition means" is a means for analyzing image data acquired by a camera or sensor and identifying the state and characteristics of an object.

[0148] The "audio guidance means" is a means for providing instructions and information to the user by voice.

[0149] "Artificial intelligence means" refers to means that use machine learning and deep learning to assist medical procedures and determine the most appropriate treatment methods.

[0150] The "sensor means" is a means for monitoring the physical condition of a patient in real time and collecting the data.

[0151] "Remote assistance means" means a means of connecting with a medical professional in real time to receive specialized guidance and support.

[0152] "Encryption and access control means" means means for encrypting patient data and ensuring that only authorized users can access the data.

[0153] The "means for analyzing questions and generating responses" refers to means for analyzing voice or text questions entered by a user and generating appropriate responses.

[0154] The "means for analyzing image data and predicting the condition" is a means for analyzing images of a patient taken by a camera and predicting the condition of the patient.

[0155] The "means for providing guidance on the procedure of medical treatment" is a means for providing the user with audio and visual guidance on the specific procedure of emergency medical treatment.

[0156] "Means for encrypting and storing data" refers to means for encrypting collected patient data and storing it securely.

[0157] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element cooperates to assist emergency medical treatment. The embodiments of the present invention will be described in detail below.

[0158] Natural language processing implementation:

[0159] The server uses natural language processing to analyze questions or instructions entered by the user via voice or text and generate an appropriate response. For example, if a user asks, "How is the patient's condition?", the server uses natural language processing technology to understand the question and generates a response such as, "The patient's heart rate is normal." This response is then communicated to the user via the terminal via voice or text.

[0160] Image recognition implementation:

[0161] To understand the patient's condition, the device sends image data captured by the camera to a server. The server then analyzes the image data using image recognition tools and predicts the patient's condition based on the patient's facial expression and appearance. The diagnosis results are notified to the user visually and audibly. For example, the server checks for abnormalities in the patient's complexion or the presence of external injuries.

[0162] Implementing audio guidance methods:

[0163] The server uses the voice guidance means to guide the user through the procedures for emergency medical treatment. For example, if cardiopulmonary resuscitation is required, specific instructions such as "Please push on the chest in a steady rhythm" are provided by voice. These instructions are transmitted to the user through the terminal.

[0164] Implementation of artificial intelligence means:

[0165] The server uses machine learning and deep learning techniques to analyze patient data in real time and determine the optimal treatment. The device then instructs the user on this treatment and encourages appropriate medical intervention. For example, an artificial intelligence model can be used to detect abnormal heart rates or respiratory arrest and generate instructions for immediate response.

[0166] Sensor means implementation:

[0167] The device has sensors that monitor the patient's heart rate, body temperature, respiratory rate, and other data in real time. This data is sent to a server, which analyzes the data and immediately generates instructions if an abnormality is detected. For example, if a sudden change in heart rate is detected, it will immediately issue instructions for CPR.

[0168] Implementing remote assistance methods:

[0169] The terminal has a remote assistance means to connect to a medical professional in real time and receive appropriate guidance. The medical professional checks the patient's condition and provides necessary guidance. The server receives the expert's instructions and relays them to the user, allowing the user to perform accurate treatment based on the expert's guidance.

[0170] Implementing encryption and access control measures:

[0171] Patient data collected by the device is encrypted before being sent to the server, where it is stored and access is controlled to ensure only authorized users can access it. Security patches are also regularly applied to maintain safety.

[0172] Examples:

[0173] As a concrete example, consider a situation where a cardiac arrest occurs in a public place. In this case, the system operates as follows:

[0174] 1. A user uses a device to ask, "How is the patient doing?"

[0175] 2. The device takes an image of the patient with its camera and sends it to the server.

[0176] 3. The server uses image recognition to diagnose the patient's condition and determine that cardiopulmonary resuscitation is necessary.

[0177] 4. The server generates instructions using voice guidance means, such as "Please press the chest at a regular rhythm," and conveys this to the user via the terminal.

[0178] 5. Use remote assistance methods to connect with medical professionals for additional guidance in real time.

[0179] 6. The data collected by the device is encrypted and securely sent to the server.

[0180] An example of a prompt sentence could be something like, "Please explain how the system would respond if someone had a sudden cardiac arrest in the park."

[0181] This enables the system to respond quickly and accurately in emergencies and support appropriate medical treatment.

[0182] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0183] Step 1:

[0184] The user enters a question.

[0185] Specific behavior:

[0186] User: Ask via voice or text, "How is the patient doing?"

[0187] Device: Converts speech to text and sends the data to the server.

[0188] Input: User's voice or text question.

[0189] Output: Question data in text format.

[0190] Step 2:

[0191] The server analyzes the user's question and generates a response.

[0192] Specific behavior:

[0193] Server: Uses natural language processing techniques to analyze the user's question and generate an appropriate response.

[0194] Server: Generates a response, for example, "The patient's heart rate is normal."

[0195] Input: The question data submitted by the user in text format.

[0196] Output: The response data in text format.

[0197] Step 3:

[0198] The user is notified of the response from the server.

[0199] Specific behavior:

[0200] Terminal: The response data from the server is converted into voice using speech synthesis technology and notified to the user.

[0201] Terminal: Also displayed as text data.

[0202] Input: Response data sent from the server in text format.

[0203] Output: Voice and text response notification.

[0204] Step 4:

[0205] Image data is acquired to confirm the patient's condition.

[0206] Specific behavior:

[0207] Device: Takes images of the patient's face and body using a camera.

[0208] Terminal: Sends the captured image data to the server.

[0209] Input: Patient images.

[0210] Output: Image data sent to the server.

[0211] Step 5:

[0212] The server analyzes the image data and predicts the condition of the disease.

[0213] Specific behavior:

[0214] Server: Uses image recognition technology to analyze a patient's facial expression, appearance, and possible injuries to predict their condition.

[0215] Server: Generates an analysis result, for example, "The patient may be unconscious."

[0216] Input: Image data sent from the device.

[0217] Output: Predicted disease state.

[0218] Step 6:

[0219] The prediction results are notified to the user.

[0220] Specific behavior:

[0221] Terminal: Notifies the user of predicted medical conditions via voice or text.

[0222] Input: Prediction results of the disease condition sent from the server.

[0223] Output: Prediction result notification in audio and text format.

[0224] Step 7:

[0225] Provide guidance on emergency medical procedures.

[0226] Specific behavior:

[0227] User: Ask "What do I do?"

[0228] Server: Analyzes the situation and determines the medical intervention required.

[0229] Server: For example, it determines that cardiopulmonary resuscitation is necessary and generates instructions such as "Push on the chest in a steady rhythm."

[0230] Terminal: Provides audio and visual instructions to the user.

[0231] Input: Questions from the user and patient data.

[0232] Output: Medical procedure instructions in audio and visual format.

[0233] Step 8:

[0234] Offer remote assistance.

[0235] Specific behavior:

[0236] Device: Establish a connection with a medical professional and receive guidance in real time.

[0237] Server: Receives expert instructions and conveys them to the user in an appropriate manner.

[0238] User: Follow the guidance of a professional and perform appropriate medical procedures.

[0239] Input: Patient status and professional instructions.

[0240] Output: Real-time expert feedback.

[0241] Step 9:

[0242] The patient data is encrypted and sent to the server.

[0243] Specific behavior:

[0244] Terminal: Encrypts patient data (heart rate, images, prediction results, etc.).

[0245] Device: Sends encrypted data to the server.

[0246] Input: Patient data.

[0247] Output: The encrypted data.

[0248] Step 10:

[0249] The server stores encrypted data and controls access.

[0250] Specific behavior:

[0251] Server: Collected data is stored in encrypted form.

[0252] Server: Ensure that only authenticated users can access the data.

[0253] Servers: Regularly apply security patches to maintain security.

[0254] Input: Encrypted patient data.

[0255] Output: Stored data and access logs.

[0256] (Application example 1)

[0257] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0258] Conventional emergency medical intervention systems have limitations in their ability to quickly reach the scene and accurately grasp the patient's condition in real time. Systems utilizing autonomous vehicles, in particular, lack the advanced technology necessary to provide appropriate medical intervention in an emergency. Another problem is the difficulty of providing accurate instructions while connecting with medical professionals in real time. The present invention aims to solve these problems and provide an advanced emergency medical intervention support system using autonomous vehicles to reduce the risk of sudden patient death.

[0259] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0260] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for automatically arriving at the emergency scene, means for diagnosing the patient's condition using an in-vehicle camera, and means for assisting medical procedures in real time within the autonomous vehicle, thereby enabling rapid arrival at the emergency scene and accurate medical procedures and real-time collaboration with medical professionals at the scene.

[0261] "Natural language processing means" refers to means that enables communication with the user in natural language.

[0262] "Image recognition means" refers to a means for analyzing image data acquired from a camera or sensor and diagnosing the patient's condition.

[0263] The "audio guidance means" is a means for providing the user with audio guidance on the specific steps of a medical procedure.

[0264] "Artificial intelligence means to assist medical procedures" refers to means that utilize machine learning and deep learning to acquire the knowledge and judgment capabilities necessary for medical procedures.

[0265] The "sensor means" is a means for monitoring the patient's physical condition, such as heart rate, body temperature, and respiratory rate, in real time.

[0266] "Remote assistance means" means a means for connecting with a medical professional in real time for guidance.

[0267] "Encryption and access control measures" means measures that encrypt patient data to ensure its security and ensure that only authorized users can access it.

[0268] "Means for arriving at an emergency scene using autonomous driving" means means that enable a vehicle to quickly arrive at an emergency scene using autonomous driving technology.

[0269] "Image recognition means for diagnosing a patient's condition using an in-vehicle camera" refers to a means for diagnosing a patient's condition by analyzing images taken by a camera installed inside an autonomous vehicle.

[0270] "Means for assisting medical procedures in real time within an autonomous vehicle" means means for assisting medical procedures in real time within an autonomous vehicle.

[0271] The present invention is an advanced emergency medical intervention assistance system using an autonomous vehicle, which integrates various technologies, including natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical intervention, sensors means, remote assistance means, encryption and access control means, means for autonomously driving to arrive at the emergency scene, image recognition means for diagnosing the patient's condition using an in-vehicle camera, and means for providing real-time medical intervention assistance within the autonomous vehicle.

[0272] Autonomous driving to emergency scene

[0273] The server receives emergency calls and uses GPS navigation technology to reach the scene via the optimal route. Utilizing Google Maps API, it analyzes traffic conditions in real time and selects the optimal route. The autonomous vehicle then automatically drives along this route.

[0274] Natural language processing tools

[0275] The server uses natural language processing models (e.g., the BERT model) to communicate with emergency responders in natural language. This allows the user to ask a question such as "How is the patient doing?" and the system understands the question and generates a response such as "The patient's heart rate is normal." This response is then conveyed to the user via voice or text.

[0276] Image Recognition Method

[0277] The server analyzes image data captured by the vehicle's camera using a convolutional neural network (CNN) to diagnose the patient's condition. This data is securely transmitted to the server, and the results of the analysis are notified to the user.

[0278] Audio guidance means

[0279] The server uses a voice guidance means to provide voice instructions on specific steps of medical procedures. For example, if cardiopulmonary resuscitation is required, the server uses a technology that provides voice instructions such as "Push on the chest in a steady rhythm."

[0280] Remote Assistance Methods

[0281] The server uses a video conferencing system (e.g., Zoom API) to connect with medical professionals in the autonomous vehicle in real time, allowing the medical professionals to monitor the situation on-site and provide instructions.

[0282] Sensor means

[0283] The server uses sensors to monitor heart rate, body temperature, breathing rate, etc. in real time. This data is collected by the vehicle's in-vehicle system and transmitted to the server. If an abnormality is detected, appropriate instructions are generated immediately.

[0284] Encryption and Access Control Measures

[0285] The server uses a method for encrypting and controlling access to patient data, using AES-256 data encryption and OAuth user authentication to ensure data security.

[0286] Specific examples of emergencies

[0287] For example, consider a sudden cardiac arrest in a public place. In this case, the system operates as follows:

[0288] 1. An autonomous vehicle receives an emergency call and arrives at the scene via the optimal route.

[0289] 2. Diagnose the patient's condition using image recognition means and provide specific instructions to the user via voice guidance.

[0290] 3. An example of a prompt for reproduction is the text, "As an example of the use of an emergency medical guidance system, please tell us how to respond to a patient in cardiac arrest."

[0291] Program processing

[0292] The server connects various technologies to efficiently support emergency medical procedures. The system integrates autonomous vehicle navigation, natural language processing, image recognition, voice guidance, remote assistance, and security management to ensure a fast and accurate response in emergencies.

[0293] Operation of this system enables rapid arrival at the emergency site and appropriate medical treatment on the spot, significantly reducing the risk of sudden patient death.

[0294] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0295] Step 1:

[0296] The server receives emergency calls. It receives emergency call data (the caller's location information and the details of the emergency) as input, and calculates the optimal route as output. Data processing involves using a GPS navigation system (e.g., Google Maps API) to analyze real-time traffic information and generate the optimal route to quickly reach the scene. The server then sends the generated route information to the autonomous vehicle.

[0297] Step 2:

[0298] The autonomous vehicle (terminal) begins autonomous driving towards the site based on the route information received from the server. The input is route information, and the output is the execution of autonomous driving to the site. Specifically, the vehicle's autonomous driving system controls the vehicle according to GPS information, and drives while recognizing traffic lights and obstacles.

[0299] Step 3:

[0300] When the device arrives at the scene, it uses natural language processing to begin communicating with the user (emergency responder). The input is the user's question or instruction, and the output is a response to the user. For data processing, a natural language processing model (e.g., BERT) is used to analyze the meaning of the question and generate an appropriate response. The server transmits the generated response from the device to the user via voice or text.

[0301] Step 4:

[0302] The server uses the device's camera to diagnose the patient's condition on-site using image recognition. The input is camera image data, and the output is the diagnosis result. Data processing involves image analysis using a convolutional neural network (CNN) to diagnose the patient's external symptoms. The diagnosis result is transmitted from the server to the user via the device.

[0303] Step 5:

[0304] The terminal uses voice guidance generated by the server to instruct the user on the steps of the medical procedure. The input is the diagnosis result and the output is the voice guidance. The server executes the voice guidance program, generates specific instructions, and transmits them to the user as voice from the terminal. For example, the instruction may be "Please press on the chest at a regular rhythm."

[0305] Step 6:

[0306] The device connects to medical professionals in real time via a server to provide remote assistance. The input is a connection request, and the output is the establishment of a video call connection. For data processing, a video conferencing system (e.g., Zoom API) is used to send and receive video and audio in real time to receive instructions from medical professionals. The instructions from the professional are then communicated to the user via the device.

[0307] Step 7:

[0308] The server securely stores all collected patient data using encryption and access control measures. The input is collected data, and the output is encrypted data. Data processing involves encrypting the data using the AES-256 encryption algorithm and controlling access via OAuth. The data is securely stored on the server and can only be accessed by authorized users.

[0309] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0310] The present invention is a system that includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, and an emotion engine for recognizing the user's emotions. The purpose is for each element to work together to assist emergency medical procedures and effectively support patient life-saving efforts.

[0311] Natural Language Processing Implementation

[0312] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question, generate a response such as, "The patient's heart rate is normal," and convey this to the user via voice or text via the terminal.

[0313] Image Recognition Implementation

[0314] Image recognition is used to analyze image data acquired from cameras and sensors and diagnose the patient's condition. For example, the device takes an image of the patient, and the server analyzes the image using image recognition technology. Based on the results of this analysis, the user is notified of the diagnosis and necessary treatment methods visually and audibly.

[0315] Implementing voice guidance

[0316] The voice guidance means guides the user through emergency medical procedures by voice. For example, the server generates cardiopulmonary resuscitation procedures and issues specific instructions such as "Please press on the chest in a steady rhythm" to the user via the terminal.

[0317] Implementation of artificial intelligence for medical procedure assistance

[0318] AI tools utilize machine learning and deep learning to have the knowledge and judgment capabilities necessary to assist medical procedures. The server collects patient data in real time, uses AI models to determine the optimal treatment method, and communicates that treatment method to the user via the device.

[0319] Sensor Implementation

[0320] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the data and issues appropriate instructions if an abnormality is detected.

[0321] Implementing Remote Assistance

[0322] The remote assistance means has a function for connecting with a medical professional in real time and receiving guidance. For example, the terminal establishes a connection with the professional and allows the professional to check the patient's condition in real time through a server, thereby transmitting instructions from the professional to the user.

[0323] Implementing encryption and access control

[0324] Encryption and access control measures are techniques for ensuring the security of patient data. Patient data collected by devices is encrypted before being sent to a server, which then securely stores the encrypted data and ensures that only authorized users can access it.

[0325] Implementing the Emotion Engine

[0326] The emotion engine analyzes the user's tone of voice and facial expressions to recognize the user's emotional state. It can use the emotion engine to assess the user's stress level and adjust responses accordingly. For example, if the server determines that the user is nervous, it can adjust its instructions to use gentler language and a more soothing tone.

[0327] Specific examples

[0328] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[0329] 1. Sensor detection

[0330] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[0331] 2. CPR instructions

[0332] Server: Uses image recognition technology to check the patient's condition and determine that they have undergone cardiac arrest.

[0333] Terminal: Based on instructions from the server, the terminal provides voice and on-screen guidance on cardiopulmonary resuscitation procedures to the user.

[0334] User: Begin CPR as instructed.

[0335] 3. Providing remote assistance

[0336] Device: Connect with medical professionals and receive real-time guidance.

[0337] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[0338] User: Perform additional medical procedures under the guidance of a professional.

[0339] 4. Emotion recognition and response adjustment

[0340] Device: Analyzes the tone of the user's voice and uses an emotion engine to determine the user's state.

[0341] Server: If the user is nervous, generate instructions using words and tones to help them relax and send them to the device.

[0342] Terminal: Provides instructions to the user in a friendly tone.

[0343] User: Stay calm and perform appropriate medical procedures.

[0344] In this way, the present invention supports the implementation of appropriate medical procedures in emergencies even without specialized knowledge, and also enables flexible responses according to the user's emotional state.

[0345] The processing flow will be explained below.

[0346] Natural Language Processing Implementation

[0347] Step 1:

[0348] Terminal: The user types a question into the microphone, for example, "How is the patient doing?"

[0349] Step 2:

[0350] Terminal: Converts the voice data into text and sends it to the server.

[0351] Step 3:

[0352] Server: The received text data is input into a natural language processing model and the question content is analyzed.

[0353] Step 4:

[0354] Server: Generates appropriate responses to questions and sends the text responses to the device.

[0355] Step 5:

[0356] Terminal: Converts the received response into speech and provides it to the user. The speech response is, "The patient's heart rate is normal."

[0357] Image Recognition Implementation

[0358] Step 1:

[0359] Terminal: Captures images of the patient through a camera and sends the image data to the server.

[0360] Step 2:

[0361] Server: Inputs the received image data into an image recognition model to predict the patient's condition.

[0362] Step 3:

[0363] Server: Generates a diagnosis of the medical condition and determines the appropriate course of treatment.

[0364] Step 4:

[0365] Server: Sends instructions on how to proceed to the terminal.

[0366] Step 5:

[0367] Terminal: The user is notified of the diagnosis and treatment method visually and audibly. "The patient is unconscious. Please begin CPR." is displayed and audible.

[0368] Implementing voice guidance

[0369] Step 1:

[0370] Server: Generates appropriate medical procedures based on the patient's condition.

[0371] Step 2:

[0372] Server: Sends the generated procedure information to the terminal.

[0373] Step 3:

[0374] Device: Provides instructions to the user through a voice guidance system, such as "Compress the chest about 5 cm at a rate of 100 times per minute."

[0375] Step 4:

[0376] User: Follow the audio prompts to perform CPR.

[0377] Implementation of artificial intelligence for medical procedure assistance

[0378] Step 1:

[0379] Terminal: Collects the patient's vital data (e.g., heart rate, blood pressure) from sensors and sends it to the server.

[0380] Step 2:

[0381] Server: Inputs collected vital data into an artificial intelligence model to determine appropriate treatment.

[0382] Step 3:

[0383] Server: Based on the judgment results, generates appropriate treatment procedures and sends them to the terminal.

[0384] Step 4:

[0385] Terminal: Provides the user with visual and audio instructions for appropriate treatment. "Patient's heart rate is dropping. Prepare oxygen for inhalation."

[0386] Sensor Implementation

[0387] Step 1:

[0388] Device: Uses sensors to capture real-time vital data such as the patient's heart rate, temperature, and respiratory rate.

[0389] Step 2:

[0390] Terminal: Sends the acquired data to the server.

[0391] Step 3:

[0392] Server: Analyzes the received data and determines whether there are any abnormalities.

[0393] Step 4:

[0394] Server: If an abnormality is detected, it determines the appropriate response method and sends instruction information to the terminal.

[0395] Step 5:

[0396] Device: Notifies the user of the abnormality and how to respond. "An abnormal heart rate has been detected. Please begin CPR immediately."

[0397] Implementing Remote Assistance

[0398] Step 1:

[0399] Terminal: When an abnormality in the patient is detected, the remote assistance function is activated and sends a request to the server to establish a connection with a medical professional.

[0400] Step 2:

[0401] Server: Establishes connection with medical professional services and transmits patient status in real time.

[0402] Step 3:

[0403] Server: Receives instructions from medical professionals and sends them to the device.

[0404] Step 4:

[0405] Terminal: Provides audio and visual instructions from a specialist to the user. "Please put an oxygen mask on the patient. Detailed instructions are below."

[0406] Step 5:

[0407] User: Follow the instructions of a professional and take appropriate measures.

[0408] Implementing encryption and access control

[0409] Step 1:

[0410] Terminal: Collects patient data (e.g., vital data, diagnostic results) and encrypts the data using an encryption algorithm.

[0411] Step 2:

[0412] Device: Sends encrypted data to the server.

[0413] Step 3:

[0414] Server: Receives the encrypted data and stores it in a secure database.

[0415] Step 4:

[0416] Server: Authenticates users accessing data and enforces access control policies.

[0417] Step 5:

[0418] Server: Regularly apply security patches to keep your systems secure.

[0419] Implementing the Emotion Engine

[0420] Step 1:

[0421] Terminal: The user types a question or comment into the microphone.

[0422] Step 2:

[0423] Device: Converts voice data into text and analyzes it with an emotion engine.

[0424] Step 3:

[0425] On the device: The emotion engine analyzes the user's tone of voice and vocabulary to assess their emotional state (e.g., stress, relief).

[0426] Step 4:

[0427] Server: Adjusts the response content based on the user's emotional state and generates a text response.

[0428] Step 5:

[0429] The device converts the tailored response into speech and provides it to the user. For example, if the device determines that the user is nervous, it responds in a gentle tone, such as "Please stay calm. Everything is fine."

[0430] Step 6:

[0431] User: Follows the coordinated response and performs appropriate medical procedures in a relaxed manner.

[0432] These are the specific processing steps of a system that includes an emotion engine, which can support prompt and appropriate medical treatment in emergencies while taking into consideration the user's emotional state.

[0433] Example 2

[0434] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0435] In modern medical settings, there is a demand for providing advanced medical support to general users who lack the knowledge and skills to perform appropriate and prompt medical procedures in emergencies. Furthermore, there is a lack of technology to accurately grasp a patient's condition and provide appropriate treatment in real time. In particular, in emergencies, it is necessary to provide appropriate responses to users who are under stress or tension. Furthermore, the safe handling of patient data and real-time communication with specialists are also issues.

[0436] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting patient data and controlling access, and emotion recognition means for recognizing the user's emotional state and adjusting responses. This allows even general users to perform appropriate medical procedures in emergencies, enables real-time diagnosis and appropriate treatment instructions, and realizes flexible responses according to the user's emotional state. It also ensures safe handling of patient data and real-time communication with professionals.

[0437] "Natural language processing means" refers to technology that understands text or voice input from a user and generates an appropriate response accordingly.

[0438] "Image recognition means" is a technology that analyzes image data acquired using a camera or sensor and understands its content.

[0439] "Audio guidance means" refers to a technique for providing information and instructions to a user by voice.

[0440] "Artificial intelligence means to assist medical procedures" refers to technologies that utilize machine learning and deep learning to provide judgments and advice to assist medical procedures.

[0441] The "sensor means for monitoring the patient's condition in real time" refers to a sensor technology that measures and monitors the patient's biological information, such as heart rate, body temperature, and respiratory rate, in real time.

[0442] "Remote assistance means for connecting with medical professionals in real time" refers to technology that allows users to connect with medical professionals in real time using the Internet or other means to receive guidance and advice.

[0443] "Means for encrypting and controlling access to patient data" refers to technology that encrypts data to protect patient data and allows only authorized users to access the data.

[0444] "Emotion recognition means that recognizes the user's emotional state and adjusts the response" is a technology that analyzes the user's tone of voice and facial expressions to determine their emotional state and generates an appropriate response according to that state.

[0445] The present invention is a system for assisting emergency medical procedures, and mainly includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, and emotion recognition means for recognizing the user's emotional state and adjusting responses.

[0446] Natural Language Processing Implementation

[0447] Natural language processing means understand the text or voice input by the user and generate an appropriate response. For example, if a user inputs "How is the patient's condition?", the server will use natural language processing technology (e.g., the BERT model) to analyze the user's question and generate a response such as "The patient's heart rate is normal." The generated response is communicated to the user via the terminal in voice or text.

[0448] Image Recognition Implementation

[0449] The image recognition means analyzes image data acquired from a camera or sensor. For example, the terminal takes an image of the patient, and the server analyzes the image using image processing technology (e.g., OpenCV or TENSORFLOW (registered trademark)). Based on the results of this analysis, a diagnosis (e.g., "The patient's skin color indicates a decrease in oxygen saturation") is generated and notified to the user visually and audibly.

[0450] Implementing voice guidance

[0451] The audio guidance means provides audio guidance on the procedures for emergency medical treatment. The guidance content generated by the server (e.g., "Please press on the chest in a steady rhythm") is transmitted to the user via the terminal. The user can follow these instructions to perform medical treatment.

[0452] Implementation of artificial intelligence for medical procedure assistance

[0453] Artificial intelligence methods use machine learning and deep learning models (e.g., models using TensorFlow) to analyze patient data in real time and determine the optimal treatment method. For example, the server analyzes the patient's vital signs, generates specific instructions such as "put on an oxygen mask," and notifies the user via the device.

[0454] Sensor Implementation

[0455] The sensor means monitors the patient's vital signs, such as heart rate, body temperature, and respiratory rate, in real time. This monitoring data is sent from the device to the server for analysis. If an abnormality is detected, the server generates instructions such as "Your fever is continuing, so please use a cooling sheet," and notifies the user via the device.

[0456] Implementing Remote Assistance

[0457] The remote assistance means has a function for connecting with a medical professional in real time. When a user requests remote assistance, the terminal establishes a connection with the medical professional. The server notifies the medical professional of the patient's condition and provides instructions to the user via the terminal. This allows the user to perform medical procedures based on the medical professional's guidance.

[0458] Implementing encryption and access control

[0459] Encryption and access control measures are techniques for ensuring the security of patient data. The device encrypts the collected data (e.g., AES encryption) and sends it to the server. The server securely stores the encrypted data and controls access to the data so that only authorized users can access it.

[0460] Implementing the Emotion Engine

[0461] The emotion recognition means analyzes the user's tone of voice and facial expressions to determine their emotional state. An emotion engine (e.g., Hume AI) can be used to assess the user's stress level and adjust the response accordingly. For example, if the server determines that the user is nervous, it can generate a gentle instruction such as "It's okay, please calm down" and send it to the device.

[0462] Specific examples

[0463] In the event of sudden cardiac arrest in a public place, the system operates as follows:

[0464] 1. Sensor detection

[0465] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[0466] 2. CPR instructions

[0467] Server: Uses image recognition technology to check the patient's condition, determines whether they have cardiac arrest, and generates audio guidance content.

[0468] Device: Provides voice and on-screen guidance to the user on how to perform CPR.

[0469] User: Begin CPR as instructed.

[0470] 3. Providing remote assistance

[0471] Device: Connect with medical professionals and receive real-time guidance.

[0472] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[0473] User: Perform additional medical procedures under the guidance of a professional.

[0474] 4. Emotion recognition and response adjustment

[0475] Device: Analyzes the tone of the user's voice and uses an emotion engine to determine the user's state.

[0476] Server: If the user is nervous, generate instructions using words and tones to help them relax and send them to the device.

[0477] Terminal: Provides instructions to the user in a friendly tone.

[0478] User: Stay calm and perform appropriate medical procedures.

[0479] Prompt Sentence Examples

[0480] "What are the specific medical steps a user should take in the event of cardiac arrest?"

[0481] "Please explain what to do if someone experiences sudden cardiac arrest in a public place."

[0482] "Please explain with a specific example the role of the emotion engine when detecting a patient's emergency situation."

[0483] In this way, the system supports general users in providing appropriate medical care in emergencies, enabling real-time diagnosis and treatment instructions, and enabling flexible responses according to the user's emotional state.It also ensures secure management of patient data and real-time communication with medical professionals.

[0484] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0485] Step 1:

[0486] User: Type the question into the terminal: "How is the patient doing?"

[0487] Input: User question (text or voice)

[0488] Output: Question text or audio data

[0489] Specific operations: The device prepares text or voice data to send the user's question to the server.

[0490] Step 2:

[0491] Terminal: Sends the user's query data to the server.

[0492] Input: Question text or audio data

[0493] Output: Question data is sent to the server

[0494] Specific operation: The terminal sends data to the server via the network.

[0495] Step 3:

[0496] Server: Uses natural language processing techniques to analyze the question and generate an appropriate response.

[0497] Input: Question text data

[0498] Output: Response text data

[0499] What happens: The server uses a natural language processing model (e.g., BERT) to analyze the question and generate an appropriate response.

[0500] Step 4:

[0501] Terminal: Receives response data from the server and conveys it to the user via voice or text.

[0502] Input: Response text data

[0503] Output: Spoken response or displayed text

[0504] Specific operation: The device converts the received text data into speech and conveys it to the user, or displays it as text on the screen.

[0505] Step 5:

[0506] User: Takes images to check the patient's condition.

[0507] Input: Patient images

[0508] Output: Captured image data

[0509] What it does: Takes an image of the patient using the device's camera.

[0510] Step 6:

[0511] Terminal: Sends the captured image data to the server.

[0512] Input: Image data

[0513] Output: Image data sent to the server

[0514] Specific operation: The terminal sends image data to the server via the network.

[0515] Step 7:

[0516] Server: Analyzes images using image recognition technology and generates diagnostic results.

[0517] Input: Image data

[0518] Output: Diagnostic result text data

[0519] Specific operation: The server analyzes image data using an image recognition model (e.g., OpenCV, TensorFlow) and generates a diagnosis result.

[0520] Step 8:

[0521] Terminal: Receives the diagnostic results from the server and notifies the user.

[0522] Input: Diagnostic result text data

[0523] Output: Audio or displayed text of diagnostic results

[0524] Specific operation: The device converts the received diagnostic results into voice and tells the user, or displays them as text on the screen.

[0525] Step 9:

[0526] Server: Generates guidance content for emergency medical treatment.

[0527] Input: Diagnostic result text data

[0528] Output: Guidance text data

[0529] Specific operation: The server generates appropriate guidance content based on the diagnosis results (e.g., "Please push on the chest in a steady rhythm").

[0530] Step 10:

[0531] Terminal: Receives guidance data from the server and conveys it to the user aloud.

[0532] Input: Guidance text data

[0533] Output: Voice guidance

[0534] Specific operation: The device converts the received guidance data into voice and conveys it to the user.

[0535] Step 11:

[0536] User: Follows the audio guidance to perform medical procedures.

[0537] Input: Voice guidance

[0538] Output: Medical procedure performed

[0539] Specific actions: The user follows the voice guidance and performs specific medical procedures.

[0540] Step 12:

[0541] Sensors: Collect patient vital data in real time.

[0542] Input: Patient vital data (heart rate, temperature, respiratory rate, etc.)

[0543] Output: Collected vital data

[0544] Specific operation: The sensor measures the patient's vital signs in real time and transmits them to the device.

[0545] Step 13:

[0546] Terminal: Sends vital data collected from sensors to the server.

[0547] Input: Collected vital data

[0548] Output: Vital data sent to the server

[0549] Specific operation: The device sends vital data to the server via the network.

[0550] Step 14:

[0551] Server: Analyzes vital data in real time and generates appropriate instructions if an abnormality is detected.

[0552] Input: Vital data

[0553] Output: Anomaly detection instruction text data

[0554] How it works: The server uses an AI model to analyze vital data and, if it detects an abnormality, generates instructions such as "use a cooling sheet."

[0555] Step 15:

[0556] Terminal: Receives instructions from the server and notifies the user.

[0557] Input: Instruction text data

[0558] Output: Spoken instructions or displayed text

[0559] Specific operation: The device converts the received instruction into voice and conveys it to the user, or displays it as text on the screen.

[0560] Step 16:

[0561] User: Follows the instructions of the server to perform appropriate medical procedures.

[0562] Input: Spoken instructions or displayed text

[0563] Output: Medical procedure performed

[0564] Specific operation: The user follows instructions from the server and performs appropriate medical procedures.

[0565] Step 17:

[0566] Device: Request remote assistance.

[0567] Input: Remote Assistance Request

[0568] Output: Request data to the server

[0569] Specific operation: The device sends a request for remote assistance to the server.

[0570] Step 18:

[0571] Server: Establishes a connection with medical professionals and communicates the patient's status.

[0572] Input: Request data, patient status data

[0573] Output: Status information for experts

[0574] Specific operation: The server connects with medical professionals in real time and relays the patient's status received from the device.

[0575] Step 19:

[0576] Expert: Checks the patient's status and sends instructions to the server.

[0577] Input: Patient status information

[0578] Output: Expert instructions

[0579] Specific operation: The expert analyzes the patient's condition and sends appropriate instructions to the server.

[0580] Step 20:

[0581] Server: Transmits the expert's instructions to the device.

[0582] Input: Expert instructions

[0583] Output: Instruction text data

[0584] Specific operation: The server sends the received instructions to the terminal.

[0585] Step 21:

[0586] Terminal: Receives instructions from the expert and notifies the user.

[0587] Input: Instruction text data

[0588] Output: Spoken instructions or displayed text

[0589] Specific operation: The device conveys the received instructions to the user via voice or text.

[0590] Step 22:

[0591] User: Act as directed by the expert.

[0592] Input: Spoken instructions or displayed text

[0593] Output: Actions to be performed

[0594] Specific Action: The user takes the necessary action based on the expert's instructions.

[0595] Step 23:

[0596] Device: Analyzes the user's tone of voice and facial expressions to determine their emotional state.

[0597] Input: User's tone of voice, facial expression data

[0598] Output: Emotional state assessment data

[0599] Specific operation: The device analyzes the user's tone of voice and facial expressions, and determines their emotional state using an emotion engine.

[0600] Step 24:

[0601] Server: Adjusts responses and generates instructions based on the emotional state.

[0602] Input: Emotional state assessment data

[0603] Output: Adjusted instruction text data

[0604] Specific behavior: The server uses an emotion engine to generate instructions based on the user's emotional state, including gentle tones and relaxing words.

[0605] Step 25:

[0606] Terminal: Receives coordinated instructions from the server and notifies the user.

[0607] Input: Adjusted instruction text data

[0608] Output: Spoken or displayed text of tailored instructions

[0609] What it does: The device provides tailored instructions to the user via voice or text to help them stay calm.

[0610] (Application example 2)

[0611] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0612] In the case of medical emergencies, it is difficult to take immediate and appropriate action. This is especially true for ordinary people without specialized knowledge who have difficulty performing accurate medical procedures in emergencies. Furthermore, existing systems do not adequately support making appropriate decisions based on the situation with the assistance of medical professionals in real time. Furthermore, there is a lack of flexible and interactive systems to respond to unexpected situations, and systems often cannot respond based on the user's emotional state. This raises the risk that users who feel nervous or anxious may not be able to take appropriate action, resulting in a worsening of the patient's condition.

[0613] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0614] In this invention, the server includes natural language processing means, image recognition means, and voice guidance means. This enables the server to accurately diagnose a patient's condition based on image data and provide appropriate voice guidance while engaging in natural communication with the user. The server also includes artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, and remote assistance means for connecting with medical professionals in real time. This provides strong support for the user in performing appropriate medical procedures even in emergencies, allowing the user to receive expert guidance in real time. The server also includes means for encrypting and controlling access to patient data, an emotion engine for recognizing the user's emotional state and adjusting communication accordingly, and a smartphone application for providing the user with assessment results and instructions in real time. This enables the user to remain calm and take appropriate action even in tense situations while ensuring the safety and privacy of patient data.

[0615] "Natural language processing means" is a technology that analyzes and understands the natural language input by the user and generates an appropriate response.

[0616] "Image recognition means" is a technology that analyzes image data acquired using a camera or sensor and recognizes objects or people.

[0617] "Audio guidance means" refers to a technique for generating audio to provide instructions and guidance to the user.

[0618] "Artificial intelligence means for assisting medical procedures" refers to technologies that utilize machine learning and deep learning to support medical procedures and determine the most appropriate treatment.

[0619] The "sensor means for monitoring the patient's condition in real time" refers to technology including sensors for measuring and monitoring biological information such as heart rate, body temperature, and respiratory rate in real time.

[0620] "Remote assistance means for connecting with medical professionals in real time" refers to technology that allows a person to connect with a medical professional in real time via the Internet and receive guidance and support from a remote location.

[0621] "Patient data encryption and access control measures" refers to technology that securely protects patient data and ensures that only authorized users can access it.

[0622] The "emotion engine" is a technology that has the function of analyzing the user's tone of voice and facial expressions to recognize the user's emotional state.

[0623] A "smartphone application" is software that runs on a smartphone and provides the various functions mentioned above in an integrated manner.

[0624] This invention is a system for supporting rapid and appropriate responses in medical emergencies, and includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring patient conditions in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, an emotion engine for recognizing the user's emotional state, and a smartphone application. Specific implementation examples of each means are shown below.

[0625] Natural language processing tools

[0626] The server uses natural language processing models from the Transformers library to understand the user's question and generate an appropriate response. For example, if a user asks, "How is the patient doing?", the server uses natural language processing to generate the response, "The patient's heart rate is normal."

[0627] Image Recognition Method

[0628] The server uses OpenCV and MediaPipe to analyze image data sent from the device and detect abnormalities in the patient's face or body. For example, when a user takes a picture of the patient's face with the device's camera, the image recognition means analyzes the image and notifies the user that "the patient's facial information has been confirmed."

[0629] Audio guidance means

[0630] The server provides audio guidance to the user on emergency medical procedures. Using the PyAudio library, the user receives audio guidance through their device. For example, specific instructions such as "Push on the chest in a steady rhythm" are provided.

[0631] Artificial intelligence methods for assisting medical procedures

[0632] The server utilizes machine learning and deep learning models to collect patient data in real time and suggest optimal medical treatments, enabling users to take appropriate medical action without specialized knowledge.

[0633] Sensor means

[0634] The device uses sensors to monitor vital signs such as heart rate, body temperature, and respiratory rate in real time, and transmits this data to a server that can then provide appropriate instructions if an abnormality is detected.

[0635] Remote Assistance Methods

[0636] The device can connect to a medical professional via the Internet and receive real-time support, for example, by establishing a connection with the professional and providing feedback of the medical professional's instructions to the user.

[0637] Encryption and Access Control Measures

[0638] The server uses the Fernet class from the cryptography library to encrypt and securely store patient data, ensuring data privacy by allowing only authorized users to access it.

[0639] Emotion Engine

[0640] The server uses the TextBlob library to analyze the user's emotional state and uses gentler words if the user is nervous. For example, if the user says, "I'm very nervous," the server determines, "The user is nervous. Please use gentler words," and adjusts its response accordingly.

[0641] Smartphone application

[0642] The device integrates these functions and provides them as a smartphone application, providing users with real-time judgment results and instructions to help them take appropriate action.

[0643] Examples:

[0644] 1. Natural Language Processing

[0645] User says: "How is the patient doing?"

[0646] Prompt: How is the patient doing?

[0647] 2. Image Recognition

[0648] User action: Take a picture of the patient's face with the camera

[0649] Prompt: Based on the patient's current image, please diagnose their condition.

[0650] 3. Audio guidance

[0651] User action: Listen to the audio guide and take emergency measures

[0652] Prompt: Push rhythmically against the chest.

[0653] 4. Emotion Engine

[0654] User says: I'm so nervous.

[0655] Prompt: Rate the user's emotional state.

[0656] In this way, by linking each means, the system becomes one that allows users to respond effectively even in emergencies.

[0657] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0658] Step 1:

[0659] The server receives questions in natural language from the user. The input is the user's speech, and the output is text data. Using natural language processing means, the server analyzes this text data and understands the intent of the question. For example, in response to the question, "How is the patient's condition?", the server generates a response such as, "The patient's heart rate is normal."

[0660] Step 2:

[0661] The device takes an image of the patient with a camera and sends the image data to the server. The input is the camera image, and the output is the image data sent to the server. The image recognition means analyzes the image data using OpenCV and MediaPipe to detect abnormalities in the patient's face or body. For example, it generates a diagnosis result such as "The patient's facial information has been confirmed."

[0662] Step 3:

[0663] The server uses the audio guidance means to guide the user through the steps of the medical procedure by voice. The input is instructions for appropriate medical procedures, and the output is audio data. The PyAudio library is used to generate audio and provide it to the user. For example, specific instructions such as "Please press on the chest in a steady rhythm" are provided by voice.

[0664] Step 4:

[0665] The terminal uses sensors to monitor the patient's vital signs, such as heart rate, body temperature, and respiratory rate, in real time, and transmits the data to a server. The input is sensor data, and the output is the vital signs transmitted to the server. The server analyzes this data and, if an abnormality is detected, instructs the appropriate countermeasures.

[0666] Step 5:

[0667] The server uses a machine learning model to suggest optimal medical treatment based on patient data. The input is real-time patient data, and the output is a recommendation for the optimal treatment method. This recommendation is then notified to the user via a smartphone application.

[0668] Step 6:

[0669] The device connects to medical professionals in real time via the Internet and receives support. The input is a connection request to a medical professional, and the output is the establishment of the connection. The server relays this connection and provides instructions from the professional to the user in real time.

[0670] Step 7:

[0671] The server uses a cryptography library to encrypt and securely store patient data. The input is patient data, and the output is encrypted data. Access control is performed so that only authorized users can access the data.

[0672] Step 8:

[0673] The server uses the TextBlob library to analyze the user's speech and recognize their emotional state. The input is the user's speech text, and the output is an evaluation of their emotional state. For example, if the user says, "I'm very nervous," the server determines that the user is nervous. Please use gentler words, and adjusts the tone of the response accordingly.

[0674] Step 9:

[0675] The device integrates the information generated at each step and provides real-time feedback to the user through a smartphone application. The input is diagnosis results and instructions from the server, and the output is notifications to the user. For example, it displays image recognition results, voice guidance, suggested medical procedures, instructions from specialists, and other information to help the user respond appropriately.

[0676] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0677] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0678] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0679] [Second embodiment]

[0680] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0681] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0682] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0683] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0684] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0685] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0686] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0687] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0688] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0689] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0690] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0691] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0692] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element works together to assist in emergency medical treatment. Specific embodiments for carrying out the present invention will be described below.

[0693] Natural Language Processing Implementation

[0694] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question and generate a response such as, "The patient's heart rate is normal." The response is conveyed to the user via the terminal in voice or text.

[0695] Image Recognition Implementation

[0696] Image data acquired from cameras and sensors is analyzed using image recognition techniques to diagnose a patient's condition. For example, a device takes an image of a patient, and the server analyzes the image using image recognition technology to predict the patient's condition from the patient's facial expression and appearance. The diagnosis result is notified to the user visually and audibly.

[0697] Implementing voice guidance

[0698] The voice guidance means guides the user through emergency medical procedures by voice. For example, if cardiopulmonary resuscitation is required, the server issues specific instructions such as "Please press on the chest in a steady rhythm" through the voice guidance program and conveys this to the user via the terminal.

[0699] Implementation of artificial intelligence for medical procedure assistance

[0700] AI tools utilize machine learning and deep learning to have the knowledge and judgment necessary to assist medical procedures. The server collects patient data in real time and uses AI models to determine the optimal treatment method. The device then instructs the user on the treatment method and encourages appropriate action.

[0701] Sensor Implementation

[0702] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the collected data and immediately generates appropriate instructions if an abnormality is detected.

[0703] Implementing Remote Assistance

[0704] The remote assistance method allows a user to connect with a medical professional in real time and receive guidance. For example, a device can establish a connection and allow the medical professional to check the patient's condition in real time. The server receives instructions from the professional and relays them to the user via the device. This allows for the management of acute symptoms.

[0705] Implementing encryption and access control

[0706] Encryption and access control are technologies used to ensure the safety of patient data. Patient data collected by the device is encrypted before being sent to the server. The server securely stores the encrypted data and ensures that only authorized users can access the data based on the access control policy. Security patches are also regularly applied to maintain the safety of the system.

[0707] Specific examples

[0708] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[0709] 1. Sensor detection

[0710] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[0711] 2. CPR instructions

[0712] Server: Uses image recognition technology to check the patient's condition and determine that they have undergone cardiac arrest.

[0713] Terminal: Based on instructions from the server, the terminal provides voice and on-screen guidance on CPR procedures to the user.

[0714] User: Begin CPR as instructed.

[0715] 3. Providing remote assistance

[0716] Device: Connect with medical professionals and receive real-time guidance.

[0717] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[0718] User: Perform additional medical procedures under the guidance of a professional.

[0719] 4. Ensuring safety

[0720] Terminal: Patient data is encrypted and securely transmitted to the server.

[0721] Server: Encrypts and stores data and applies access controls.

[0722] In this way, the present invention aims to support the implementation of appropriate medical procedures in emergencies even without specialized knowledge, thereby reducing the risk of sudden death.

[0723] The processing flow will be explained below.

[0724] Natural Language Processing Implementation

[0725] Step 1:

[0726] Terminal: The user types a question into the microphone, for example, "How is the patient doing?"

[0727] Step 2:

[0728] Terminal: Converts voice data into text and sends it to the server.

[0729] Step 3:

[0730] Server: The received text data is input into a natural language processing model and the question content is analyzed.

[0731] Step 4:

[0732] Server: Generates appropriate responses to questions and sends them to the terminal in text format.

[0733] Step 5:

[0734] Terminal: Converts the received response into speech and provides it to the user. The speech response is, "The patient's heart rate is normal."

[0735] Image Recognition Implementation

[0736] Step 1:

[0737] Terminal: Captures images of the patient through a camera and sends the image data to the server.

[0738] Step 2:

[0739] Server: Inputs the received image data into an image recognition model to predict the patient's condition.

[0740] Step 3:

[0741] Server: Generates a diagnosis of the medical condition and determines the appropriate course of treatment.

[0742] Step 4:

[0743] Server: Sends instructions on how to proceed to the terminal.

[0744] Step 5:

[0745] Terminal: The user is notified of the diagnosis and treatment method visually and audibly. "The patient is unconscious. Please begin CPR." is displayed and audible.

[0746] Implementing voice guidance

[0747] Step 1:

[0748] Server: Generates appropriate medical procedures based on the patient's condition.

[0749] Step 2:

[0750] Server: Sends the generated procedure information to the terminal.

[0751] Step 3:

[0752] Device: Provides instructions to the user through a voice guidance system, such as "Compress the chest about 5 cm at a rate of 100 times per minute."

[0753] Step 4:

[0754] User: Follow the audio prompts to perform CPR.

[0755] Implementation of artificial intelligence for medical procedure assistance

[0756] Step 1:

[0757] Terminal: Collects the patient's vital data (e.g., heart rate, blood pressure) from sensors and sends it to the server.

[0758] Step 2:

[0759] Server: Inputs collected vital data into an artificial intelligence model to determine appropriate treatment.

[0760] Step 3:

[0761] Server: Based on the judgment results, generates appropriate treatment procedures and sends them to the terminal.

[0762] Step 4:

[0763] Terminal: Provides the user with visual and audio instructions for appropriate treatment. "Patient's heart rate is dropping. Prepare oxygen for inhalation."

[0764] Sensor Implementation

[0765] Step 1:

[0766] Device: Uses sensors to capture real-time vital data such as the patient's heart rate, temperature, and respiratory rate.

[0767] Step 2:

[0768] Terminal: Sends the acquired data to the server.

[0769] Step 3:

[0770] Server: Analyzes the received data and determines whether there are any abnormalities.

[0771] Step 4:

[0772] Server: If an abnormality is detected, it determines the appropriate response method and sends instruction information to the terminal.

[0773] Step 5:

[0774] Device: Notifies the user of the abnormality and how to respond. "An abnormal heart rate has been detected. Please begin CPR immediately."

[0775] Implementing Remote Assistance

[0776] Step 1:

[0777] Terminal: When an abnormality in the patient is detected, the remote assistance function is activated and sends a request to the server to establish a connection with a medical professional.

[0778] Step 2:

[0779] Server: Establishes connection with medical professional services and transmits patient status in real time.

[0780] Step 3:

[0781] Server: Receives instructions from medical professionals and sends them to the device.

[0782] Step 4:

[0783] Terminal: Provides audio and visual instructions to the user from a specialist. "Please put an oxygen mask on the patient. Detailed instructions are below."

[0784] Step 5:

[0785] User: Follow the instructions of a professional and take appropriate measures.

[0786] Implementing encryption and access control

[0787] Step 1:

[0788] Terminal: Collects patient data (e.g., vital data, diagnostic results) and encrypts the data using an encryption algorithm.

[0789] Step 2:

[0790] Device: Sends encrypted data to the server.

[0791] Step 3:

[0792] Server: Receives the encrypted data and stores it in a secure database.

[0793] Step 4:

[0794] Server: Authenticates users accessing data and enforces access control policies.

[0795] Step 5:

[0796] Server: Regularly apply security patches to keep your systems secure.

[0797] The above is the flow of specific processing steps for each function, which will realize a system that supports appropriate responses at the scene of emergency medical care, even without specialized knowledge.

[0798] Example 1

[0799] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0800] In modern society, rapid and accurate responses are required in emergency medical situations, but appropriate treatment is often not provided due to the absence of experts or lack of knowledge. Furthermore, in order to accurately grasp the patient's condition and provide the necessary treatment, a wide range of technologies and knowledge must be integrated. In particular, the lack of real-time monitoring of patient conditions, accurate diagnosis of illness, and appropriate medical guidance is a problem.

[0801] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0802] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for analyzing user questions and generating appropriate responses, means for analyzing image data acquired by a camera and predicting the patient's condition, means for providing audio and visual guidance for medical procedure procedures, and means for encrypting and storing collected patient data, thereby enabling prompt and appropriate responses in emergency medical situations.

[0803] "Natural language processing means" refers to means for communicating with users in natural language and for understanding and analyzing questions and instructions.

[0804] "Image recognition means" is a means for analyzing image data acquired by a camera or sensor and identifying the state and characteristics of an object.

[0805] The "audio guidance means" is a means for providing instructions and information to the user by voice.

[0806] "Artificial intelligence means" refers to means that use machine learning and deep learning to assist medical procedures and determine the most appropriate treatment methods.

[0807] The "sensor means" is a means for monitoring the physical condition of a patient in real time and collecting the data.

[0808] "Remote assistance means" means a means of connecting with a medical professional in real time to receive specialized guidance and support.

[0809] "Encryption and access control means" means means for encrypting patient data and ensuring that only authorized users can access the data.

[0810] The "means for analyzing questions and generating responses" refers to means for analyzing voice or text questions entered by a user and generating appropriate responses.

[0811] The "means for analyzing image data and predicting the condition" is a means for analyzing images of a patient taken by a camera and predicting the condition of the patient.

[0812] The "means for providing guidance on the procedure of medical treatment" is a means for providing the user with audio and visual guidance on the specific procedure of emergency medical treatment.

[0813] "Means for encrypting and storing data" refers to means for encrypting collected patient data and storing it securely.

[0814] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element cooperates to assist emergency medical treatment. The embodiments of the present invention will be described in detail below.

[0815] Natural language processing implementation:

[0816] The server uses natural language processing to analyze questions or instructions entered by the user via voice or text and generate an appropriate response. For example, if a user asks, "How is the patient's condition?", the server uses natural language processing technology to understand the question and generates a response such as, "The patient's heart rate is normal." This response is then communicated to the user via the terminal via voice or text.

[0817] Image recognition implementation:

[0818] To understand the patient's condition, the device sends image data captured by the camera to a server. The server then analyzes the image data using image recognition tools and predicts the patient's condition based on the patient's facial expression and appearance. The diagnosis results are notified to the user visually and audibly. For example, the server checks for abnormalities in the patient's complexion or the presence of external injuries.

[0819] Implementing audio guidance methods:

[0820] The server uses the voice guidance means to guide the user through the procedures for emergency medical treatment. For example, if cardiopulmonary resuscitation is required, specific instructions such as "Please push on the chest in a steady rhythm" are provided by voice. These instructions are transmitted to the user through the terminal.

[0821] Implementation of artificial intelligence means:

[0822] The server uses machine learning and deep learning techniques to analyze patient data in real time and determine the optimal treatment. The device then instructs the user on this treatment and encourages appropriate medical intervention. For example, an artificial intelligence model can be used to detect abnormal heart rates or respiratory arrest and generate instructions for immediate response.

[0823] Sensor means implementation:

[0824] The device has sensors that monitor the patient's heart rate, body temperature, respiratory rate, and other data in real time. This data is sent to a server, which analyzes the data and immediately generates instructions if an abnormality is detected. For example, if a sudden change in heart rate is detected, it will immediately issue instructions for CPR.

[0825] Implementing remote assistance methods:

[0826] The terminal has a remote assistance means to connect to a medical professional in real time and receive appropriate guidance. The medical professional checks the patient's condition and provides necessary guidance. The server receives the expert's instructions and relays them to the user, allowing the user to perform accurate treatment based on the expert's guidance.

[0827] Implementing encryption and access control measures:

[0828] Patient data collected by the device is encrypted before being sent to the server, where it is stored and access is controlled to ensure only authorized users can access it. Security patches are also regularly applied to maintain safety.

[0829] Examples:

[0830] As a concrete example, consider a situation where a cardiac arrest occurs in a public place. In this case, the system operates as follows:

[0831] 1. A user uses a device to ask, "How is the patient doing?"

[0832] 2. The device takes an image of the patient with its camera and sends it to the server.

[0833] 3. The server uses image recognition to diagnose the patient's condition and determine that cardiopulmonary resuscitation is necessary.

[0834] 4. The server generates instructions using voice guidance means, such as "Please press the chest at a regular rhythm," and conveys this to the user via the terminal.

[0835] 5. Use remote assistance methods to connect with medical professionals for additional guidance in real time.

[0836] 6. The data collected by the device is encrypted and securely sent to the server.

[0837] An example of a prompt sentence could be something like, "Please explain how the system would respond if someone had a sudden cardiac arrest in the park."

[0838] This enables the system to respond quickly and accurately in emergencies and support appropriate medical treatment.

[0839] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0840] Step 1:

[0841] The user enters a question.

[0842] Specific behavior:

[0843] User: Ask via voice or text, "How is the patient doing?"

[0844] Device: Converts speech to text and sends the data to the server.

[0845] Input: User's voice or text question.

[0846] Output: Question data in text format.

[0847] Step 2:

[0848] The server analyzes the user's question and generates a response.

[0849] Specific behavior:

[0850] Server: Uses natural language processing techniques to analyze the user's question and generate an appropriate response.

[0851] Server: Generates a response, for example, "The patient's heart rate is normal."

[0852] Input: The question data submitted by the user in text format.

[0853] Output: The response data in text format.

[0854] Step 3:

[0855] The user is notified of the response from the server.

[0856] Specific behavior:

[0857] Terminal: The response data from the server is converted into voice using speech synthesis technology and notified to the user.

[0858] Terminal: Also displayed as text data.

[0859] Input: Response data sent from the server in text format.

[0860] Output: Voice and text response notification.

[0861] Step 4:

[0862] Image data is acquired to confirm the patient's condition.

[0863] Specific behavior:

[0864] Device: Takes images of the patient's face and body using a camera.

[0865] Terminal: Sends the captured image data to the server.

[0866] Input: Patient images.

[0867] Output: Image data sent to the server.

[0868] Step 5:

[0869] The server analyzes the image data and predicts the condition of the disease.

[0870] Specific behavior:

[0871] Server: Uses image recognition technology to analyze a patient's facial expression, appearance, and possible injuries to predict their condition.

[0872] Server: Generates an analysis result, for example, "The patient may be unconscious."

[0873] Input: Image data sent from the device.

[0874] Output: Predicted disease state.

[0875] Step 6:

[0876] The prediction results are notified to the user.

[0877] Specific behavior:

[0878] Terminal: Notifies the user of predicted medical conditions via voice or text.

[0879] Input: Prediction results of the disease condition sent from the server.

[0880] Output: Prediction result notification in audio and text format.

[0881] Step 7:

[0882] Provide guidance on emergency medical procedures.

[0883] Specific behavior:

[0884] User: Ask "What do I do?"

[0885] Server: Analyzes the situation and determines the medical intervention required.

[0886] Server: For example, it determines that cardiopulmonary resuscitation is necessary and generates instructions such as "Push on the chest in a steady rhythm."

[0887] Terminal: Provides audio and visual instructions to the user.

[0888] Input: Questions from the user and patient data.

[0889] Output: Medical procedure instructions in audio and visual format.

[0890] Step 8:

[0891] Offer remote assistance.

[0892] Specific behavior:

[0893] Device: Establish a connection with a medical professional and receive guidance in real time.

[0894] Server: Receives expert instructions and conveys them to the user in an appropriate manner.

[0895] User: Follow the guidance of a professional and perform appropriate medical procedures.

[0896] Input: Patient status and professional instructions.

[0897] Output: Real-time expert feedback.

[0898] Step 9:

[0899] The patient data is encrypted and sent to the server.

[0900] Specific behavior:

[0901] Terminal: Encrypts patient data (heart rate, images, prediction results, etc.).

[0902] Device: Sends encrypted data to the server.

[0903] Input: Patient data.

[0904] Output: The encrypted data.

[0905] Step 10:

[0906] The server stores encrypted data and controls access.

[0907] Specific behavior:

[0908] Server: Collected data is stored in encrypted form.

[0909] Server: Ensure that only authenticated users can access the data.

[0910] Servers: Regularly apply security patches to maintain security.

[0911] Input: Encrypted patient data.

[0912] Output: Stored data and access logs.

[0913] (Application example 1)

[0914] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0915] Conventional emergency medical intervention systems have limitations in their ability to quickly reach the scene and accurately grasp the patient's condition in real time. Systems utilizing autonomous vehicles, in particular, lack the advanced technology necessary to provide appropriate medical intervention in an emergency. Another problem is the difficulty of providing accurate instructions while connecting with medical professionals in real time. The present invention aims to solve these problems and provide an advanced emergency medical intervention support system using autonomous vehicles to reduce the risk of sudden patient death.

[0916] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0917] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for automatically arriving at the emergency scene, means for diagnosing the patient's condition using an in-vehicle camera, and means for assisting medical procedures in real time within the autonomous vehicle, thereby enabling rapid arrival at the emergency scene and accurate medical procedures and real-time collaboration with medical professionals at the scene.

[0918] "Natural language processing means" refers to means that enables communication with the user in natural language.

[0919] "Image recognition means" refers to a means for analyzing image data acquired from a camera or sensor and diagnosing the patient's condition.

[0920] The "audio guidance means" is a means for providing the user with audio guidance on the specific steps of a medical procedure.

[0921] "Artificial intelligence means to assist medical procedures" refers to means that utilize machine learning and deep learning to acquire the knowledge and judgment capabilities necessary for medical procedures.

[0922] The "sensor means" is a means for monitoring the patient's physical condition, such as heart rate, body temperature, and respiratory rate, in real time.

[0923] "Remote assistance means" means a means for connecting with a medical professional in real time for guidance.

[0924] "Encryption and access control measures" means measures that encrypt patient data to ensure its security and ensure that only authorized users can access it.

[0925] "Means for arriving at an emergency scene using autonomous driving" means means that enable a vehicle to quickly arrive at an emergency scene using autonomous driving technology.

[0926] "Image recognition means for diagnosing a patient's condition using an in-vehicle camera" refers to a means for diagnosing a patient's condition by analyzing images taken by a camera installed inside an autonomous vehicle.

[0927] "Means for assisting medical procedures in real time within an autonomous vehicle" means means for assisting medical procedures in real time within an autonomous vehicle.

[0928] The present invention is an advanced emergency medical intervention assistance system using an autonomous vehicle, which integrates various technologies, including natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical intervention, sensors means, remote assistance means, encryption and access control means, means for autonomously driving to arrive at the emergency scene, image recognition means for diagnosing the patient's condition using an in-vehicle camera, and means for providing real-time medical intervention assistance within the autonomous vehicle.

[0929] Autonomous driving to emergency scene

[0930] The server receives emergency calls and uses GPS navigation technology to determine the optimal route to the scene. It uses the Google Maps API to analyze traffic conditions in real time and select the optimal route. The autonomous vehicle then automatically follows this route.

[0931] Natural language processing tools

[0932] The server uses natural language processing models (e.g., the BERT model) to communicate with emergency responders in natural language. This allows the user to ask a question such as "How is the patient doing?" and the system understands the question and generates a response such as "The patient's heart rate is normal." This response is then conveyed to the user via voice or text.

[0933] Image Recognition Method

[0934] The server analyzes image data captured by the vehicle's camera using a convolutional neural network (CNN) to diagnose the patient's condition. This data is securely transmitted to the server, and the results of the analysis are notified to the user.

[0935] Audio guidance means

[0936] The server uses a voice guidance means to provide voice instructions on specific steps of medical procedures. For example, if cardiopulmonary resuscitation is required, the server uses a technology that provides voice instructions such as "Push on the chest in a steady rhythm."

[0937] Remote Assistance Methods

[0938] The server uses a video conferencing system (e.g., Zoom API) to connect with medical professionals in the autonomous vehicle in real time, allowing the medical professionals to monitor the situation on-site and provide instructions.

[0939] Sensor means

[0940] The server uses sensors to monitor heart rate, body temperature, breathing rate, etc. in real time. This data is collected by the vehicle's in-vehicle system and transmitted to the server. If an abnormality is detected, appropriate instructions are generated immediately.

[0941] Encryption and Access Control Measures

[0942] The server uses a method for encrypting and controlling access to patient data, using AES-256 data encryption and OAuth user authentication to ensure data security.

[0943] Specific examples of emergencies

[0944] For example, consider a sudden cardiac arrest in a public place. In this case, the system operates as follows:

[0945] 1. An autonomous vehicle receives an emergency call and arrives at the scene via the optimal route.

[0946] 2. Diagnose the patient's condition using image recognition means and provide specific instructions to the user via voice guidance.

[0947] 3. An example of a prompt for reproduction is the text, "As an example of the use of an emergency medical guidance system, please tell us how to respond to a patient in cardiac arrest."

[0948] Program processing

[0949] The server connects various technologies to efficiently support emergency medical procedures. The system integrates autonomous vehicle navigation, natural language processing, image recognition, voice guidance, remote assistance, and security management to ensure a fast and accurate response in emergencies.

[0950] Operation of this system enables rapid arrival at the emergency site and appropriate medical treatment on the spot, significantly reducing the risk of sudden patient death.

[0951] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0952] Step 1:

[0953] The server receives emergency calls. It receives emergency call data (the caller's location information and the details of the emergency) as input, and calculates the optimal route as output. Data processing involves using a GPS navigation system (e.g., Google Maps API) to analyze real-time traffic information and generate the optimal route to quickly reach the scene. The server then sends the generated route information to the autonomous vehicle.

[0954] Step 2:

[0955] The autonomous vehicle (terminal) begins autonomous driving towards the site based on the route information received from the server. The input is route information, and the output is the execution of autonomous driving to the site. Specifically, the vehicle's autonomous driving system controls the vehicle according to GPS information, and drives while recognizing traffic lights and obstacles.

[0956] Step 3:

[0957] When the device arrives at the scene, it uses natural language processing to begin communicating with the user (emergency responder). The input is the user's question or instruction, and the output is a response to the user. For data processing, a natural language processing model (e.g., BERT) is used to analyze the meaning of the question and generate an appropriate response. The server transmits the generated response from the device to the user via voice or text.

[0958] Step 4:

[0959] The server uses the device's camera to diagnose the patient's condition on-site using image recognition. The input is camera image data, and the output is the diagnosis result. Data processing involves image analysis using a convolutional neural network (CNN) to diagnose the patient's external symptoms. The diagnosis result is transmitted from the server to the user via the device.

[0960] Step 5:

[0961] The terminal uses voice guidance generated by the server to instruct the user on the steps of the medical procedure. The input is the diagnosis result and the output is the voice guidance. The server executes the voice guidance program, generates specific instructions, and transmits them to the user as voice from the terminal. For example, the instruction may be "Please press on the chest at a regular rhythm."

[0962] Step 6:

[0963] The device connects to medical professionals in real time via a server to provide remote assistance. The input is a connection request, and the output is the establishment of a video call connection. For data processing, a video conferencing system (e.g., Zoom API) is used to send and receive video and audio in real time to receive instructions from medical professionals. The instructions from the professional are then communicated to the user via the device.

[0964] Step 7:

[0965] The server securely stores all collected patient data using encryption and access control measures. The input is collected data, and the output is encrypted data. Data processing involves encrypting the data using the AES-256 encryption algorithm and controlling access via OAuth. The data is securely stored on the server and can only be accessed by authorized users.

[0966] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0967] The present invention is a system that includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, and an emotion engine for recognizing the user's emotions. The purpose is for each element to work together to assist emergency medical procedures and effectively support patient life-saving efforts.

[0968] Natural Language Processing Implementation

[0969] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question, generate a response such as, "The patient's heart rate is normal," and convey this to the user via voice or text via the terminal.

[0970] Image Recognition Implementation

[0971] Image recognition is used to analyze image data acquired from cameras and sensors and diagnose the patient's condition. For example, the device takes an image of the patient, and the server analyzes the image using image recognition technology. Based on the results of this analysis, the user is notified of the diagnosis and necessary treatment methods visually and audibly.

[0972] Implementing voice guidance

[0973] The voice guidance means guides the user through emergency medical procedures by voice. For example, the server generates cardiopulmonary resuscitation procedures and issues specific instructions such as "Please press on the chest in a steady rhythm" to the user via the terminal.

[0974] Implementation of artificial intelligence for medical procedure assistance

[0975] AI tools utilize machine learning and deep learning to have the knowledge and judgment capabilities necessary to assist medical procedures. The server collects patient data in real time, uses AI models to determine the optimal treatment method, and communicates that treatment method to the user via the device.

[0976] Sensor Implementation

[0977] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the data and issues appropriate instructions if an abnormality is detected.

[0978] Implementing Remote Assistance

[0979] The remote assistance means has a function for connecting with a medical professional in real time and receiving guidance. For example, the terminal establishes a connection with the professional and allows the professional to check the patient's condition in real time through a server, thereby transmitting instructions from the professional to the user.

[0980] Implementing encryption and access control

[0981] Encryption and access control measures are techniques for ensuring the security of patient data. Patient data collected by devices is encrypted before being sent to a server, which then securely stores the encrypted data and ensures that only authorized users can access it.

[0982] Implementing the Emotion Engine

[0983] The emotion engine analyzes the user's tone of voice and facial expressions to recognize the user's emotional state. It can use the emotion engine to assess the user's stress level and adjust responses accordingly. For example, if the server determines that the user is nervous, it can adjust its instructions to use gentler language and a more soothing tone.

[0984] Specific examples

[0985] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[0986] 1. Sensor detection

[0987] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[0988] 2. CPR instructions

[0989] Server: Uses image recognition technology to check the patient's condition and determine that they have undergone cardiac arrest.

[0990] Terminal: Based on instructions from the server, the terminal provides voice and on-screen guidance on cardiopulmonary resuscitation procedures to the user.

[0991] User: Begin CPR as instructed.

[0992] 3. Providing remote assistance

[0993] Device: Connect with medical professionals and receive real-time guidance.

[0994] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[0995] User: Perform additional medical procedures under the guidance of a professional.

[0996] 4. Emotion recognition and response adjustment

[0997] Device: Analyzes the tone of the user's voice and uses an emotion engine to determine the user's state.

[0998] Server: If the user is nervous, generate instructions using words and tones to help them relax and send them to the device.

[0999] Terminal: Provides instructions to the user in a friendly tone.

[1000] User: Stay calm and perform appropriate medical procedures.

[1001] In this way, the present invention supports the implementation of appropriate medical procedures in emergencies even without specialized knowledge, and also enables flexible responses according to the user's emotional state.

[1002] The processing flow will be explained below.

[1003] Natural Language Processing Implementation

[1004] Step 1:

[1005] Terminal: The user types a question into the microphone, for example, "How is the patient doing?"

[1006] Step 2:

[1007] Terminal: Converts the voice data into text and sends it to the server.

[1008] Step 3:

[1009] Server: The received text data is input into a natural language processing model and the question content is analyzed.

[1010] Step 4:

[1011] Server: Generates appropriate responses to questions and sends the text responses to the device.

[1012] Step 5:

[1013] Terminal: Converts the received response into speech and provides it to the user. The speech response is, "The patient's heart rate is normal."

[1014] Image Recognition Implementation

[1015] Step 1:

[1016] Terminal: Captures images of the patient through a camera and sends the image data to the server.

[1017] Step 2:

[1018] Server: Inputs the received image data into an image recognition model to predict the patient's condition.

[1019] Step 3:

[1020] Server: Generates a diagnosis of the medical condition and determines the appropriate course of treatment.

[1021] Step 4:

[1022] Server: Sends instructions on how to proceed to the terminal.

[1023] Step 5:

[1024] Terminal: The user is notified of the diagnosis and treatment method visually and audibly. "The patient is unconscious. Please begin CPR." is displayed and audible.

[1025] Implementing voice guidance

[1026] Step 1:

[1027] Server: Generates appropriate medical procedures based on the patient's condition.

[1028] Step 2:

[1029] Server: Sends the generated procedure information to the terminal.

[1030] Step 3:

[1031] Device: Provides instructions to the user through a voice guidance system, such as "Compress the chest about 5 cm at a rate of 100 times per minute."

[1032] Step 4:

[1033] User: Follow the audio prompts to perform CPR.

[1034] Implementation of artificial intelligence for medical procedure assistance

[1035] Step 1:

[1036] Terminal: Collects the patient's vital data (e.g., heart rate, blood pressure) from sensors and sends it to the server.

[1037] Step 2:

[1038] Server: Inputs collected vital data into an artificial intelligence model to determine appropriate treatment.

[1039] Step 3:

[1040] Server: Based on the judgment results, generates appropriate treatment procedures and sends them to the terminal.

[1041] Step 4:

[1042] Terminal: Provides the user with visual and audio instructions for appropriate treatment. "Patient's heart rate is dropping. Prepare oxygen for inhalation."

[1043] Sensor Implementation

[1044] Step 1:

[1045] Device: Uses sensors to capture real-time vital data such as the patient's heart rate, temperature, and respiratory rate.

[1046] Step 2:

[1047] Terminal: Sends the acquired data to the server.

[1048] Step 3:

[1049] Server: Analyzes the received data and determines whether there are any abnormalities.

[1050] Step 4:

[1051] Server: If an abnormality is detected, it determines the appropriate response method and sends instruction information to the terminal.

[1052] Step 5:

[1053] Device: Notifies the user of the abnormality and how to respond. "An abnormal heart rate has been detected. Please begin CPR immediately."

[1054] Implementing Remote Assistance

[1055] Step 1:

[1056] Terminal: When an abnormality in the patient is detected, the remote assistance function is activated and sends a request to the server to establish a connection with a medical professional.

[1057] Step 2:

[1058] Server: Establishes connection with medical professional services and transmits patient status in real time.

[1059] Step 3:

[1060] Server: Receives instructions from medical professionals and sends them to the device.

[1061] Step 4:

[1062] Terminal: Provides audio and visual instructions from a specialist to the user. "Please put an oxygen mask on the patient. Detailed instructions are below."

[1063] Step 5:

[1064] User: Follow the instructions of a professional and take appropriate measures.

[1065] Implementing encryption and access control

[1066] Step 1:

[1067] Terminal: Collects patient data (e.g., vital data, diagnostic results) and encrypts the data using an encryption algorithm.

[1068] Step 2:

[1069] Device: Sends encrypted data to the server.

[1070] Step 3:

[1071] Server: Receives the encrypted data and stores it in a secure database.

[1072] Step 4:

[1073] Server: Authenticates users accessing data and enforces access control policies.

[1074] Step 5:

[1075] Server: Regularly apply security patches to keep your systems secure.

[1076] Implementing the Emotion Engine

[1077] Step 1:

[1078] Terminal: The user types a question or comment into the microphone.

[1079] Step 2:

[1080] Device: Converts voice data into text and analyzes it with an emotion engine.

[1081] Step 3:

[1082] On the device: The emotion engine analyzes the user's tone of voice and vocabulary to assess their emotional state (e.g., stress, relief).

[1083] Step 4:

[1084] Server: Adjusts the response content based on the user's emotional state and generates a text response.

[1085] Step 5:

[1086] The device converts the tailored response into speech and provides it to the user. For example, if the device determines that the user is nervous, it responds in a gentle tone, such as "Please stay calm. Everything is fine."

[1087] Step 6:

[1088] User: Follows the coordinated response and performs appropriate medical procedures in a relaxed manner.

[1089] These are the specific processing steps of a system that includes an emotion engine, which can support prompt and appropriate medical treatment in emergencies while taking into consideration the user's emotional state.

[1090] Example 2

[1091] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1092] In modern medical settings, there is a demand for providing advanced medical support to general users who lack the knowledge and skills to perform appropriate and prompt medical procedures in emergencies. Furthermore, there is a lack of technology to accurately grasp a patient's condition and provide appropriate treatment in real time. In particular, in emergencies, it is necessary to provide appropriate responses to users who are under stress or tension. Furthermore, the safe handling of patient data and real-time communication with specialists are also issues.

[1093] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting patient data and controlling access, and emotion recognition means for recognizing the user's emotional state and adjusting responses. This allows even general users to perform appropriate medical procedures in emergencies, enables real-time diagnosis and appropriate treatment instructions, and realizes flexible responses according to the user's emotional state. It also ensures safe handling of patient data and real-time communication with professionals.

[1094] "Natural language processing means" refers to technology that understands text or voice input from a user and generates an appropriate response accordingly.

[1095] "Image recognition means" is a technology that analyzes image data acquired using a camera or sensor and understands its content.

[1096] "Audio guidance means" refers to a technique for providing information and instructions to a user by voice.

[1097] "Artificial intelligence means to assist medical procedures" refers to technologies that utilize machine learning and deep learning to provide judgments and advice to assist medical procedures.

[1098] The "sensor means for monitoring the patient's condition in real time" refers to a sensor technology that measures and monitors the patient's biological information, such as heart rate, body temperature, and respiratory rate, in real time.

[1099] "Remote assistance means for connecting with medical professionals in real time" refers to technology that allows users to connect with medical professionals in real time using the Internet or other means to receive guidance and advice.

[1100] "Means for encrypting and controlling access to patient data" refers to technology that encrypts data to protect patient data and allows only authorized users to access the data.

[1101] "Emotion recognition means that recognizes the user's emotional state and adjusts the response" is a technology that analyzes the user's tone of voice and facial expressions to determine their emotional state and generates an appropriate response according to that state.

[1102] The present invention is a system for assisting emergency medical procedures, and mainly includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, and emotion recognition means for recognizing the user's emotional state and adjusting responses.

[1103] Natural Language Processing Implementation

[1104] Natural language processing means understand the text or voice input by the user and generate an appropriate response. For example, if a user inputs "How is the patient's condition?", the server will use natural language processing technology (e.g., the BERT model) to analyze the user's question and generate a response such as "The patient's heart rate is normal." The generated response is communicated to the user via the terminal in voice or text.

[1105] Image Recognition Implementation

[1106] The image recognition means analyzes image data acquired from cameras and sensors. For example, the device takes an image of the patient, and the server analyzes the image using image processing technology (e.g., OpenCV or TensorFlow). Based on the results of this analysis, a diagnosis (e.g., "The patient's skin color indicates a decrease in oxygen saturation") is generated and notified to the user visually and audibly.

[1107] Implementing voice guidance

[1108] The audio guidance means provides audio guidance on the procedures for emergency medical treatment. The guidance content generated by the server (e.g., "Please press on the chest in a steady rhythm") is transmitted to the user via the terminal. The user can follow these instructions to perform medical treatment.

[1109] Implementation of artificial intelligence for medical procedure assistance

[1110] Artificial intelligence methods use machine learning and deep learning models (e.g., models using TensorFlow) to analyze patient data in real time and determine the optimal treatment method. For example, the server analyzes the patient's vital signs, generates specific instructions such as "put on an oxygen mask," and notifies the user via the device.

[1111] Sensor Implementation

[1112] The sensor means monitors the patient's vital signs, such as heart rate, body temperature, and respiratory rate, in real time. This monitoring data is sent from the device to the server for analysis. If an abnormality is detected, the server generates instructions such as "Your fever is continuing, so please use a cooling sheet," and notifies the user via the device.

[1113] Implementing Remote Assistance

[1114] The remote assistance means has a function for connecting with a medical professional in real time. When a user requests remote assistance, the terminal establishes a connection with the medical professional. The server notifies the medical professional of the patient's condition and provides instructions to the user via the terminal. This allows the user to perform medical procedures based on the medical professional's guidance.

[1115] Implementing encryption and access control

[1116] Encryption and access control measures are techniques for ensuring the security of patient data. The device encrypts the collected data (e.g., AES encryption) and sends it to the server. The server securely stores the encrypted data and controls access to the data so that only authorized users can access it.

[1117] Implementing the Emotion Engine

[1118] The emotion recognition means analyzes the user's tone of voice and facial expressions to determine their emotional state. An emotion engine (e.g., Hume AI) can be used to assess the user's stress level and adjust the response accordingly. For example, if the server determines that the user is nervous, it can generate a gentle instruction such as "It's okay, please calm down" and send it to the device.

[1119] Specific examples

[1120] In the event of sudden cardiac arrest in a public place, the system operates as follows:

[1121] 1. Sensor detection

[1122] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[1123] 2. CPR instructions

[1124] Server: Uses image recognition technology to check the patient's condition, determines whether they have cardiac arrest, and generates audio guidance content.

[1125] Device: Provides voice and on-screen guidance to the user on how to perform CPR.

[1126] User: Begin CPR as instructed.

[1127] 3. Providing remote assistance

[1128] Device: Connect with medical professionals and receive real-time guidance.

[1129] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[1130] User: Perform additional medical procedures under the guidance of a professional.

[1131] 4. Emotion recognition and response adjustment

[1132] Device: Analyzes the tone of the user's voice and uses an emotion engine to determine the user's state.

[1133] Server: If the user is nervous, generate instructions using words and tones to help them relax and send them to the device.

[1134] Terminal: Provides instructions to the user in a friendly tone.

[1135] User: Stay calm and perform appropriate medical procedures.

[1136] Prompt Sentence Examples

[1137] "What are the specific medical steps a user should take in the event of cardiac arrest?"

[1138] "Please explain what to do if someone experiences sudden cardiac arrest in a public place."

[1139] "Please explain with a specific example the role of the emotion engine when detecting a patient's emergency situation."

[1140] In this way, the system supports general users in providing appropriate medical care in emergencies, enabling real-time diagnosis and treatment instructions, and enabling flexible responses according to the user's emotional state.It also ensures secure management of patient data and real-time communication with medical professionals.

[1141] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1142] Step 1:

[1143] User: Type the question into the terminal: "How is the patient doing?"

[1144] Input: User question (text or voice)

[1145] Output: Question text or audio data

[1146] Specific operations: The device prepares text or voice data to send the user's question to the server.

[1147] Step 2:

[1148] Terminal: Sends the user's query data to the server.

[1149] Input: Question text or audio data

[1150] Output: Question data is sent to the server

[1151] Specific operation: The terminal sends data to the server via the network.

[1152] Step 3:

[1153] Server: Uses natural language processing techniques to analyze the question and generate an appropriate response.

[1154] Input: Question text data

[1155] Output: Response text data

[1156] What happens: The server uses a natural language processing model (e.g., BERT) to analyze the question and generate an appropriate response.

[1157] Step 4:

[1158] Terminal: Receives response data from the server and conveys it to the user via voice or text.

[1159] Input: Response text data

[1160] Output: Spoken response or displayed text

[1161] Specific operation: The device converts the received text data into speech and conveys it to the user, or displays it as text on the screen.

[1162] Step 5:

[1163] User: Takes images to check the patient's condition.

[1164] Input: Patient images

[1165] Output: Captured image data

[1166] What it does: Takes an image of the patient using the device's camera.

[1167] Step 6:

[1168] Terminal: Sends the captured image data to the server.

[1169] Input: Image data

[1170] Output: Image data sent to the server

[1171] Specific operation: The terminal sends image data to the server via the network.

[1172] Step 7:

[1173] Server: Analyzes images using image recognition technology and generates diagnostic results.

[1174] Input: Image data

[1175] Output: Diagnostic result text data

[1176] Specific operation: The server analyzes image data using an image recognition model (e.g., OpenCV, TensorFlow) and generates a diagnosis result.

[1177] Step 8:

[1178] Terminal: Receives the diagnostic results from the server and notifies the user.

[1179] Input: Diagnostic result text data

[1180] Output: Audio or displayed text of diagnostic results

[1181] Specific operation: The device converts the received diagnostic results into voice and tells the user, or displays them as text on the screen.

[1182] Step 9:

[1183] Server: Generates guidance content for emergency medical treatment.

[1184] Input: Diagnostic result text data

[1185] Output: Guidance text data

[1186] Specific operation: The server generates appropriate guidance content based on the diagnosis results (e.g., "Please push on the chest in a steady rhythm").

[1187] Step 10:

[1188] Terminal: Receives guidance data from the server and conveys it to the user aloud.

[1189] Input: Guidance text data

[1190] Output: Voice guidance

[1191] Specific operation: The device converts the received guidance data into voice and conveys it to the user.

[1192] Step 11:

[1193] User: Follows the audio guidance to perform medical procedures.

[1194] Input: Voice guidance

[1195] Output: Medical procedure performed

[1196] Specific actions: The user follows the voice guidance and performs specific medical procedures.

[1197] Step 12:

[1198] Sensors: Collect patient vital data in real time.

[1199] Input: Patient vital data (heart rate, temperature, respiratory rate, etc.)

[1200] Output: Collected vital data

[1201] Specific operation: The sensor measures the patient's vital signs in real time and transmits them to the device.

[1202] Step 13:

[1203] Terminal: Sends vital data collected from sensors to the server.

[1204] Input: Collected vital data

[1205] Output: Vital data sent to the server

[1206] Specific operation: The device sends vital data to the server via the network.

[1207] Step 14:

[1208] Server: Analyzes vital data in real time and generates appropriate instructions if an abnormality is detected.

[1209] Input: Vital data

[1210] Output: Anomaly detection instruction text data

[1211] How it works: The server uses an AI model to analyze vital data and, if it detects an abnormality, generates instructions such as "use a cooling sheet."

[1212] Step 15:

[1213] Terminal: Receives instructions from the server and notifies the user.

[1214] Input: Instruction text data

[1215] Output: Spoken instructions or displayed text

[1216] Specific operation: The device converts the received instruction into voice and conveys it to the user, or displays it as text on the screen.

[1217] Step 16:

[1218] User: Follows the instructions of the server to perform appropriate medical procedures.

[1219] Input: Spoken instructions or displayed text

[1220] Output: Medical procedure performed

[1221] Specific operation: The user follows instructions from the server and performs appropriate medical procedures.

[1222] Step 17:

[1223] Device: Request remote assistance.

[1224] Input: Remote Assistance Request

[1225] Output: Request data to the server

[1226] Specific operation: The device sends a request for remote assistance to the server.

[1227] Step 18:

[1228] Server: Establishes a connection with medical professionals and communicates the patient's status.

[1229] Input: Request data, patient status data

[1230] Output: Status information for experts

[1231] Specific operation: The server connects with medical professionals in real time and relays the patient's status received from the device.

[1232] Step 19:

[1233] Expert: Checks the patient's status and sends instructions to the server.

[1234] Input: Patient status information

[1235] Output: Expert instructions

[1236] Specific operation: The expert analyzes the patient's condition and sends appropriate instructions to the server.

[1237] Step 20:

[1238] Server: Transmits the expert's instructions to the device.

[1239] Input: Expert instructions

[1240] Output: Instruction text data

[1241] Specific operation: The server sends the received instructions to the terminal.

[1242] Step 21:

[1243] Terminal: Receives instructions from the expert and notifies the user.

[1244] Input: Instruction text data

[1245] Output: Spoken instructions or displayed text

[1246] Specific operation: The device conveys the received instructions to the user via voice or text.

[1247] Step 22:

[1248] User: Act as directed by the expert.

[1249] Input: Spoken instructions or displayed text

[1250] Output: Actions to be performed

[1251] Specific Action: The user takes the necessary action based on the expert's instructions.

[1252] Step 23:

[1253] Device: Analyzes the user's tone of voice and facial expressions to determine their emotional state.

[1254] Input: User's tone of voice, facial expression data

[1255] Output: Emotional state assessment data

[1256] Specific operation: The device analyzes the user's tone of voice and facial expressions, and determines their emotional state using an emotion engine.

[1257] Step 24:

[1258] Server: Adjusts responses and generates instructions based on the emotional state.

[1259] Input: Emotional state assessment data

[1260] Output: Adjusted instruction text data

[1261] Specific behavior: The server uses an emotion engine to generate instructions based on the user's emotional state, including gentle tones and relaxing words.

[1262] Step 25:

[1263] Terminal: Receives coordinated instructions from the server and notifies the user.

[1264] Input: Adjusted instruction text data

[1265] Output: Spoken or displayed text of tailored instructions

[1266] What it does: The device provides tailored instructions to the user via voice or text to help them stay calm.

[1267] (Application example 2)

[1268] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1269] In the case of medical emergencies, it is difficult to take immediate and appropriate action. This is especially true for ordinary people without specialized knowledge who have difficulty performing accurate medical procedures in emergencies. Furthermore, existing systems do not adequately support making appropriate decisions based on the situation with the assistance of medical professionals in real time. Furthermore, there is a lack of flexible and interactive systems to respond to unexpected situations, and systems often cannot respond based on the user's emotional state. This raises the risk that users who feel nervous or anxious may not be able to take appropriate action, resulting in a worsening of the patient's condition.

[1270] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1271] In this invention, the server includes natural language processing means, image recognition means, and voice guidance means. This enables the server to accurately diagnose a patient's condition based on image data and provide appropriate voice guidance while engaging in natural communication with the user. The server also includes artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, and remote assistance means for connecting with medical professionals in real time. This provides strong support for the user in performing appropriate medical procedures even in emergencies, allowing the user to receive expert guidance in real time. The server also includes means for encrypting and controlling access to patient data, an emotion engine for recognizing the user's emotional state and adjusting communication accordingly, and a smartphone application for providing the user with assessment results and instructions in real time. This enables the user to remain calm and take appropriate action even in tense situations while ensuring the safety and privacy of patient data.

[1272] "Natural language processing means" is a technology that analyzes and understands the natural language input by the user and generates an appropriate response.

[1273] "Image recognition means" is a technology that analyzes image data acquired using a camera or sensor and recognizes objects or people.

[1274] "Audio guidance means" refers to a technique for generating audio to provide instructions and guidance to the user.

[1275] "Artificial intelligence means for assisting medical procedures" refers to technologies that utilize machine learning and deep learning to support medical procedures and determine the most appropriate treatment.

[1276] The "sensor means for monitoring the patient's condition in real time" refers to technology including sensors for measuring and monitoring biological information such as heart rate, body temperature, and respiratory rate in real time.

[1277] "Remote assistance means for connecting with medical professionals in real time" refers to technology that allows a person to connect with a medical professional in real time via the Internet and receive guidance and support from a remote location.

[1278] "Patient data encryption and access control measures" refers to technology that securely protects patient data and ensures that only authorized users can access it.

[1279] The "emotion engine" is a technology that has the function of analyzing the user's tone of voice and facial expressions to recognize the user's emotional state.

[1280] A "smartphone application" is software that runs on a smartphone and provides the various functions mentioned above in an integrated manner.

[1281] This invention is a system for supporting rapid and appropriate responses in medical emergencies, and includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring patient conditions in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, an emotion engine for recognizing the user's emotional state, and a smartphone application. Specific implementation examples of each means are shown below.

[1282] Natural language processing tools

[1283] The server uses natural language processing models from the Transformers library to understand the user's question and generate an appropriate response. For example, if a user asks, "How is the patient doing?", the server uses natural language processing to generate the response, "The patient's heart rate is normal."

[1284] Image Recognition Method

[1285] The server uses OpenCV and MediaPipe to analyze image data sent from the device and detect abnormalities in the patient's face or body. For example, when a user takes a picture of the patient's face with the device's camera, the image recognition means analyzes the image and notifies the user that "the patient's facial information has been confirmed."

[1286] Audio guidance means

[1287] The server provides audio guidance to the user on emergency medical procedures. Using the PyAudio library, the user receives audio guidance through their device. For example, specific instructions such as "Push on the chest in a steady rhythm" are provided.

[1288] Artificial intelligence methods for assisting medical procedures

[1289] The server utilizes machine learning and deep learning models to collect patient data in real time and suggest optimal medical treatments, enabling users to take appropriate medical action without specialized knowledge.

[1290] Sensor means

[1291] The device uses sensors to monitor vital signs such as heart rate, body temperature, and respiratory rate in real time, and transmits this data to a server that can then provide appropriate instructions if an abnormality is detected.

[1292] Remote Assistance Methods

[1293] The device can connect to a medical professional via the Internet and receive real-time support, for example, by establishing a connection with the professional and providing feedback of the medical professional's instructions to the user.

[1294] Encryption and Access Control Measures

[1295] The server uses the Fernet class from the cryptography library to encrypt and securely store patient data, ensuring data privacy by allowing only authorized users to access it.

[1296] Emotion Engine

[1297] The server uses the TextBlob library to analyze the user's emotional state and uses gentler words if the user is nervous. For example, if the user says, "I'm very nervous," the server determines, "The user is nervous. Please use gentler words," and adjusts its response accordingly.

[1298] Smartphone application

[1299] The device integrates these functions and provides them as a smartphone application, providing users with real-time judgment results and instructions to help them take appropriate action.

[1300] Examples:

[1301] 1. Natural Language Processing

[1302] User says: "How is the patient doing?"

[1303] Prompt: How is the patient doing?

[1304] 2. Image Recognition

[1305] User action: Take a picture of the patient's face with the camera

[1306] Prompt: Based on the patient's current image, please diagnose their condition.

[1307] 3. Audio guidance

[1308] User action: Listen to the audio guide and take emergency measures

[1309] Prompt: Push rhythmically against the chest.

[1310] 4. Emotion Engine

[1311] User says: I'm so nervous.

[1312] Prompt: Rate the user's emotional state.

[1313] In this way, by linking each means, the system becomes one that allows users to respond effectively even in emergencies.

[1314] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1315] Step 1:

[1316] The server receives questions in natural language from the user. The input is the user's speech, and the output is text data. Using natural language processing means, the server analyzes this text data and understands the intent of the question. For example, in response to the question, "How is the patient's condition?", the server generates a response such as, "The patient's heart rate is normal."

[1317] Step 2:

[1318] The device takes an image of the patient with a camera and sends the image data to the server. The input is the camera image, and the output is the image data sent to the server. The image recognition means analyzes the image data using OpenCV and MediaPipe to detect abnormalities in the patient's face or body. For example, it generates a diagnosis result such as "The patient's facial information has been confirmed."

[1319] Step 3:

[1320] The server uses the audio guidance means to guide the user through the steps of the medical procedure by voice. The input is instructions for appropriate medical procedures, and the output is audio data. The PyAudio library is used to generate audio and provide it to the user. For example, specific instructions such as "Please press on the chest in a steady rhythm" are provided by voice.

[1321] Step 4:

[1322] The terminal uses sensors to monitor the patient's vital signs, such as heart rate, body temperature, and respiratory rate, in real time, and transmits the data to a server. The input is sensor data, and the output is the vital signs transmitted to the server. The server analyzes this data and, if an abnormality is detected, instructs the appropriate countermeasures.

[1323] Step 5:

[1324] The server uses a machine learning model to suggest optimal medical treatment based on patient data. The input is real-time patient data, and the output is a recommendation for the optimal treatment method. This recommendation is then notified to the user via a smartphone application.

[1325] Step 6:

[1326] The device connects to medical professionals in real time via the Internet and receives support. The input is a connection request to a medical professional, and the output is the establishment of the connection. The server relays this connection and provides instructions from the professional to the user in real time.

[1327] Step 7:

[1328] The server uses a cryptography library to encrypt and securely store patient data. The input is patient data, and the output is encrypted data. Access control is performed so that only authorized users can access the data.

[1329] Step 8:

[1330] The server uses the TextBlob library to analyze the user's speech and recognize their emotional state. The input is the user's speech text, and the output is an evaluation of their emotional state. For example, if the user says, "I'm very nervous," the server determines that the user is nervous. Please use gentler words, and adjusts the tone of the response accordingly.

[1331] Step 9:

[1332] The device integrates the information generated at each step and provides real-time feedback to the user through a smartphone application. The input is diagnosis results and instructions from the server, and the output is notifications to the user. For example, it displays image recognition results, voice guidance, suggested medical procedures, instructions from specialists, and other information to help the user respond appropriately.

[1333] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1334] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1335] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1336] [Third embodiment]

[1337] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1338] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1339] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1340] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1341] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1342] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1343] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1344] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1345] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1346] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1347] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1348] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1349] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element works together to assist in emergency medical treatment. Specific embodiments for carrying out the present invention will be described below.

[1350] Natural Language Processing Implementation

[1351] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question and generate a response such as, "The patient's heart rate is normal." The response is conveyed to the user via the terminal in voice or text.

[1352] Image Recognition Implementation

[1353] Image data acquired from cameras and sensors is analyzed using image recognition techniques to diagnose a patient's condition. For example, a device takes an image of a patient, and the server analyzes the image using image recognition technology to predict the patient's condition from the patient's facial expression and appearance. The diagnosis result is notified to the user visually and audibly.

[1354] Implementing voice guidance

[1355] The voice guidance means guides the user through emergency medical procedures by voice. For example, if cardiopulmonary resuscitation is required, the server issues specific instructions such as "Please press on the chest in a steady rhythm" through the voice guidance program and conveys this to the user via the terminal.

[1356] Implementation of artificial intelligence for medical procedure assistance

[1357] AI tools utilize machine learning and deep learning to have the knowledge and judgment necessary to assist medical procedures. The server collects patient data in real time and uses AI models to determine the optimal treatment method. The device then instructs the user on the treatment method and encourages appropriate action.

[1358] Sensor Implementation

[1359] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the collected data and immediately generates appropriate instructions if an abnormality is detected.

[1360] Implementing Remote Assistance

[1361] The remote assistance method allows a user to connect with a medical professional in real time and receive guidance. For example, a device can establish a connection and allow the medical professional to check the patient's condition in real time. The server receives instructions from the professional and relays them to the user via the device. This allows for the management of acute symptoms.

[1362] Implementing encryption and access control

[1363] Encryption and access control are technologies used to ensure the safety of patient data. Patient data collected by the device is encrypted before being sent to the server. The server securely stores the encrypted data and ensures that only authorized users can access the data based on the access control policy. Security patches are also regularly applied to maintain the safety of the system.

[1364] Specific examples

[1365] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[1366] 1. Sensor detection

[1367] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[1368] 2. CPR instructions

[1369] Server: Uses image recognition technology to check the patient's condition and determine that they have undergone cardiac arrest.

[1370] Terminal: Based on instructions from the server, the terminal provides voice and on-screen guidance on CPR procedures to the user.

[1371] User: Begin CPR as instructed.

[1372] 3. Providing remote assistance

[1373] Device: Connect with medical professionals and receive real-time guidance.

[1374] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[1375] User: Perform additional medical procedures under the guidance of a professional.

[1376] 4. Ensuring safety

[1377] Terminal: Patient data is encrypted and securely transmitted to the server.

[1378] Server: Encrypts and stores data and applies access controls.

[1379] In this way, the present invention aims to support the implementation of appropriate medical procedures in emergencies even without specialized knowledge, thereby reducing the risk of sudden death.

[1380] The processing flow will be explained below.

[1381] Natural Language Processing Implementation

[1382] Step 1:

[1383] Terminal: The user types a question into the microphone, for example, "How is the patient doing?"

[1384] Step 2:

[1385] Terminal: Converts voice data into text and sends it to the server.

[1386] Step 3:

[1387] Server: The received text data is input into a natural language processing model and the question content is analyzed.

[1388] Step 4:

[1389] Server: Generates appropriate responses to questions and sends them to the terminal in text format.

[1390] Step 5:

[1391] Terminal: Converts the received response into speech and provides it to the user. The speech response is, "The patient's heart rate is normal."

[1392] Image Recognition Implementation

[1393] Step 1:

[1394] Terminal: Captures images of the patient through a camera and sends the image data to the server.

[1395] Step 2:

[1396] Server: Inputs the received image data into an image recognition model to predict the patient's condition.

[1397] Step 3:

[1398] Server: Generates a diagnosis of the medical condition and determines the appropriate course of treatment.

[1399] Step 4:

[1400] Server: Sends instructions on how to proceed to the terminal.

[1401] Step 5:

[1402] Terminal: The user is notified of the diagnosis and treatment method visually and audibly. "The patient is unconscious. Please begin CPR." is displayed and audible.

[1403] Implementing voice guidance

[1404] Step 1:

[1405] Server: Generates appropriate medical procedures based on the patient's condition.

[1406] Step 2:

[1407] Server: Sends the generated procedure information to the terminal.

[1408] Step 3:

[1409] Device: Provides instructions to the user through a voice guidance system, such as "Compress the chest about 5 cm at a rate of 100 times per minute."

[1410] Step 4:

[1411] User: Follow the audio prompts to perform CPR.

[1412] Implementation of artificial intelligence for medical procedure assistance

[1413] Step 1:

[1414] Terminal: Collects the patient's vital data (e.g., heart rate, blood pressure) from sensors and sends it to the server.

[1415] Step 2:

[1416] Server: Inputs collected vital data into an artificial intelligence model to determine appropriate treatment.

[1417] Step 3:

[1418] Server: Based on the judgment results, generates appropriate treatment procedures and sends them to the terminal.

[1419] Step 4:

[1420] Terminal: Provides the user with visual and audio instructions for appropriate treatment. "Patient's heart rate is dropping. Prepare oxygen for inhalation."

[1421] Sensor Implementation

[1422] Step 1:

[1423] Device: Uses sensors to capture real-time vital data such as the patient's heart rate, temperature, and respiratory rate.

[1424] Step 2:

[1425] Terminal: Sends the acquired data to the server.

[1426] Step 3:

[1427] Server: Analyzes the received data and determines whether there are any abnormalities.

[1428] Step 4:

[1429] Server: If an abnormality is detected, it determines the appropriate response method and sends instruction information to the terminal.

[1430] Step 5:

[1431] Device: Notifies the user of the abnormality and how to respond. "An abnormal heart rate has been detected. Please begin CPR immediately."

[1432] Implementing Remote Assistance

[1433] Step 1:

[1434] Terminal: When an abnormality in the patient is detected, the remote assistance function is activated and sends a request to the server to establish a connection with a medical professional.

[1435] Step 2:

[1436] Server: Establishes connection with medical professional services and transmits patient status in real time.

[1437] Step 3:

[1438] Server: Receives instructions from medical professionals and sends them to the device.

[1439] Step 4:

[1440] Terminal: Provides audio and visual instructions from a specialist to the user. "Please put an oxygen mask on the patient. Detailed instructions are below."

[1441] Step 5:

[1442] User: Follow the instructions of a professional and take appropriate measures.

[1443] Implementing encryption and access control

[1444] Step 1:

[1445] Terminal: Collects patient data (e.g., vital data, diagnostic results) and encrypts the data using an encryption algorithm.

[1446] Step 2:

[1447] Device: Sends encrypted data to the server.

[1448] Step 3:

[1449] Server: Receives the encrypted data and stores it in a secure database.

[1450] Step 4:

[1451] Server: Authenticates users accessing data and enforces access control policies.

[1452] Step 5:

[1453] Server: Regularly apply security patches to keep your systems secure.

[1454] The above is the flow of specific processing steps for each function, which will realize a system that supports appropriate responses at the scene of emergency medical care, even without specialized knowledge.

[1455] Example 1

[1456] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1457] In modern society, rapid and accurate responses are required in emergency medical situations, but appropriate treatment is often not provided due to the absence of experts or lack of knowledge. Furthermore, in order to accurately grasp the patient's condition and provide the necessary treatment, a wide range of technologies and knowledge must be integrated. In particular, the lack of real-time monitoring of patient conditions, accurate diagnosis of illness, and appropriate medical guidance is a problem.

[1458] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1459] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for analyzing user questions and generating appropriate responses, means for analyzing image data acquired by a camera and predicting the patient's condition, means for providing audio and visual guidance for medical procedure procedures, and means for encrypting and storing collected patient data, thereby enabling prompt and appropriate responses in emergency medical situations.

[1460] "Natural language processing means" refers to means for communicating with users in natural language and for understanding and analyzing questions and instructions.

[1461] "Image recognition means" is a means for analyzing image data acquired by a camera or sensor and identifying the state and characteristics of an object.

[1462] The "audio guidance means" is a means for providing instructions and information to the user by voice.

[1463] "Artificial intelligence means" refers to means that use machine learning and deep learning to assist medical procedures and determine the most appropriate treatment methods.

[1464] The "sensor means" is a means for monitoring the physical condition of a patient in real time and collecting the data.

[1465] "Remote assistance means" means a means of connecting with a medical professional in real time to receive specialized guidance and support.

[1466] "Encryption and access control means" means means for encrypting patient data and ensuring that only authorized users can access the data.

[1467] The "means for analyzing questions and generating responses" refers to means for analyzing voice or text questions entered by a user and generating appropriate responses.

[1468] The "means for analyzing image data and predicting the condition" is a means for analyzing images of a patient taken by a camera and predicting the condition of the patient.

[1469] The "means for providing guidance on the procedure of medical treatment" is a means for providing the user with audio and visual guidance on the specific procedure of emergency medical treatment.

[1470] "Means for encrypting and storing data" refers to means for encrypting collected patient data and storing it securely.

[1471] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element cooperates to assist emergency medical treatment. The embodiments of the present invention will be described in detail below.

[1472] Natural language processing implementation:

[1473] The server uses natural language processing to analyze questions or instructions entered by the user via voice or text and generate an appropriate response. For example, if a user asks, "How is the patient's condition?", the server uses natural language processing technology to understand the question and generates a response such as, "The patient's heart rate is normal." This response is then communicated to the user via the terminal via voice or text.

[1474] Image recognition implementation:

[1475] To understand the patient's condition, the device sends image data captured by the camera to a server. The server then analyzes the image data using image recognition tools and predicts the patient's condition based on the patient's facial expression and appearance. The diagnosis results are notified to the user visually and audibly. For example, the server checks for abnormalities in the patient's complexion or the presence of external injuries.

[1476] Implementing audio guidance methods:

[1477] The server uses the voice guidance means to guide the user through the procedures for emergency medical treatment. For example, if cardiopulmonary resuscitation is required, specific instructions such as "Please push on the chest in a steady rhythm" are provided by voice. These instructions are transmitted to the user through the terminal.

[1478] Implementation of artificial intelligence means:

[1479] The server uses machine learning and deep learning techniques to analyze patient data in real time and determine the optimal treatment. The device then instructs the user on this treatment and encourages appropriate medical intervention. For example, an artificial intelligence model can be used to detect abnormal heart rates or respiratory arrest and generate instructions for immediate response.

[1480] Sensor means implementation:

[1481] The device has sensors that monitor the patient's heart rate, body temperature, respiratory rate, and other data in real time. This data is sent to a server, which analyzes the data and immediately generates instructions if an abnormality is detected. For example, if a sudden change in heart rate is detected, it will immediately issue instructions for CPR.

[1482] Implementing remote assistance methods:

[1483] The terminal has a remote assistance means to connect to a medical professional in real time and receive appropriate guidance. The medical professional checks the patient's condition and provides necessary guidance. The server receives the expert's instructions and relays them to the user, allowing the user to perform accurate treatment based on the expert's guidance.

[1484] Implementing encryption and access control measures:

[1485] Patient data collected by the device is encrypted before being sent to the server, where it is stored and access is controlled to ensure only authorized users can access it. Security patches are also regularly applied to maintain safety.

[1486] Examples:

[1487] As a concrete example, consider a situation where a cardiac arrest occurs in a public place. In this case, the system operates as follows:

[1488] 1. A user uses a device to ask, "How is the patient doing?"

[1489] 2. The device takes an image of the patient with its camera and sends it to the server.

[1490] 3. The server uses image recognition to diagnose the patient's condition and determine that cardiopulmonary resuscitation is necessary.

[1491] 4. The server generates instructions using voice guidance means, such as "Please press the chest at a regular rhythm," and conveys this to the user via the terminal.

[1492] 5. Use remote assistance methods to connect with medical professionals for additional guidance in real time.

[1493] 6. The data collected by the device is encrypted and securely sent to the server.

[1494] An example of a prompt sentence could be something like, "Please explain how the system would respond if someone had a sudden cardiac arrest in the park."

[1495] This enables the system to respond quickly and accurately in emergencies and support appropriate medical treatment.

[1496] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1497] Step 1:

[1498] The user enters a question.

[1499] Specific behavior:

[1500] User: Ask via voice or text, "How is the patient doing?"

[1501] Device: Converts speech to text and sends the data to the server.

[1502] Input: User's voice or text question.

[1503] Output: Question data in text format.

[1504] Step 2:

[1505] The server analyzes the user's question and generates a response.

[1506] Specific behavior:

[1507] Server: Uses natural language processing techniques to analyze the user's question and generate an appropriate response.

[1508] Server: Generates a response, for example, "The patient's heart rate is normal."

[1509] Input: The question data submitted by the user in text format.

[1510] Output: The response data in text format.

[1511] Step 3:

[1512] The user is notified of the response from the server.

[1513] Specific behavior:

[1514] Terminal: The response data from the server is converted into voice using speech synthesis technology and notified to the user.

[1515] Terminal: Also displayed as text data.

[1516] Input: Response data sent from the server in text format.

[1517] Output: Voice and text response notification.

[1518] Step 4:

[1519] Image data is acquired to confirm the patient's condition.

[1520] Specific behavior:

[1521] Device: Takes images of the patient's face and body using a camera.

[1522] Terminal: Sends the captured image data to the server.

[1523] Input: Patient images.

[1524] Output: Image data sent to the server.

[1525] Step 5:

[1526] The server analyzes the image data and predicts the condition of the disease.

[1527] Specific behavior:

[1528] Server: Uses image recognition technology to analyze a patient's facial expression, appearance, and possible injuries to predict their condition.

[1529] Server: Generates an analysis result, for example, "The patient may be unconscious."

[1530] Input: Image data sent from the device.

[1531] Output: Predicted disease state.

[1532] Step 6:

[1533] The prediction results are notified to the user.

[1534] Specific behavior:

[1535] Terminal: Notifies the user of predicted medical conditions via voice or text.

[1536] Input: Prediction results of the disease condition sent from the server.

[1537] Output: Prediction result notification in audio and text format.

[1538] Step 7:

[1539] Provide guidance on emergency medical procedures.

[1540] Specific behavior:

[1541] User: Ask "What do I do?"

[1542] Server: Analyzes the situation and determines the medical intervention required.

[1543] Server: For example, it determines that cardiopulmonary resuscitation is necessary and generates instructions such as "Push on the chest in a steady rhythm."

[1544] Terminal: Provides audio and visual instructions to the user.

[1545] Input: Questions from the user and patient data.

[1546] Output: Medical procedure instructions in audio and visual format.

[1547] Step 8:

[1548] Offer remote assistance.

[1549] Specific behavior:

[1550] Device: Establish a connection with a medical professional and receive guidance in real time.

[1551] Server: Receives expert instructions and conveys them to the user in an appropriate manner.

[1552] User: Follow the guidance of a professional and perform appropriate medical procedures.

[1553] Input: Patient status and professional instructions.

[1554] Output: Real-time expert feedback.

[1555] Step 9:

[1556] The patient data is encrypted and sent to the server.

[1557] Specific behavior:

[1558] Terminal: Encrypts patient data (heart rate, images, prediction results, etc.).

[1559] Device: Sends encrypted data to the server.

[1560] Input: Patient data.

[1561] Output: The encrypted data.

[1562] Step 10:

[1563] The server stores encrypted data and controls access.

[1564] Specific behavior:

[1565] Server: Collected data is stored in encrypted form.

[1566] Server: Ensure that only authenticated users can access the data.

[1567] Servers: Regularly apply security patches to maintain security.

[1568] Input: Encrypted patient data.

[1569] Output: Stored data and access logs.

[1570] (Application example 1)

[1571] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1572] Conventional emergency medical intervention systems have limitations in their ability to quickly reach the scene and accurately grasp the patient's condition in real time. Systems utilizing autonomous vehicles, in particular, lack the advanced technology necessary to provide appropriate medical intervention in an emergency. Another problem is the difficulty of providing accurate instructions while connecting with medical professionals in real time. The present invention aims to solve these problems and provide an advanced emergency medical intervention support system using autonomous vehicles to reduce the risk of sudden patient death.

[1573] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1574] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for automatically arriving at the emergency scene, means for diagnosing the patient's condition using an in-vehicle camera, and means for assisting medical procedures in real time within the autonomous vehicle, thereby enabling rapid arrival at the emergency scene and accurate medical procedures and real-time collaboration with medical professionals at the scene.

[1575] "Natural language processing means" refers to means that enables communication with the user in natural language.

[1576] "Image recognition means" refers to a means for analyzing image data acquired from a camera or sensor and diagnosing the patient's condition.

[1577] The "audio guidance means" is a means for providing the user with audio guidance on the specific steps of a medical procedure.

[1578] "Artificial intelligence means to assist medical procedures" refers to means that utilize machine learning and deep learning to acquire the knowledge and judgment capabilities necessary for medical procedures.

[1579] The "sensor means" is a means for monitoring the patient's physical condition, such as heart rate, body temperature, and respiratory rate, in real time.

[1580] "Remote assistance means" means a means for connecting with a medical professional in real time for guidance.

[1581] "Encryption and access control measures" means measures that encrypt patient data to ensure its security and ensure that only authorized users can access it.

[1582] "Means for arriving at an emergency scene using autonomous driving" means means that enable a vehicle to quickly arrive at an emergency scene using autonomous driving technology.

[1583] "Image recognition means for diagnosing a patient's condition using an in-vehicle camera" refers to a means for diagnosing a patient's condition by analyzing images taken by a camera installed inside an autonomous vehicle.

[1584] "Means for assisting medical procedures in real time within an autonomous vehicle" means means for assisting medical procedures in real time within an autonomous vehicle.

[1585] The present invention is an advanced emergency medical intervention assistance system using an autonomous vehicle, which integrates various technologies, including natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical intervention, sensors means, remote assistance means, encryption and access control means, means for autonomously driving to arrive at the emergency scene, image recognition means for diagnosing the patient's condition using an in-vehicle camera, and means for providing real-time medical intervention assistance within the autonomous vehicle.

[1586] Autonomous driving to emergency scene

[1587] The server receives emergency calls and uses GPS navigation technology to determine the optimal route to the scene. It uses the Google Maps API to analyze traffic conditions in real time and select the optimal route. The autonomous vehicle then automatically follows this route.

[1588] Natural language processing tools

[1589] The server uses natural language processing models (e.g., the BERT model) to communicate with emergency responders in natural language. This allows the user to ask a question such as "How is the patient doing?" and the system understands the question and generates a response such as "The patient's heart rate is normal." This response is then conveyed to the user via voice or text.

[1590] Image Recognition Method

[1591] The server analyzes image data captured by the vehicle's camera using a convolutional neural network (CNN) to diagnose the patient's condition. This data is securely transmitted to the server, and the results of the analysis are notified to the user.

[1592] Audio guidance means

[1593] The server uses a voice guidance means to provide voice instructions on specific steps of medical procedures. For example, if cardiopulmonary resuscitation is required, the server uses a technology that provides voice instructions such as "Push on the chest in a steady rhythm."

[1594] Remote Assistance Methods

[1595] The server uses a video conferencing system (e.g., Zoom API) to connect with medical professionals in the autonomous vehicle in real time, allowing the medical professionals to monitor the situation on-site and provide instructions.

[1596] Sensor means

[1597] The server uses sensors to monitor heart rate, body temperature, breathing rate, etc. in real time. This data is collected by the vehicle's in-vehicle system and transmitted to the server. If an abnormality is detected, appropriate instructions are generated immediately.

[1598] Encryption and Access Control Measures

[1599] The server uses a method for encrypting and controlling access to patient data, using AES-256 data encryption and OAuth user authentication to ensure data security.

[1600] Specific examples of emergencies

[1601] For example, consider a sudden cardiac arrest in a public place. In this case, the system operates as follows:

[1602] 1. An autonomous vehicle receives an emergency call and arrives at the scene via the optimal route.

[1603] 2. Diagnose the patient's condition using image recognition means and provide specific instructions to the user via voice guidance.

[1604] 3. An example of a prompt for reproduction is the text, "As an example of the use of an emergency medical guidance system, please tell us how to respond to a patient in cardiac arrest."

[1605] Program processing

[1606] The server connects various technologies to efficiently support emergency medical procedures. The system integrates autonomous vehicle navigation, natural language processing, image recognition, voice guidance, remote assistance, and security management to ensure a fast and accurate response in emergencies.

[1607] Operation of this system enables rapid arrival at the emergency site and appropriate medical treatment on the spot, significantly reducing the risk of sudden patient death.

[1608] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1609] Step 1:

[1610] The server receives emergency calls. It receives emergency call data (the caller's location information and the details of the emergency) as input, and calculates the optimal route as output. Data processing involves using a GPS navigation system (e.g., Google Maps API) to analyze real-time traffic information and generate the optimal route to quickly reach the scene. The server then sends the generated route information to the autonomous vehicle.

[1611] Step 2:

[1612] The autonomous vehicle (terminal) begins autonomous driving towards the site based on the route information received from the server. The input is route information, and the output is the execution of autonomous driving to the site. Specifically, the vehicle's autonomous driving system controls the vehicle according to GPS information, and drives while recognizing traffic lights and obstacles.

[1613] Step 3:

[1614] When the device arrives at the scene, it uses natural language processing to begin communicating with the user (emergency responder). The input is the user's question or instruction, and the output is a response to the user. For data processing, a natural language processing model (e.g., BERT) is used to analyze the meaning of the question and generate an appropriate response. The server transmits the generated response from the device to the user via voice or text.

[1615] Step 4:

[1616] The server uses the device's camera to diagnose the patient's condition on-site using image recognition. The input is camera image data, and the output is the diagnosis result. Data processing involves image analysis using a convolutional neural network (CNN) to diagnose the patient's external symptoms. The diagnosis result is transmitted from the server to the user via the device.

[1617] Step 5:

[1618] The terminal uses voice guidance generated by the server to instruct the user on the steps of the medical procedure. The input is the diagnosis result and the output is the voice guidance. The server executes the voice guidance program, generates specific instructions, and transmits them to the user as voice from the terminal. For example, the instruction may be "Please press on the chest at a regular rhythm."

[1619] Step 6:

[1620] The device connects to medical professionals in real time via a server to provide remote assistance. The input is a connection request, and the output is the establishment of a video call connection. For data processing, a video conferencing system (e.g., Zoom API) is used to send and receive video and audio in real time to receive instructions from medical professionals. The instructions from the professional are then communicated to the user via the device.

[1621] Step 7:

[1622] The server securely stores all collected patient data using encryption and access control measures. The input is collected data, and the output is encrypted data. Data processing involves encrypting the data using the AES-256 encryption algorithm and controlling access via OAuth. The data is securely stored on the server and can only be accessed by authorized users.

[1623] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1624] The present invention is a system that includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, and an emotion engine for recognizing the user's emotions. The purpose is for each element to work together to assist emergency medical procedures and effectively support patient life-saving efforts.

[1625] Natural Language Processing Implementation

[1626] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question, generate a response such as, "The patient's heart rate is normal," and convey this to the user via voice or text via the terminal.

[1627] Image Recognition Implementation

[1628] Image recognition is used to analyze image data acquired from cameras and sensors and diagnose the patient's condition. For example, the device takes an image of the patient, and the server analyzes the image using image recognition technology. Based on the results of this analysis, the user is notified of the diagnosis and necessary treatment methods visually and audibly.

[1629] Implementing voice guidance

[1630] The voice guidance means guides the user through emergency medical procedures by voice. For example, the server generates cardiopulmonary resuscitation procedures and issues specific instructions such as "Please press on the chest in a steady rhythm" to the user via the terminal.

[1631] Implementation of artificial intelligence for medical procedure assistance

[1632] AI tools utilize machine learning and deep learning to have the knowledge and judgment capabilities necessary to assist medical procedures. The server collects patient data in real time, uses AI models to determine the optimal treatment method, and communicates that treatment method to the user via the device.

[1633] Sensor Implementation

[1634] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the data and issues appropriate instructions if an abnormality is detected.

[1635] Implementing Remote Assistance

[1636] The remote assistance means has a function for connecting with a medical professional in real time and receiving guidance. For example, the terminal establishes a connection with the professional and allows the professional to check the patient's condition in real time through a server, thereby transmitting instructions from the professional to the user.

[1637] Implementing encryption and access control

[1638] Encryption and access control measures are techniques for ensuring the security of patient data. Patient data collected by devices is encrypted before being sent to a server, which then securely stores the encrypted data and ensures that only authorized users can access it.

[1639] Implementing the Emotion Engine

[1640] The emotion engine analyzes the user's tone of voice and facial expressions to recognize the user's emotional state. It can use the emotion engine to assess the user's stress level and adjust responses accordingly. For example, if the server determines that the user is nervous, it can adjust its instructions to use gentler language and a more soothing tone.

[1641] Specific examples

[1642] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[1643] 1. Sensor detection

[1644] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[1645] 2. CPR instructions

[1646] Server: Uses image recognition technology to check the patient's condition and determine that they have undergone cardiac arrest.

[1647] Terminal: Based on instructions from the server, the terminal provides voice and on-screen guidance on CPR procedures to the user.

[1648] User: Begin CPR as instructed.

[1649] 3. Providing remote assistance

[1650] Device: Connect with medical professionals and receive real-time guidance.

[1651] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[1652] User: Perform additional medical procedures under the guidance of a professional.

[1653] 4. Emotion recognition and response adjustment

[1654] Device: Analyzes the tone of the user's voice and uses an emotion engine to determine the user's state.

[1655] Server: If the user is nervous, generate instructions using words and tones to help them relax and send them to the device.

[1656] Terminal: Provides instructions to the user in a friendly tone.

[1657] User: Stay calm and perform appropriate medical procedures.

[1658] In this way, the present invention supports the implementation of appropriate medical procedures in emergencies even without specialized knowledge, and also enables flexible responses according to the user's emotional state.

[1659] The processing flow will be explained below.

[1660] Natural Language Processing Implementation

[1661] Step 1:

[1662] Terminal: The user types a question into the microphone, for example, "How is the patient doing?"

[1663] Step 2:

[1664] Terminal: Converts the voice data into text and sends it to the server.

[1665] Step 3:

[1666] Server: The received text data is input into a natural language processing model and the question content is analyzed.

[1667] Step 4:

[1668] Server: Generates appropriate responses to questions and sends the text responses to the device.

[1669] Step 5:

[1670] Terminal: Converts the received response into speech and provides it to the user. The speech response is, "The patient's heart rate is normal."

[1671] Image Recognition Implementation

[1672] Step 1:

[1673] Terminal: Captures images of the patient through a camera and sends the image data to the server.

[1674] Step 2:

[1675] Server: Inputs the received image data into an image recognition model to predict the patient's condition.

[1676] Step 3:

[1677] Server: Generates a diagnosis of the medical condition and determines the appropriate course of treatment.

[1678] Step 4:

[1679] Server: Sends instructions on how to proceed to the terminal.

[1680] Step 5:

[1681] Terminal: The user is notified of the diagnosis and treatment method visually and audibly. "The patient is unconscious. Please begin CPR." is displayed and audible.

[1682] Implementing voice guidance

[1683] Step 1:

[1684] Server: Generates appropriate medical procedures based on the patient's condition.

[1685] Step 2:

[1686] Server: Sends the generated procedure information to the terminal.

[1687] Step 3:

[1688] Device: Provides instructions to the user through a voice guidance system, such as "Compress the chest about 5 cm at a rate of 100 times per minute."

[1689] Step 4:

[1690] User: Follow the audio prompts to perform CPR.

[1691] Implementation of artificial intelligence for medical procedure assistance

[1692] Step 1:

[1693] Terminal: Collects the patient's vital data (e.g., heart rate, blood pressure) from sensors and sends it to the server.

[1694] Step 2:

[1695] Server: Inputs collected vital data into an artificial intelligence model to determine appropriate treatment.

[1696] Step 3:

[1697] Server: Based on the judgment results, generates appropriate treatment procedures and sends them to the terminal.

[1698] Step 4:

[1699] Terminal: Provides the user with visual and audio instructions for appropriate treatment. "Patient's heart rate is dropping. Prepare oxygen for inhalation."

[1700] Sensor Implementation

[1701] Step 1:

[1702] Device: Uses sensors to capture real-time vital data such as the patient's heart rate, temperature, and respiratory rate.

[1703] Step 2:

[1704] Terminal: Sends the acquired data to the server.

[1705] Step 3:

[1706] Server: Analyzes the received data and determines whether there are any abnormalities.

[1707] Step 4:

[1708] Server: If an abnormality is detected, it determines the appropriate response method and sends instruction information to the terminal.

[1709] Step 5:

[1710] Device: Notifies the user of the abnormality and how to respond. "An abnormal heart rate has been detected. Please begin CPR immediately."

[1711] Implementing Remote Assistance

[1712] Step 1:

[1713] Terminal: When an abnormality in the patient is detected, the remote assistance function is activated and sends a request to the server to establish a connection with a medical professional.

[1714] Step 2:

[1715] Server: Establishes connection with medical professional services and transmits patient status in real time.

[1716] Step 3:

[1717] Server: Receives instructions from medical professionals and sends them to the device.

[1718] Step 4:

[1719] Terminal: Provides audio and visual instructions from a specialist to the user. "Please put an oxygen mask on the patient. Detailed instructions are below."

[1720] Step 5:

[1721] User: Follow the instructions of a professional and take appropriate measures.

[1722] Implementing encryption and access control

[1723] Step 1:

[1724] Terminal: Collects patient data (e.g., vital data, diagnostic results) and encrypts the data using an encryption algorithm.

[1725] Step 2:

[1726] Device: Sends encrypted data to the server.

[1727] Step 3:

[1728] Server: Receives the encrypted data and stores it in a secure database.

[1729] Step 4:

[1730] Server: Authenticates users accessing data and enforces access control policies.

[1731] Step 5:

[1732] Server: Regularly apply security patches to keep your systems secure.

[1733] Implementing the Emotion Engine

[1734] Step 1:

[1735] Terminal: The user types a question or comment into the microphone.

[1736] Step 2:

[1737] Device: Converts voice data into text and analyzes it with an emotion engine.

[1738] Step 3:

[1739] On the device: The emotion engine analyzes the user's tone of voice and vocabulary to assess their emotional state (e.g., stress, relief).

[1740] Step 4:

[1741] Server: Adjusts the response content based on the user's emotional state and generates a text response.

[1742] Step 5:

[1743] The device converts the tailored response into speech and provides it to the user. For example, if the device determines that the user is nervous, it responds in a gentle tone, such as "Please stay calm. Everything is fine."

[1744] Step 6:

[1745] User: Follows the coordinated response and performs appropriate medical procedures in a relaxed manner.

[1746] These are the specific processing steps of a system that includes an emotion engine, which can support prompt and appropriate medical treatment in emergencies while taking into consideration the user's emotional state.

[1747] Example 2

[1748] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1749] In modern medical settings, there is a demand for providing advanced medical support to general users who lack the knowledge and skills to perform appropriate and prompt medical procedures in emergencies. Furthermore, there is a lack of technology to accurately grasp a patient's condition and provide appropriate treatment in real time. In particular, in emergencies, it is necessary to provide appropriate responses to users who are under stress or tension. Furthermore, the safe handling of patient data and real-time communication with specialists are also issues.

[1750] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting patient data and controlling access, and emotion recognition means for recognizing the user's emotional state and adjusting responses. This allows even general users to perform appropriate medical procedures in emergencies, enables real-time diagnosis and appropriate treatment instructions, and realizes flexible responses according to the user's emotional state. It also ensures safe handling of patient data and real-time communication with professionals.

[1751] "Natural language processing means" refers to technology that understands text or voice input from a user and generates an appropriate response accordingly.

[1752] "Image recognition means" is a technology that analyzes image data acquired using a camera or sensor and understands its content.

[1753] "Audio guidance means" refers to a technique for providing information and instructions to a user by voice.

[1754] "Artificial intelligence means to assist medical procedures" refers to technologies that utilize machine learning and deep learning to provide judgments and advice to assist medical procedures.

[1755] The "sensor means for monitoring the patient's condition in real time" refers to a sensor technology that measures and monitors the patient's biological information, such as heart rate, body temperature, and respiratory rate, in real time.

[1756] "Remote assistance means for connecting with medical professionals in real time" refers to technology that allows users to connect with medical professionals in real time using the Internet or other means to receive guidance and advice.

[1757] "Means for encrypting and controlling access to patient data" refers to technology that encrypts data to protect patient data and allows only authorized users to access the data.

[1758] "Emotion recognition means that recognizes the user's emotional state and adjusts the response" is a technology that analyzes the user's tone of voice and facial expressions to determine their emotional state and generates an appropriate response according to that state.

[1759] The present invention is a system for assisting emergency medical procedures, and mainly includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, and emotion recognition means for recognizing the user's emotional state and adjusting responses.

[1760] Natural Language Processing Implementation

[1761] Natural language processing means understand the text or voice input by the user and generate an appropriate response. For example, if a user inputs "How is the patient's condition?", the server will use natural language processing technology (e.g., the BERT model) to analyze the user's question and generate a response such as "The patient's heart rate is normal." The generated response is communicated to the user via the terminal in voice or text.

[1762] Image Recognition Implementation

[1763] The image recognition means analyzes image data acquired from cameras and sensors. For example, the device takes an image of the patient, and the server analyzes the image using image processing technology (e.g., OpenCV or TensorFlow). Based on the results of this analysis, a diagnosis (e.g., "The patient's skin color indicates a decrease in oxygen saturation") is generated and notified to the user visually and audibly.

[1764] Implementing voice guidance

[1765] The audio guidance means provides audio guidance on the procedures for emergency medical treatment. The guidance content generated by the server (e.g., "Please press on the chest in a steady rhythm") is transmitted to the user via the terminal. The user can follow these instructions to perform medical treatment.

[1766] Implementation of artificial intelligence for medical procedure assistance

[1767] Artificial intelligence methods use machine learning and deep learning models (e.g., models using TensorFlow) to analyze patient data in real time and determine the optimal treatment method. For example, the server analyzes the patient's vital signs, generates specific instructions such as "put on an oxygen mask," and notifies the user via the device.

[1768] Sensor Implementation

[1769] The sensor means monitors the patient's vital signs, such as heart rate, body temperature, and respiratory rate, in real time. This monitoring data is sent from the device to the server for analysis. If an abnormality is detected, the server generates instructions such as "Your fever is continuing, so please use a cooling sheet," and notifies the user via the device.

[1770] Implementing Remote Assistance

[1771] The remote assistance means has a function for connecting with a medical professional in real time. When a user requests remote assistance, the terminal establishes a connection with the medical professional. The server notifies the medical professional of the patient's condition and provides instructions to the user via the terminal. This allows the user to perform medical procedures based on the medical professional's guidance.

[1772] Implementing encryption and access control

[1773] Encryption and access control measures are techniques for ensuring the security of patient data. The device encrypts the collected data (e.g., AES encryption) and sends it to the server. The server securely stores the encrypted data and controls access to the data so that only authorized users can access it.

[1774] Implementing the Emotion Engine

[1775] The emotion recognition means analyzes the user's tone of voice and facial expressions to determine their emotional state. An emotion engine (e.g., Hume AI) can be used to assess the user's stress level and adjust the response accordingly. For example, if the server determines that the user is nervous, it can generate a gentle instruction such as "It's okay, please calm down" and send it to the device.

[1776] Specific examples

[1777] In the event of sudden cardiac arrest in a public place, the system operates as follows:

[1778] 1. Sensor detection

[1779] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[1780] 2. CPR instructions

[1781] Server: Uses image recognition technology to check the patient's condition, determines whether they have cardiac arrest, and generates audio guidance content.

[1782] Device: Provides voice and on-screen guidance to the user on how to perform CPR.

[1783] User: Begin CPR as instructed.

[1784] 3. Providing remote assistance

[1785] Device: Connect with medical professionals and receive real-time guidance.

[1786] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[1787] User: Perform additional medical procedures under the guidance of a professional.

[1788] 4. Emotion recognition and response adjustment

[1789] Device: Analyzes the tone of the user's voice and uses an emotion engine to determine the user's state.

[1790] Server: If the user is nervous, generate instructions using words and tones to help them relax and send them to the device.

[1791] Terminal: Provides instructions to the user in a friendly tone.

[1792] User: Stay calm and perform appropriate medical procedures.

[1793] Prompt Sentence Examples

[1794] "What are the specific medical steps a user should take in the event of cardiac arrest?"

[1795] "Please explain what to do if someone experiences sudden cardiac arrest in a public place."

[1796] "Please explain with a specific example the role of the emotion engine when detecting a patient's emergency situation."

[1797] In this way, the system supports general users in providing appropriate medical care in emergencies, enabling real-time diagnosis and treatment instructions, and enabling flexible responses according to the user's emotional state.It also ensures secure management of patient data and real-time communication with medical professionals.

[1798] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1799] Step 1:

[1800] User: Type the question into the terminal: "How is the patient doing?"

[1801] Input: User question (text or voice)

[1802] Output: Question text or audio data

[1803] Specific operations: The device prepares text or voice data to send the user's question to the server.

[1804] Step 2:

[1805] Terminal: Sends the user's query data to the server.

[1806] Input: Question text or audio data

[1807] Output: Question data is sent to the server

[1808] Specific operation: The terminal sends data to the server via the network.

[1809] Step 3:

[1810] Server: Uses natural language processing techniques to analyze the question and generate an appropriate response.

[1811] Input: Question text data

[1812] Output: Response text data

[1813] What happens: The server uses a natural language processing model (e.g., BERT) to analyze the question and generate an appropriate response.

[1814] Step 4:

[1815] Terminal: Receives response data from the server and conveys it to the user via voice or text.

[1816] Input: Response text data

[1817] Output: Spoken response or displayed text

[1818] Specific operation: The device converts the received text data into speech and conveys it to the user, or displays it as text on the screen.

[1819] Step 5:

[1820] User: Takes images to check the patient's condition.

[1821] Input: Patient images

[1822] Output: Captured image data

[1823] What it does: Takes an image of the patient using the device's camera.

[1824] Step 6:

[1825] Terminal: Sends the captured image data to the server.

[1826] Input: Image data

[1827] Output: Image data sent to the server

[1828] Specific operation: The terminal sends image data to the server via the network.

[1829] Step 7:

[1830] Server: Analyzes images using image recognition technology and generates diagnostic results.

[1831] Input: Image data

[1832] Output: Diagnostic result text data

[1833] Specific operation: The server analyzes image data using an image recognition model (e.g., OpenCV, TensorFlow) and generates a diagnosis result.

[1834] Step 8:

[1835] Terminal: Receives the diagnostic results from the server and notifies the user.

[1836] Input: Diagnostic result text data

[1837] Output: Audio or displayed text of diagnostic results

[1838] Specific operation: The device converts the received diagnostic results into voice and tells the user, or displays them as text on the screen.

[1839] Step 9:

[1840] Server: Generates guidance content for emergency medical treatment.

[1841] Input: Diagnostic result text data

[1842] Output: Guidance text data

[1843] Specific operation: The server generates appropriate guidance content based on the diagnosis results (e.g., "Please push on the chest in a steady rhythm").

[1844] Step 10:

[1845] Terminal: Receives guidance data from the server and conveys it to the user aloud.

[1846] Input: Guidance text data

[1847] Output: Voice guidance

[1848] Specific operation: The device converts the received guidance data into voice and conveys it to the user.

[1849] Step 11:

[1850] User: Follows the audio guidance to perform medical procedures.

[1851] Input: Voice guidance

[1852] Output: Medical procedure performed

[1853] Specific actions: The user follows the voice guidance and performs specific medical procedures.

[1854] Step 12:

[1855] Sensors: Collect patient vital data in real time.

[1856] Input: Patient vital data (heart rate, temperature, respiratory rate, etc.)

[1857] Output: Collected vital data

[1858] Specific operation: The sensor measures the patient's vital signs in real time and transmits them to the device.

[1859] Step 13:

[1860] Terminal: Sends vital data collected from sensors to the server.

[1861] Input: Collected vital data

[1862] Output: Vital data sent to the server

[1863] Specific operation: The device sends vital data to the server via the network.

[1864] Step 14:

[1865] Server: Analyzes vital data in real time and generates appropriate instructions if an abnormality is detected.

[1866] Input: Vital data

[1867] Output: Anomaly detection instruction text data

[1868] How it works: The server uses an AI model to analyze vital data and, if it detects an abnormality, generates instructions such as "use a cooling sheet."

[1869] Step 15:

[1870] Terminal: Receives instructions from the server and notifies the user.

[1871] Input: Instruction text data

[1872] Output: Spoken instructions or displayed text

[1873] Specific operation: The device converts the received instruction into voice and conveys it to the user, or displays it as text on the screen.

[1874] Step 16:

[1875] User: Follows the instructions of the server to perform appropriate medical procedures.

[1876] Input: Spoken instructions or displayed text

[1877] Output: Medical procedure performed

[1878] Specific operation: The user follows instructions from the server and performs appropriate medical procedures.

[1879] Step 17:

[1880] Device: Request remote assistance.

[1881] Input: Remote Assistance Request

[1882] Output: Request data to the server

[1883] Specific operation: The device sends a request for remote assistance to the server.

[1884] Step 18:

[1885] Server: Establishes a connection with medical professionals and communicates the patient's status.

[1886] Input: Request data, patient status data

[1887] Output: Status information for experts

[1888] Specific operation: The server connects with medical professionals in real time and relays the patient's status received from the device.

[1889] Step 19:

[1890] Expert: Checks the patient's status and sends instructions to the server.

[1891] Input: Patient status information

[1892] Output: Expert instructions

[1893] Specific operation: The expert analyzes the patient's condition and sends appropriate instructions to the server.

[1894] Step 20:

[1895] Server: Transmits the expert's instructions to the device.

[1896] Input: Expert instructions

[1897] Output: Instruction text data

[1898] Specific operation: The server sends the received instructions to the terminal.

[1899] Step 21:

[1900] Terminal: Receives instructions from the expert and notifies the user.

[1901] Input: Instruction text data

[1902] Output: Spoken instructions or displayed text

[1903] Specific operation: The device conveys the received instructions to the user via voice or text.

[1904] Step 22:

[1905] User: Act as directed by the expert.

[1906] Input: Spoken instructions or displayed text

[1907] Output: Actions to be performed

[1908] Specific Action: The user takes the necessary action based on the expert's instructions.

[1909] Step 23:

[1910] Device: Analyzes the user's tone of voice and facial expressions to determine their emotional state.

[1911] Input: User's tone of voice, facial expression data

[1912] Output: Emotional state assessment data

[1913] Specific operation: The device analyzes the user's tone of voice and facial expressions, and determines their emotional state using an emotion engine.

[1914] Step 24:

[1915] Server: Adjusts responses and generates instructions based on the emotional state.

[1916] Input: Emotional state assessment data

[1917] Output: Adjusted instruction text data

[1918] Specific behavior: The server uses an emotion engine to generate instructions based on the user's emotional state, including gentle tones and relaxing words.

[1919] Step 25:

[1920] Terminal: Receives coordinated instructions from the server and notifies the user.

[1921] Input: Adjusted instruction text data

[1922] Output: Spoken or displayed text of tailored instructions

[1923] What it does: The device provides tailored instructions to the user via voice or text to help them stay calm.

[1924] (Application example 2)

[1925] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1926] In the case of medical emergencies, it is difficult to take immediate and appropriate action. This is especially true for ordinary people without specialized knowledge who have difficulty performing accurate medical procedures in emergencies. Furthermore, existing systems do not adequately support making appropriate decisions based on the situation with the assistance of medical professionals in real time. Furthermore, there is a lack of flexible and interactive systems to respond to unexpected situations, and systems often cannot respond based on the user's emotional state. This raises the risk that users who feel nervous or anxious may not be able to take appropriate action, resulting in a worsening of the patient's condition.

[1927] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1928] In this invention, the server includes natural language processing means, image recognition means, and voice guidance means. This enables the server to accurately diagnose a patient's condition based on image data and provide appropriate voice guidance while engaging in natural communication with the user. The server also includes artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, and remote assistance means for connecting with medical professionals in real time. This provides strong support for the user in performing appropriate medical procedures even in emergencies, allowing the user to receive expert guidance in real time. The server also includes means for encrypting and controlling access to patient data, an emotion engine for recognizing the user's emotional state and adjusting communication accordingly, and a smartphone application for providing the user with assessment results and instructions in real time. This enables the user to remain calm and take appropriate action even in tense situations while ensuring the safety and privacy of patient data.

[1929] "Natural language processing means" is a technology that analyzes and understands the natural language input by the user and generates an appropriate response.

[1930] "Image recognition means" is a technology that analyzes image data acquired using a camera or sensor and recognizes objects or people.

[1931] "Audio guidance means" refers to a technique for generating audio to provide instructions and guidance to the user.

[1932] "Artificial intelligence means for assisting medical procedures" refers to technologies that utilize machine learning and deep learning to support medical procedures and determine the most appropriate treatment.

[1933] The "sensor means for monitoring the patient's condition in real time" refers to technology including sensors for measuring and monitoring biological information such as heart rate, body temperature, and respiratory rate in real time.

[1934] "Remote assistance means for connecting with medical professionals in real time" refers to technology that allows a person to connect with a medical professional in real time via the Internet and receive guidance and support from a remote location.

[1935] "Patient data encryption and access control measures" refers to technology that securely protects patient data and ensures that only authorized users can access it.

[1936] The "emotion engine" is a technology that has the function of analyzing the user's tone of voice and facial expressions to recognize the user's emotional state.

[1937] A "smartphone application" is software that runs on a smartphone and provides the various functions mentioned above in an integrated manner.

[1938] This invention is a system for supporting rapid and appropriate responses in medical emergencies, and includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring patient conditions in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, an emotion engine for recognizing the user's emotional state, and a smartphone application. Specific implementation examples of each means are shown below.

[1939] Natural language processing tools

[1940] The server uses natural language processing models from the Transformers library to understand the user's question and generate an appropriate response. For example, if a user asks, "How is the patient doing?", the server uses natural language processing to generate the response, "The patient's heart rate is normal."

[1941] Image Recognition Method

[1942] The server uses OpenCV and MediaPipe to analyze image data sent from the device and detect abnormalities in the patient's face or body. For example, when a user takes a picture of the patient's face with the device's camera, the image recognition means analyzes the image and notifies the user that "the patient's facial information has been confirmed."

[1943] Audio guidance means

[1944] The server provides audio guidance to the user on emergency medical procedures. Using the PyAudio library, the user receives audio guidance through their device. For example, specific instructions such as "Push on the chest in a steady rhythm" are provided.

[1945] Artificial intelligence means for assisting medical procedures

[1946] The server utilizes machine learning and deep learning models to collect patient data in real time and suggest optimal medical treatments, enabling users to take appropriate medical action without specialized knowledge.

[1947] Sensor means

[1948] The device uses sensors to monitor vital signs such as heart rate, body temperature, and respiratory rate in real time, and transmits this data to a server that can then provide appropriate instructions if an abnormality is detected.

[1949] Remote Assistance Methods

[1950] The device can connect to a medical professional via the Internet and receive real-time support, for example, by establishing a connection with the professional and providing feedback of the medical professional's instructions to the user.

[1951] Encryption and Access Control Measures

[1952] The server uses the Fernet class from the cryptography library to encrypt and securely store patient data, ensuring data privacy by allowing only authorized users to access it.

[1953] Emotion Engine

[1954] The server uses the TextBlob library to analyze the user's emotional state and uses gentler words if the user is nervous. For example, if the user says, "I'm very nervous," the server determines, "The user is nervous. Please use gentler words," and adjusts its response accordingly.

[1955] Smartphone application

[1956] The device integrates these functions and provides them as a smartphone application, providing users with real-time judgment results and instructions to help them take appropriate action.

[1957] Examples:

[1958] 1. Natural Language Processing

[1959] User says: "How is the patient doing?"

[1960] Prompt: How is the patient doing?

[1961] 2. Image Recognition

[1962] User action: Take a picture of the patient's face with the camera

[1963] Prompt: Based on the patient's current image, please diagnose their condition.

[1964] 3. Audio guidance

[1965] User action: Listen to the audio guide and take emergency measures

[1966] Prompt: Push rhythmically against the chest.

[1967] 4. Emotion Engine

[1968] User says: I'm so nervous.

[1969] Prompt: Rate the user's emotional state.

[1970] In this way, by linking each means, the system becomes one that allows users to respond effectively even in emergencies.

[1971] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1972] Step 1:

[1973] The server receives questions in natural language from the user. The input is the user's speech, and the output is text data. Using natural language processing means, the server analyzes this text data and understands the intent of the question. For example, in response to the question, "How is the patient's condition?", the server generates a response such as, "The patient's heart rate is normal."

[1974] Step 2:

[1975] The device takes an image of the patient with a camera and sends the image data to the server. The input is the camera image, and the output is the image data sent to the server. The image recognition means analyzes the image data using OpenCV and MediaPipe to detect abnormalities in the patient's face or body. For example, it generates a diagnosis result such as "The patient's facial information has been confirmed."

[1976] Step 3:

[1977] The server uses the audio guidance means to guide the user through the steps of the medical procedure by voice. The input is instructions for appropriate medical procedures, and the output is audio data. The PyAudio library is used to generate audio and provide it to the user. For example, specific instructions such as "Please press on the chest in a steady rhythm" are provided by voice.

[1978] Step 4:

[1979] The terminal uses sensors to monitor the patient's vital signs, such as heart rate, body temperature, and respiratory rate, in real time, and transmits the data to a server. The input is sensor data, and the output is the vital signs transmitted to the server. The server analyzes this data and, if an abnormality is detected, instructs the appropriate countermeasures.

[1980] Step 5:

[1981] The server uses a machine learning model to suggest optimal medical treatment based on patient data. The input is real-time patient data, and the output is a recommendation for the optimal treatment method. This recommendation is then notified to the user via a smartphone application.

[1982] Step 6:

[1983] The device connects to medical professionals in real time via the Internet and receives support. The input is a connection request to a medical professional, and the output is the establishment of the connection. The server relays this connection and provides instructions from the professional to the user in real time.

[1984] Step 7:

[1985] The server uses a cryptography library to encrypt and securely store patient data. The input is patient data, and the output is encrypted data. Access control is performed so that only authorized users can access the data.

[1986] Step 8:

[1987] The server uses the TextBlob library to analyze the user's speech and recognize their emotional state. The input is the user's speech text, and the output is an evaluation of their emotional state. For example, if the user says, "I'm very nervous," the server determines that the user is nervous. Please use gentler words, and adjusts the tone of the response accordingly.

[1988] Step 9:

[1989] The device integrates the information generated at each step and provides real-time feedback to the user through a smartphone application. The input is diagnosis results and instructions from the server, and the output is notifications to the user. For example, it displays image recognition results, voice guidance, suggested medical procedures, instructions from specialists, and other information to help the user respond appropriately.

[1990] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1991] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1992] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1993] [Fourth embodiment]

[1994] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1995] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1996] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1997] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1998] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1999] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2000] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2001] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2002] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2003] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2004] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2005] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2006] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2007] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element works together to assist in emergency medical treatment. Specific embodiments for carrying out the present invention will be described below.

[2008] Natural Language Processing Implementation

[2009] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question and generate a response such as, "The patient's heart rate is normal." The response is conveyed to the user via the terminal in voice or text.

[2010] Image Recognition Implementation

[2011] Image data acquired from cameras and sensors is analyzed using image recognition techniques to diagnose a patient's condition. For example, a device takes an image of a patient, and the server analyzes the image using image recognition technology to predict the patient's condition from the patient's facial expression and appearance. The diagnosis result is notified to the user visually and audibly.

[2012] Implementing voice guidance

[2013] The voice guidance means guides the user through emergency medical procedures by voice. For example, if cardiopulmonary resuscitation is required, the server issues specific instructions such as "Please press on the chest in a steady rhythm" through the voice guidance program and conveys this to the user via the terminal.

[2014] Implementation of artificial intelligence for medical procedure assistance

[2015] AI tools utilize machine learning and deep learning to have the knowledge and judgment necessary to assist medical procedures. The server collects patient data in real time and uses AI models to determine the optimal treatment method. The device then instructs the user on the treatment method and encourages appropriate action.

[2016] Sensor Implementation

[2017] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the collected data and immediately generates appropriate instructions if an abnormality is detected.

[2018] Implementing Remote Assistance

[2019] The remote assistance method allows a user to connect with a medical professional in real time and receive guidance. For example, a device can establish a connection and allow the medical professional to check the patient's condition in real time. The server receives instructions from the professional and relays them to the user via the device. This allows for the management of acute symptoms.

[2020] Implementing encryption and access control

[2021] Encryption and access control are technologies used to ensure the safety of patient data. Patient data collected by the device is encrypted before being sent to the server. The server securely stores the encrypted data and ensures that only authorized users can access the data based on the access control policy. Security patches are also regularly applied to maintain the safety of the system.

[2022] Specific examples

[2023] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[2024] 1. Sensor detection

[2025] Terminal: The sensor detects abnormalities in the patient and notifies the server.

[2026] 2. CPR instructions

[2027] Server: Uses image recognition technology to check the patient's condition and determine that they have undergone cardiac arrest.

[2028] Terminal: Based on instructions from the server, the terminal provides voice and on-screen guidance on CPR procedures to the user.

[2029] User: Begin CPR as instructed.

[2030] 3. Providing remote assistance

[2031] Device: Connect with medical professionals and receive real-time guidance.

[2032] Server: Transmits the expert's instructions to the device and provides feedback to the user.

[2033] User: Perform additional medical procedures under the guidance of a professional.

[2034] 4. Ensuring safety

[2035] Terminal: Patient data is encrypted and securely transmitted to the server.

[2036] Server: Encrypts and stores data and applies access controls.

[2037] In this way, the present invention aims to support the implementation of appropriate medical procedures in emergencies even without specialized knowledge, thereby reducing the risk of sudden death.

[2038] The processing flow will be explained below.

[2039] Natural Language Processing Implementation

[2040] Step 1:

[2041] Terminal: The user types a question into the microphone, for example, "How is the patient doing?"

[2042] Step 2:

[2043] Terminal: Converts voice data into text and sends it to the server.

[2044] Step 3:

[2045] Server: The received text data is input into a natural language processing model and the question content is analyzed.

[2046] Step 4:

[2047] Server: Generates appropriate responses to questions and sends them to the terminal in text format.

[2048] Step 5:

[2049] Terminal: Converts the received response into speech and provides it to the user. The speech response is, "The patient's heart rate is normal."

[2050] Image Recognition Implementation

[2051] Step 1:

[2052] Terminal: Captures images of the patient through a camera and sends the image data to the server.

[2053] Step 2:

[2054] Server: Inputs the received image data into an image recognition model to predict the patient's condition.

[2055] Step 3:

[2056] Server: Generates a diagnosis of the medical condition and determines the appropriate course of treatment.

[2057] Step 4:

[2058] Server: Sends instructions on how to proceed to the terminal.

[2059] Step 5:

[2060] Terminal: The user is notified of the diagnosis and treatment method visually and audibly. "The patient is unconscious. Please begin CPR." is displayed and audible.

[2061] Implementing voice guidance

[2062] Step 1:

[2063] Server: Generates appropriate medical procedures based on the patient's condition.

[2064] Step 2:

[2065] Server: Sends the generated procedure information to the terminal.

[2066] Step 3:

[2067] Device: Provides instructions to the user through a voice guidance system, such as "Compress the chest about 5 cm at a rate of 100 times per minute."

[2068] Step 4:

[2069] User: Follow the audio prompts to perform CPR.

[2070] Implementation of artificial intelligence for medical procedure assistance

[2071] Step 1:

[2072] Terminal: Collects the patient's vital data (e.g., heart rate, blood pressure) from sensors and sends it to the server.

[2073] Step 2:

[2074] Server: Inputs collected vital data into an artificial intelligence model to determine appropriate treatment.

[2075] Step 3:

[2076] Server: Based on the judgment results, generates appropriate treatment procedures and sends them to the terminal.

[2077] Step 4:

[2078] Terminal: Provides the user with visual and audio instructions for appropriate treatment. "Patient's heart rate is dropping. Prepare oxygen for inhalation."

[2079] Sensor Implementation

[2080] Step 1:

[2081] Device: Uses sensors to capture real-time vital data such as the patient's heart rate, temperature, and respiratory rate.

[2082] Step 2:

[2083] Terminal: Sends the acquired data to the server.

[2084] Step 3:

[2085] Server: Analyzes the received data and determines whether there are any abnormalities.

[2086] Step 4:

[2087] Server: If an abnormality is detected, it determines the appropriate response method and sends instruction information to the terminal.

[2088] Step 5:

[2089] Device: Notifies the user of the abnormality and how to respond. "An abnormal heart rate has been detected. Please begin CPR immediately."

[2090] Implementing Remote Assistance

[2091] Step 1:

[2092] Terminal: When an abnormality in the patient is detected, the remote assistance function is activated and sends a request to the server to establish a connection with a medical professional.

[2093] Step 2:

[2094] Server: Establishes connection with medical professional services and transmits patient status in real time.

[2095] Step 3:

[2096] Server: Receives instructions from medical professionals and sends them to the device.

[2097] Step 4:

[2098] Terminal: Provides audio and visual instructions to the user from a specialist. "Please put an oxygen mask on the patient. Detailed instructions are below."

[2099] Step 5:

[2100] User: Follow the instructions of a professional and take appropriate measures.

[2101] Implementing encryption and access control

[2102] Step 1:

[2103] Terminal: Collects patient data (e.g., vital data, diagnostic results) and encrypts the data using an encryption algorithm.

[2104] Step 2:

[2105] Device: Sends encrypted data to the server.

[2106] Step 3:

[2107] Server: Receives the encrypted data and stores it in a secure database.

[2108] Step 4:

[2109] Server: Authenticates users accessing data and enforces access control policies.

[2110] Step 5:

[2111] Server: Regularly apply security patches to keep your systems secure.

[2112] The above is the flow of specific processing steps for each function, which will realize a system that supports appropriate responses at the scene of emergency medical care, even without specialized knowledge.

[2113] Example 1

[2114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2115] In modern society, rapid and accurate responses are required in emergency medical situations, but appropriate treatment is often not provided due to the absence of experts or lack of knowledge. Furthermore, in order to accurately grasp the patient's condition and provide the necessary treatment, a wide range of technologies and knowledge must be integrated. In particular, the lack of real-time monitoring of patient conditions, accurate diagnosis of illness, and appropriate medical guidance is a problem.

[2116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2117] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for analyzing user questions and generating appropriate responses, means for analyzing image data acquired by a camera and predicting the patient's condition, means for providing audio and visual guidance for medical procedure procedures, and means for encrypting and storing collected patient data, thereby enabling prompt and appropriate responses in emergency medical situations.

[2118] "Natural language processing means" refers to means for communicating with users in natural language and for understanding and analyzing questions and instructions.

[2119] "Image recognition means" is a means for analyzing image data acquired by a camera or sensor and identifying the state and characteristics of an object.

[2120] The "audio guidance means" is a means for providing instructions and information to the user by voice.

[2121] "Artificial intelligence means" refers to means that use machine learning and deep learning to assist medical procedures and determine the most appropriate treatment methods.

[2122] The "sensor means" is a means for monitoring the physical condition of a patient in real time and collecting the data.

[2123] "Remote assistance means" means a means of connecting with a medical professional in real time to receive specialized guidance and support.

[2124] "Encryption and access control means" means means for encrypting patient data and ensuring that only authorized users can access the data.

[2125] The "means for analyzing questions and generating responses" refers to means for analyzing voice or text questions entered by a user and generating appropriate responses.

[2126] The "means for analyzing image data and predicting the condition" is a means for analyzing images of a patient taken by a camera and predicting the condition of the patient.

[2127] The "means for providing guidance on the procedure of medical treatment" is a means for providing the user with audio and visual guidance on the specific procedure of emergency medical treatment.

[2128] "Means for encrypting and storing data" refers to means for encrypting collected patient data and storing it securely.

[2129] The present invention is a system including a natural language processing means, an image recognition means, a voice guidance means, an artificial intelligence means, a sensor means, a remote assistance means, and an encryption and access control means, and each element cooperates to assist emergency medical treatment. The embodiments of the present invention will be described in detail below.

[2130] Natural language processing implementation:

[2131] The server uses natural language processing to analyze questions or instructions entered by the user via voice or text and generate an appropriate response. For example, if a user asks, "How is the patient's condition?", the server uses natural language processing technology to understand the question and generates a response such as, "The patient's heart rate is normal." This response is then communicated to the user via the terminal via voice or text.

[2132] Image recognition implementation:

[2133] To understand the patient's condition, the device sends image data captured by the camera to a server. The server then analyzes the image data using image recognition tools and predicts the patient's condition based on the patient's facial expression and appearance. The diagnosis results are notified to the user visually and audibly. For example, the server checks for abnormalities in the patient's complexion or the presence of external injuries.

[2134] Implementing audio guidance methods:

[2135] The server uses the voice guidance means to guide the user through the procedures for emergency medical treatment. For example, if cardiopulmonary resuscitation is required, specific instructions such as "Please push on the chest in a steady rhythm" are provided by voice. These instructions are transmitted to the user through the terminal.

[2136] Implementation of artificial intelligence means:

[2137] The server uses machine learning and deep learning techniques to analyze patient data in real time and determine the optimal treatment. The device then instructs the user on this treatment and encourages appropriate medical intervention. For example, an artificial intelligence model could be used to detect abnormal heart rates or respiratory arrest and generate instructions for immediate response.

[2138] Sensor means implementation:

[2139] The device has sensors that monitor the patient's heart rate, body temperature, respiratory rate, and other data in real time. This data is sent to a server, which analyzes the data and generates immediate instructions if an abnormality is detected. For example, if a sudden change in heart rate is detected, it will immediately issue instructions for CPR.

[2140] Implementing remote assistance methods:

[2141] The terminal has a remote assistance means to connect to a medical professional in real time and receive appropriate guidance. The medical professional checks the patient's condition and provides necessary guidance. The server receives the expert's instructions and relays them to the user, allowing the user to perform accurate treatment based on the expert's guidance.

[2142] Implementing encryption and access control measures:

[2143] Patient data collected by the device is encrypted before being sent to the server, where it is stored and access is controlled to ensure only authorized users can access it. Security patches are also regularly applied to maintain safety.

[2144] Examples:

[2145] As a concrete example, consider a situation where a cardiac arrest occurs in a public place. In this case, the system operates as follows:

[2146] 1. A user uses a device to ask, "How is the patient doing?"

[2147] 2. The device takes an image of the patient with its camera and sends it to the server.

[2148] 3. The server uses image recognition to diagnose the patient's condition and determine that cardiopulmonary resuscitation is necessary.

[2149] 4. The server generates instructions using voice guidance means, such as "Please press the chest at a regular rhythm," and conveys this to the user via the terminal.

[2150] 5. Use remote assistance methods to connect with medical professionals for additional guidance in real time.

[2151] 6. The data collected by the device is encrypted and securely sent to the server.

[2152] An example of a prompt sentence could be something like, "Please explain how the system would respond if someone had a sudden cardiac arrest in the park."

[2153] This enables the system to respond quickly and accurately in emergencies and support appropriate medical treatment.

[2154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2155] Step 1:

[2156] The user enters a question.

[2157] Specific behavior:

[2158] User: Ask via voice or text, "How is the patient doing?"

[2159] Device: Converts speech to text and sends the data to the server.

[2160] Input: User's voice or text question.

[2161] Output: Question data in text format.

[2162] Step 2:

[2163] The server analyzes the user's question and generates a response.

[2164] Specific behavior:

[2165] Server: Uses natural language processing techniques to analyze the user's question and generate an appropriate response.

[2166] Server: Generates a response, for example, "The patient's heart rate is normal."

[2167] Input: The question data submitted by the user in text format.

[2168] Output: The response data in text format.

[2169] Step 3:

[2170] The user is notified of the response from the server.

[2171] Specific behavior:

[2172] Terminal: The response data from the server is converted into voice using speech synthesis technology and notified to the user.

[2173] Terminal: Also displayed as text data.

[2174] Input: Response data sent from the server in text format.

[2175] Output: Voice and text response notification.

[2176] Step 4:

[2177] Image data is acquired to confirm the patient's condition.

[2178] Specific behavior:

[2179] Device: Takes images of the patient's face and body using a camera.

[2180] Terminal: Sends the captured image data to the server.

[2181] Input: Patient images.

[2182] Output: Image data sent to the server.

[2183] Step 5:

[2184] The server analyzes the image data and predicts the condition of the disease.

[2185] Specific behavior:

[2186] Server: Uses image recognition technology to analyze a patient's facial expression, appearance, and possible injuries to predict their condition.

[2187] Server: Generates an analysis result, for example, "The patient may be unconscious."

[2188] Input: Image data sent from the device.

[2189] Output: Predicted disease state.

[2190] Step 6:

[2191] The prediction results are notified to the user.

[2192] Specific behavior:

[2193] Terminal: Notifies the user of predicted medical conditions via voice or text.

[2194] Input: Prediction results of the disease condition sent from the server.

[2195] Output: Prediction result notification in audio and text format.

[2196] Step 7:

[2197] Provides guidance on emergency medical procedures.

[2198] Specific behavior:

[2199] User: Ask "What do I do?"

[2200] Server: Analyzes the situation and determines the medical intervention required.

[2201] Server: For example, it determines that cardiopulmonary resuscitation is necessary and generates instructions such as "Push on the chest in a steady rhythm."

[2202] Terminal: Provides audio and visual instructions to the user.

[2203] Input: Questions from the user and patient data.

[2204] Output: Medical procedure instructions in audio and visual format.

[2205] Step 8:

[2206] Offer remote assistance.

[2207] Specific behavior:

[2208] Device: Establish a connection with a medical professional and receive guidance in real time.

[2209] Server: Receives expert instructions and conveys them to the user in an appropriate manner.

[2210] User: Follow the guidance of a professional and perform appropriate medical procedures.

[2211] Input: Patient status and professional instructions.

[2212] Output: Real-time expert feedback.

[2213] Step 9:

[2214] The patient data is encrypted and sent to the server.

[2215] Specific behavior:

[2216] Terminal: Encrypts patient data (heart rate, images, prediction results, etc.).

[2217] Device: Sends encrypted data to the server.

[2218] Input: Patient data.

[2219] Output: The encrypted data.

[2220] Step 10:

[2221] The server stores encrypted data and controls access.

[2222] Specific behavior:

[2223] Server: Collected data is stored in encrypted form.

[2224] Server: Ensure that only authenticated users can access the data.

[2225] Servers: Regularly apply security patches to maintain security.

[2226] Input: Encrypted patient data.

[2227] Output: Stored data and access logs.

[2228] (Application example 1)

[2229] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2230] Conventional emergency medical intervention systems have limitations in their ability to quickly reach the scene and accurately grasp the patient's condition in real time. Systems utilizing autonomous vehicles, in particular, lack the advanced technology necessary to provide appropriate medical intervention in an emergency. Another problem is the difficulty of providing accurate instructions while connecting with medical professionals in real time. The present invention aims to solve these problems and provide an advanced emergency medical intervention support system using autonomous vehicles to reduce the risk of sudden patient death.

[2231] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2232] In this invention, the server includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, means for automatically arriving at the emergency scene, means for diagnosing the patient's condition using an in-vehicle camera, and means for assisting medical procedures in real time within the autonomous vehicle, thereby enabling rapid arrival at the emergency scene and accurate medical procedures and real-time collaboration with medical professionals at the scene.

[2233] "Natural language processing means" refers to means that enables communication with the user in natural language.

[2234] "Image recognition means" refers to a means for analyzing image data acquired from a camera or sensor and diagnosing the patient's condition.

[2235] The "audio guidance means" is a means for providing the user with audio guidance on the specific steps of a medical procedure.

[2236] "Artificial intelligence means to assist medical procedures" refers to means that utilize machine learning and deep learning to acquire the knowledge and judgment capabilities necessary for medical procedures.

[2237] The "sensor means" is a means for monitoring the patient's physical condition, such as heart rate, body temperature, and respiratory rate, in real time.

[2238] "Remote assistance means" means a means for connecting with a medical professional in real time for guidance.

[2239] "Encryption and access control measures" means measures that encrypt patient data to ensure its security and ensure that only authorized users can access it.

[2240] "Means for arriving at an emergency scene using autonomous driving" means means that enable a vehicle to quickly arrive at an emergency scene using autonomous driving technology.

[2241] "Image recognition means for diagnosing a patient's condition using an in-vehicle camera" refers to a means for diagnosing a patient's condition by analyzing images taken by a camera installed inside an autonomous vehicle.

[2242] "Means for assisting medical procedures in real time within an autonomous vehicle" means means for assisting medical procedures in real time within an autonomous vehicle.

[2243] The present invention is an advanced emergency medical intervention assistance system using an autonomous vehicle, which integrates various technologies, including natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical intervention, sensors means, remote assistance means, encryption and access control means, means for autonomously driving to arrive at the emergency scene, image recognition means for diagnosing the patient's condition using an in-vehicle camera, and means for providing real-time medical intervention assistance within the autonomous vehicle.

[2244] Autonomous driving to emergency scene

[2245] The server receives emergency calls and uses GPS navigation technology to determine the optimal route to the scene. It uses the Google Maps API to analyze traffic conditions in real time and select the optimal route. The autonomous vehicle then automatically follows this route.

[2246] Natural language processing tools

[2247] The server uses natural language processing models (e.g., the BERT model) to communicate with emergency responders in natural language. This allows the user to ask a question such as "How is the patient doing?" and the system understands the question and generates a response such as "The patient's heart rate is normal." This response is then conveyed to the user via voice or text.

[2248] Image Recognition Method

[2249] The server analyzes image data captured by the vehicle's camera using a convolutional neural network (CNN) to diagnose the patient's condition. This data is securely transmitted to the server, and the results of the analysis are notified to the user.

[2250] Audio guidance means

[2251] The server uses a voice guidance means to provide voice instructions on specific steps of medical procedures. For example, if cardiopulmonary resuscitation is required, the server uses a technology that provides voice instructions such as "Push on the chest in a steady rhythm."

[2252] Remote Assistance Methods

[2253] The server uses a video conferencing system (e.g., Zoom API) to connect with medical professionals in the autonomous vehicle in real time, allowing the medical professionals to monitor the situation on-site and provide instructions.

[2254] Sensor means

[2255] The server uses sensors to monitor heart rate, body temperature, breathing rate, etc. in real time. This data is collected by the vehicle's in-vehicle system and transmitted to the server. If an abnormality is detected, appropriate instructions are generated immediately.

[2256] Encryption and Access Control Measures

[2257] The server uses a method for encrypting and controlling access to patient data, using AES-256 data encryption and OAuth user authentication to ensure data security.

[2258] Specific examples of emergencies

[2259] For example, consider a sudden cardiac arrest in a public place. In this case, the system operates as follows:

[2260] 1. An autonomous vehicle receives an emergency call and arrives at the scene via the optimal route.

[2261] 2. Diagnose the patient's condition using image recognition means and provide specific instructions to the user via voice guidance.

[2262] 3. An example of a prompt for reproduction is the text, "As an example of the use of an emergency medical guidance system, please tell us how to respond to a patient in cardiac arrest."

[2263] Program processing

[2264] The server connects various technologies to efficiently support emergency medical procedures. The system integrates autonomous vehicle navigation, natural language processing, image recognition, voice guidance, remote assistance, and security management to ensure a fast and accurate response in emergencies.

[2265] Operation of this system enables rapid arrival at the emergency site and appropriate medical treatment on the spot, significantly reducing the risk of sudden patient death.

[2266] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2267] Step 1:

[2268] The server receives emergency calls. It receives emergency call data (the caller's location information and the details of the emergency) as input, and calculates the optimal route as output. Data processing involves using a GPS navigation system (e.g., Google Maps API) to analyze real-time traffic information and generate the optimal route to quickly reach the scene. The server then sends the generated route information to the autonomous vehicle.

[2269] Step 2:

[2270] The autonomous vehicle (terminal) begins autonomous driving towards the site based on the route information received from the server. The input is route information, and the output is the execution of autonomous driving to the site. Specifically, the vehicle's autonomous driving system controls the vehicle according to GPS information, and drives while recognizing traffic lights and obstacles.

[2271] Step 3:

[2272] When the device arrives at the scene, it uses natural language processing to begin communicating with the user (emergency responder). The input is the user's question or instruction, and the output is a response to the user. For data processing, a natural language processing model (e.g., BERT) is used to analyze the meaning of the question and generate an appropriate response. The server transmits the generated response from the device to the user via voice or text.

[2273] Step 4:

[2274] The server uses the device's camera to diagnose the patient's condition on-site using image recognition. The input is camera image data, and the output is the diagnosis result. Data processing involves image analysis using a convolutional neural network (CNN) to diagnose the patient's external symptoms. The diagnosis result is transmitted from the server to the user via the device.

[2275] Step 5:

[2276] The terminal uses voice guidance generated by the server to instruct the user on the steps of the medical procedure. The input is the diagnosis result and the output is the voice guidance. The server executes the voice guidance program, generates specific instructions, and transmits them to the user as voice from the terminal. For example, the instruction may be "Please press on the chest at a regular rhythm."

[2277] Step 6:

[2278] The device connects to medical professionals in real time via a server to provide remote assistance. The input is a connection request, and the output is the establishment of a video call connection. For data processing, a video conferencing system (e.g., Zoom API) is used to send and receive video and audio in real time to receive instructions from medical professionals. The instructions from the professional are then communicated to the user via the device.

[2279] Step 7:

[2280] The server securely stores all collected patient data using encryption and access control measures. The input is collected data, and the output is encrypted data. Data processing involves encrypting the data using the AES-256 encryption algorithm and controlling access via OAuth. The data is securely stored on the server and can only be accessed by authorized users.

[2281] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2282] The present invention is a system that includes natural language processing means, image recognition means, voice guidance means, artificial intelligence means for assisting medical procedures, sensor means for monitoring the patient's condition in real time, remote assistance means for connecting with medical professionals in real time, means for encrypting and controlling access to patient data, and an emotion engine for recognizing the user's emotions. The purpose is for each element to work together to assist emergency medical procedures and effectively support patient life-saving efforts.

[2283] Natural Language Processing Implementation

[2284] Using natural language processing means, it is possible to communicate with users in natural language. For example, if a user asks, "How is the patient's condition?", the server will use natural language processing technology to understand the question, generate a response such as, "The patient's heart rate is normal," and convey this to the user via voice or text via the terminal.

[2285] Image Recognition Implementation

[2286] Image recognition is used to analyze image data acquired from cameras and sensors and diagnose the patient's condition. For example, the device takes an image of the patient, and the server analyzes the image using image recognition technology. Based on the results of this analysis, the user is notified of the diagnosis and necessary treatment methods visually and audibly.

[2287] Implementing voice guidance

[2288] The voice guidance means guides the user through emergency medical procedures by voice. For example, the server generates cardiopulmonary resuscitation procedures and issues specific instructions such as "Please press on the chest in a steady rhythm" to the user via the terminal.

[2289] Implementation of artificial intelligence for medical procedure assistance

[2290] AI tools utilize machine learning and deep learning to have the knowledge and judgment capabilities necessary to assist medical procedures. The server collects patient data in real time, uses AI models to determine the optimal treatment method, and communicates that treatment method to the user via the device.

[2291] Sensor Implementation

[2292] The sensor means monitors the patient's physical condition in real time, such as heart rate, body temperature, respiratory rate, etc. The terminal collects this information and sends it to the server, which analyzes the data and issues appropriate instructions if an abnormality is detected.

[2293] Implementing Remote Assistance

[2294] The remote assistance means has a function for connecting with a medical professional in real time and receiving guidance. For example, the terminal establishes a connection with the professional and allows the professional to check the patient's condition in real time through a server, thereby transmitting instructions from the professional to the user.

[2295] Implementing encryption and access control

[2296] Encryption and access control measures are techniques for ensuring the security of patient data. Patient data collected by devices is encrypted before being sent to a server, which then securely stores the encrypted data and ensures that only authorized users can access it.

[2297] Implementing the Emotion Engine

[2298] The emotion engine analyzes the user's tone of voice and facial expressions to recognize the user's emotional state. It can use the emotion engine to assess the user's stress level and adjust responses accordingly. For example, if the server determines that the user is nervous, it can adjust its instructions to use gentler language and a more soothing tone.

[2299] Specific examples

[2300] As a concrete example, consider a situation where a person suddenly experiences cardiac arrest in a public place. In this case, the device operates as follows:

[2301] 1. Sen...

Claims

1. natural language processing means; Image recognition means; audio guidance means; artificial intelligence means for assisting medical procedures; a sensor means for monitoring the patient's condition in real time; remote assistance means to connect with medical professionals in real time; a means for encrypting and controlling access to patient data; A system including:

2. The system according to claim 1 , further comprising an image recognition means for diagnosing a patient's condition using a deep learning model and prescribing appropriate medical treatment based on the diagnosis result.

3. 2. The system according to claim 1, further comprising means for smoothly communicating with medical practitioners and patients using natural language processing means, collecting detailed information, and providing appropriate medical guidance.

4. 10. The system of claim 1, further comprising a remote assistance means for managing acute conditions by connecting with a medical professional for guidance in real time.

5. 10. The system of claim 1, including security and privacy measures that automatically encrypt patient data, control access, and apply security patches.

Images