system

Abstract

Provide a system. 【Solution means】 Means for receiving various information from a user in the forms of voice, text, image, and video; Means for data-processing the received information according to each form; Means for analyzing the pre-processed information using an artificial intelligence algorithm and generating information for diagnosis and options for treatment; Means for transmitting the generated information for diagnosis and options for treatment to a user terminal; Means for receiving opinions from experts and updating the medical treatment policy; Means for monitoring the health status and detecting abnormalities from voice, video, and image data collected during daily activities; Means for transmitting a notification for promoting a prompt response when an abnormality is detected; A system including the above.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the medical field, due to the increase in patients and the shortage of medical staff, there is a need to speed up diagnosis and treatment and improve accuracy. In this context, it is necessary to provide a system that comprehensively analyzes various forms of medical data and efficiently provides appropriate medical information to medical staff. Such a system aims to reduce the burden on medical staff and improve the quality of medical services for patients by collaborating with doctors in various specialized fields and improving the immediacy and accuracy of data.

Means for Solving the Problems

[0005] This invention provides a system that includes means for receiving diverse data from a user in the form of audio, text, images, and video, and for pre-processing the received data according to each format. It also includes means for analyzing the pre-processed data using an artificial intelligence model to generate diagnostic information and treatment options. Furthermore, it includes means for transmitting the generated diagnostic information and treatment options to a user terminal, receiving feedback from experts, and updating the treatment plan. This system enables the artificial intelligence to select data highly relevant to different specialists, integrate and present information from the latest academic research results and past case databases, thereby improving the efficiency and accuracy of medical care.

[0006] "Audio data" refers to information that records spoken language and sound environments in a format where sound vibrations are converted into digital signals.

[0007] "Text data" refers to digital representations of text information such as sentences and words, and is used in electronic medical records and other medical documents.

[0008] "Image data" refers to visual information represented digitally as a collection of pixels, and includes medical images such as X-rays and MRIs.

[0009] "Video data" refers to a digital representation of moving images, and is used as video information for purposes such as endoscopic examinations and surgical records.

[0010] A "user terminal" is a digital device used by healthcare professionals for inputting patient data and receiving and displaying analysis results.

[0011] An "artificial intelligence model" is a computer program that learns from large amounts of data, recognizes specific patterns, and uses them for diagnosis and prediction.

[0012] "Diagnostic information" refers to information that evaluates a patient's medical condition and summarizes medical opinions, which is used to determine treatment plans.

[0013] "Treatment options" refer to multiple means and methods of medical response to a specific medical condition, providing information to help patients choose the most suitable treatment.

[0014] "Feedback" refers to opinions and evaluations from experts and users, and is information used to improve the system and adjust treatment policies. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Modes for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention provides a system that comprehensively collects and analyzes data in various formats, including audio, text, images, and video data, to support diagnosis and treatment planning in medical settings. The processing of this system's program is described below in natural language.

[0037] First, the terminal receives patient data entered by healthcare professionals. This data includes recorded voice interviews from consultations, textual information from patient charts, X-ray and MRI images, and video data from endoscopic examinations. This data is encrypted and transmitted to the server via secure communication.

[0038] Next, the server preprocesses the received data according to a defined format. For audio data, noise is removed and the data is converted to text using speech recognition technology. For text data, important sections are extracted using text mining technology. Image data is analyzed based on computer vision technology to detect any anomalies. For video data, frame by frame analysis is performed.

[0039] Once preprocessing is complete, the server inputs this data into an artificial intelligence model and begins analysis. Based on the trained medical dataset, the AI ​​evaluates the interpretation of symptoms and the likelihood of disease, generating diagnostic information. Furthermore, treatment options are suggested, and appropriate treatment plans are indicated.

[0040] The generated diagnostic information and treatment suggestions are sent to the user's terminal and made accessible to healthcare professionals. The user, together with a specialist, evaluates the provided information and sends feedback to the server. This feedback is used to determine the next steps in the patient's treatment.

[0041] As a concrete example, consider a case where a patient visits a clinic complaining of abdominal pain. The user records an audio interview and simultaneously acquires image data from a CT scan, inputting it into the terminal. The server analyzes this data, understanding the patient's symptoms from the audio and detecting abnormalities from the CT images. Based on this information, the AI ​​diagnoses a high probability of acute appendicitis and generates a treatment plan recommending immediate surgical intervention. The user can then proceed with treatment quickly based on this information.

[0042] In this way, this system efficiently handles diverse data and supports rapid and accurate diagnosis and treatment planning in medical settings.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The terminal collects patient voice, text, image, and video data entered by healthcare professionals. The data is encrypted for privacy reasons and transmitted to the server via a secure connection.

[0046] Step 2:

[0047] The server receives various types of data sent from the terminal. It performs preprocessing according to the data format, removes background noise from audio data, and converts it into text format using speech recognition.

[0048] Step 3:

[0049] The server performs text mining on the text data. Using natural language processing techniques, it extracts important keywords and organizes information related to symptoms and medical history.

[0050] Step 4:

[0051] The server analyzes image data and uses computer vision algorithms to detect abnormal areas and characteristic patterns. It utilizes specialized deep learning models to identify abnormalities such as tumors and inflammation.

[0052] Step 5:

[0053] The server divides the video data frame by frame and performs similar image analysis on each frame. It highlights any dynamic changes and generates information that is useful for detecting anomalies and monitoring treatment.

[0054] Step 6:

[0055] The server integrates the pre-processed data and performs analysis using an artificial intelligence model. It compares the results with similar past cases, lists potential diagnoses, and presents treatment options.

[0056] Step 7:

[0057] The server sends the generated diagnostic information and treatment suggestions to the user's terminal. Specifically, it provides a report containing an overview of the diagnostic results and details of recommended treatment methods.

[0058] Step 8:

[0059] Users view information from the server on their devices and collaborate with specialists to review the diagnosis. They can send additional feedback to the server as needed for further data analysis and optimization of treatment plans.

[0060] Step 9:

[0061] The user will perform specific medical procedures based on the established treatment plan, including explaining the treatment plan to the patient and arranging the procedures.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In recent years, with the increasing volume of diverse data in medical settings, there has been a growing need for more efficient information integration and analysis. However, current systems struggle to effectively process data in different formats, such as audio, text, images, and video, and to quickly determine diagnoses and treatment plans. This challenge needs to be addressed.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for receiving diverse information from experts in the form of audio, text data, still images, and video; means for processing the received information according to each format; and means for interpreting the processed information using an automated knowledge model to generate diagnostic results and treatment plans. This enables rapid and efficient analysis of diverse data in different formats, allowing for accurate diagnosis and determination of optimal treatment plans in medical settings.

[0067] "Sound" refers to information emitted by humans that travels through the air as sound waves, and is used as data through recording and playback.

[0068] "Text data" refers to information written in language that is represented in digital format, and is a collection of characters and symbols.

[0069] A "still image" is a representation of a single moment of visual information on a flat surface, and is treated as digital data.

[0070] A "moving image format" is a data format that visually represents movement by playing a series of images over time.

[0071] The term "expert" refers to a professional who possesses specialized knowledge and skills in a particular field.

[0072] "Information" refers to meaningful data and knowledge, especially content that provides value to others.

[0073] "Means of receiving" refers to methods or devices used to acquire signals or data from external sources.

[0074] "Means of data processing" refers to methods or techniques used in the process of analyzing received information and converting it into a format suitable for a specific purpose.

[0075] An "automated knowledge model" refers to a virtual model built using artificial intelligence or machine learning techniques, which is a system that automatically makes decisions and inferences based on the input information.

[0076] "Interpreting" refers to the act of understanding the meaning based on given information and deriving relevant conclusions or implications.

[0077] "Diagnosis" refers to a medical evaluation or judgment provided based on analyzed data and expert knowledge.

[0078] A "treatment plan" is a proposed medical response or treatment plan for a specific symptom or disease.

[0079] This invention is a system that integrates and processes various data formats, providing immediacy and accuracy in medical settings. Embodiments are shown below.

[0080] The terminal receives patient information in the form of audio, text data, still images, and video provided by healthcare professionals. Examples include recordings of consultations, medical record information, X-ray and MRI images, and endoscopic examination videos. The received information is encrypted to maintain data confidentiality and security. Common secure communication protocols (e.g., SSL / TLS) are used for encryption.

[0081] Next, the server receives this information and performs preprocessing according to each data format. Audio data is converted to text using a speech recognition engine (e.g., widely known speech recognition technologies) after applying noise reduction techniques. Text data is then processed using natural language processing libraries (e.g., NLTK, spaCy) to extract important information. Image data is analyzed and anomaly detection is performed using computer vision technologies (e.g., OpenCV or deep learning models). Video data is analyzed frame by frame, allowing for the extraction of important scenes.

[0082] The processed information is comprehensively analyzed by an automated knowledge model (generative AI model) on the server. Based on a wide range of medical datasets, this model can comprehensively evaluate a patient's symptoms and generate diagnostic results and treatment plans. This process can improve the quality of medical care in healthcare settings where rapid response is crucial.

[0083] As a concrete example, suppose a patient comes to the hospital complaining of abdominal pain. The user inputs audio data from the medical interview and CT scan images into the terminal. The server analyzes this data comprehensively, derives a diagnosis suggesting the possibility of acute appendicitis, and presents a treatment plan recommending immediate surgical intervention.

[0084] An example of a prompt message might be, "Generate a diagnosis and treatment plan based on the audio interview data and CT scan image data of a patient complaining of abdominal pain." This system strongly supports rapid and highly accurate diagnosis and treatment planning in medical settings.

[0085] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0086] Step 1:

[0087] The terminal receives information in the form of audio, text data, still images, and video provided by healthcare professionals. This input includes audio recordings of patient examinations, transcribed medical record information, examination images (X-rays and MRIs), and videos of endoscopic examinations. Specifically, the terminal encrypts this data to prepare it for secure handling.

[0088] Step 2:

[0089] The terminal transmits received data to the server using a secure communication protocol. The input here is encrypted patient data, and the output is the secure transmission of data to the server. Specifically, the transfer is performed using a communication protocol (e.g., SSL / TLS) to ensure data integrity and privacy.

[0090] Step 3:

[0091] The server preprocesses received patient data according to its format. Audio data is subjected to a noise reduction algorithm and converted to text using speech recognition technology. Text data is analyzed through text mining techniques to extract important keywords. Image data is analyzed using computer vision technology (e.g., OpenCV) to identify signs of abnormality. Video is processed frame by frame, and critical scenes are extracted. Input is raw data in various formats, and output is data in an analyzable format.

[0092] Step 4:

[0093] The server inputs pre-processed data into an automated knowledge model and performs complex analysis. All input data is pre-processed, and based on this, the generating AI model compares it with the patient's symptoms to generate diagnostic results and treatment suggestions. Specifically, the AI ​​uses pattern recognition based on past medical data to improve the accuracy of the diagnosis. The output is a highly accurate diagnostic result and suggestions based on the analysis.

[0094] Step 5:

[0095] The diagnostic results and treatment plan generated by the server are sent back to the user terminal in a secure format. The input is the diagnostic results and treatment plan, and the output is the receipt to the user's device. The terminal prepares to display this information, allowing healthcare professionals and specialists to properly evaluate it.

[0096] Step 6:

[0097] The user consults with a specialist and evaluates the diagnostic results and treatment plan displayed on the terminal. Specifically, this evaluation is sent to the server as feedback data, which is then used for subsequent medical responses and decisions. The input in this step is the displayed diagnostic information, and the output is the feedback data sent to the server.

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] In recent years, with the advancement of an aging society, the increasing burden on caregivers has become a social problem. In particular, there is a need to constantly monitor the health status of residents and respond quickly and appropriately, but in reality, it is difficult to achieve this with limited resources. Therefore, there is a need for a system that can more efficiently monitor health status and immediately detect and respond to abnormalities.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes means for receiving diverse information from a user in the form of voice, text, images, and video; means for processing the received information according to each format; means for analyzing the pre-processed information using an artificial intelligence algorithm to generate information for diagnosis and treatment options; means for monitoring health status from voice, video, and image data collected during daily activities and detecting abnormalities; and means for sending notifications to facilitate a rapid response when an abnormality is detected. This enables efficient monitoring of the health status of residents in care settings and allows for a rapid response in the event of an abnormality.

[0103] "Voice" refers to all spoken language and acoustic information input by the user, and data processing and analysis are performed based on this.

[0104] "Characters" refers to information in document or text format, and the analysis targets are written and electronic data entered by users.

[0105] "Images" refer to photographs or scanned visual information provided by users, and are used for information processing.

[0106] "Video" refers to dynamic visual information, and analysis and monitoring are performed based on the video provided by the user.

[0107] "Diverse information" refers to all different data formats, such as audio, text, images, and video, and is necessary to enable comprehensive information processing.

[0108] "Means of receiving" refers to mechanisms and devices that acquire user information in various forms and use it for subsequent processing.

[0109] "Means of data processing" refers to the technologies and methods used to preprocess received information and convert it into a format that is easy to analyze.

[0110] An "artificial intelligence algorithm" refers to a computational method used to analyze received information and generate optimal diagnostic information and treatment options.

[0111] "Diagnostic information" refers to information that includes the possibility of disease and interpretation of symptoms generated as a result of analysis.

[0112] "Treatment options" refer to recommended countermeasures for a specific health problem, and their implementation is desirable.

[0113] "Means of monitoring health status" refers to technologies and mechanisms that observe the user's daily activities and contribute to the early detection of abnormalities.

[0114] "Means for detecting anomalies" refers to a system that identifies unusual conditions or events, enabling a rapid response.

[0115] "Means of sending notifications to facilitate a rapid response" refers to communication technologies and methods for quickly transmitting information to relevant parties when an anomaly is detected.

[0116] To implement this invention, a system is used that receives diverse information from the user in the form of audio, text, images, and video. The server processes the received information according to its respective format. In this invention, received audio data is denoised and converted into text data using speech recognition technology. Text information is analyzed using text mining technology to extract important parts. Image data is analyzed using computer vision technology to detect anomalies. Video data is analyzed frame by frame.

[0117] The program uses Python for processing, a third-party API for speech recognition, and machine learning libraries such as TENSORFLOW® for image analysis. The user terminal receives the analysis results and displays the outputted diagnostic information and treatment options for evaluation by medical or care professionals. Based on the information provided, the user can send feedback to the server and update the treatment plan.

[0118] As a concrete example, in nursing homes, robots monitor the environment within a room and collect audio and video data to monitor residents' daily activities. This allows for rapid notification if any abnormalities in health conditions are detected. By using a prompt such as, "Analyze the audio data and propose an appropriate care plan if the resident shows discomfort," the AI ​​can suggest the optimal care plan.

[0119] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0120] Step 1:

[0121] The terminal receives audio, text, image, and video information provided by the user. This data is classified and encoded according to its intended use and sent to the server. Input is user data, and output is classified and encoded data sent to the server.

[0122] Step 2:

[0123] The server preprocesses the received data according to its format. Audio data is denoised and converted into text data using a speech recognition API. Text information is then extracted using text mining techniques. Image data is analyzed using computer vision techniques to detect anomalies. Video data is analyzed by dividing it into frames. The input is encoded data, and the output is preprocessed, analyzable data.

[0124] Step 3:

[0125] The server uses the preprocessed data as input to a generating AI model and performs analysis. Based on the trained dataset, the server generates diagnostic information and treatment options. The input is preprocessed data, and the output is diagnostic information and treatment options.

[0126] Step 4:

[0127] The server sends the generated diagnostic information and treatment options to the user terminal. The terminal visualizes the received information, and the user (a medical or care professional) evaluates it. The input is the diagnostic information and treatment options, and the output is the display of the information by the terminal.

[0128] Step 5:

[0129] Users use their devices to view diagnostic information and treatment options, input feedback, and send it to the server. This feedback is used to update treatment plans. The input is the user's feedback, and the output is the updated treatment plan.

[0130] Step 6:

[0131] The server analyzes audio, video, and image data collected through daily activities and monitors for anomalies. If an anomaly is detected, it sends a notification to the terminal to enable prompt action. The input is data from daily activities, and the output is anomaly notifications.

[0132] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0133] This invention provides a medical support system that incorporates an emotion engine that recognizes the user's emotions along with voice, text, image, and video data. The program processing of this system is described below in natural language.

[0134] First, the terminal receives voice data entered by healthcare professionals, text information from patient medical records, medical images, and video data. In addition, an emotion engine analyzes the user's emotional state from voice intonation and facial expression data. This data is encrypted and securely transmitted to the server.

[0135] Based on the received data, the server preprocesses each data format according to its type. Audio data is de-noised and converted to text. Text data is searched for important keywords, and image and video data are subjected to algorithmic detection for anomalies. Simultaneously, the emotion engine quantifies the user's emotional state and provides the results in a format easily understood by healthcare professionals.

[0136] Once the pre-processed data is available, the server performs integrated analysis using an artificial intelligence model to generate diagnostic information and treatment options. The AI ​​compares multiple data points and proposes the optimal treatment plan, while also taking the user's emotional state into consideration. This allows for the provision of information in a format that is more easily accepted by the user.

[0137] The generated diagnostic information, treatment options, and emotion-based feedback are sent to the user's device. The user reviews this information and consults with a specialist to determine the final treatment plan. This feedback process leads to improved communication that takes the patient's emotional state into account.

[0138] For example, if a patient experiencing high stress levels receives treatment, the user's device evaluates the patient's voice and facial expressions, and the emotion engine recognizes this as a high-stress state. In response, the server can recommend a less stressful communication approach and provide treatment plans that require particular attention. As a result, the patient may feel more at ease during treatment, potentially leading to improved treatment outcomes.

[0139] This system makes it possible to improve the quality of medical care through integrated analysis of diverse data and evaluation of users' emotional states.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The terminal collects voice data, text data, image data, and video data entered by healthcare professionals. This includes recordings of patient interviews, entries in electronic medical records, images from CT and MRI scans, and endoscopic images. The terminal also uses sensors to capture the patient's voice tone and facial expressions to acquire data necessary for the emotion engine.

[0143] Step 2:

[0144] The device encrypts the collected data and sends it to the server. Secure communication methods such as SSL / TLS are used to ensure data privacy and security.

[0145] Step 3:

[0146] The server preprocesses each received data according to its format. Noise is removed from audio data, and it is converted into text data using speech recognition technology. Important information is extracted from the text data using natural language processing. Image data is analyzed using deep learning models to detect anomalies. Video data is analyzed frame by frame, and dynamic anomalies are also detected.

[0147] Step 4:

[0148] The server uses an emotion engine to analyze received audio and facial expression data and evaluate the user's emotional state. Emotional states are classified into basic emotional categories such as joy, sadness, anger, fear, surprise, and disgust.

[0149] Step 5:

[0150] The server integrates pre-processed medical data and emotional information, and performs analysis using an artificial intelligence model. Based on the patient's medical history and current data, the AI ​​generates diagnostic information and treatment options. In doing so, it also takes the user's emotional state into consideration and proposes an emotionally sensitive approach to treatment.

[0151] Step 6:

[0152] The server transmits generated diagnostic information, treatment options, and emotional information to the user's terminal. This information is presented in a format that healthcare professionals can use to communicate with patients.

[0153] Step 7:

[0154] The user reviews the received information and works with a specialist to determine the treatment plan. The user also takes emotionally charged information into consideration and adjusts treatment methods and communication approaches accordingly.

[0155] Step 8:

[0156] The user implements specific medical procedures based on the decided treatment plan. They explain the treatment plan to the patient and plan the treatment procedure accordingly.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] Conventional medical support systems have been unable to consider the user's emotional state when generating diagnostic information and treatment options, leading to challenges in patient communication and acceptance of treatment plans. Furthermore, there is a lack of means for healthcare professionals to safely and efficiently handle diverse data, highlighting the need for effective methods to improve the quality of medical care.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes means for encrypting and securely transmitting data, means for quantifying and analyzing emotions using an emotion engine, and means for comprehensively analyzing data using a generative AI model to generate diagnostic information and treatment options that take emotions into consideration. This enables the provision of information that takes the user's emotions into consideration and improves the quality of medical care.

[0162] A "terminal" is a device that receives data in the form of voice, text, images, and video, and is equipped with the function to analyze the user's emotions.

[0163] "Encryption" is the process of converting information from plaintext to ciphertext in order to transmit data securely.

[0164] A "server" is a system that processes received data, generates diagnostic information and treatment options, and transmits them to the user's terminal.

[0165] An "emotion engine" is a program that analyzes a user's emotional state from audio and video and quantifies it.

[0166] A "generative AI model" is an artificial intelligence algorithm that comprehensively analyzes received data to generate diagnostic information and treatment options.

[0167] A "prompt statement" is an instruction statement used to present generated information to the user.

[0168] "Treatment policy" refers to the plan or policy for diagnosis and treatment that healthcare professionals implement for a patient.

[0169] This medical support system aims to generate diagnostic information and treatment options by integrating and analyzing diverse data using terminals, servers, and generative AI models. A specific implementation is shown below.

[0170] The terminal has the ability to receive data in the form of voice, text, images, and video. Healthcare workers input voice data through a microphone and provide text information through the electronic medical record system. Medical devices and cameras are used to acquire patient images and video data. Furthermore, the terminal has an emotion engine that analyzes voice intonation and facial expressions captured in videos to evaluate the emotional state of patients and healthcare workers. The analyzed emotion data is quantified, encrypted for security, and sent to the server.

[0171] The server performs format-appropriate preprocessing on the received data. Specifically, it uses an acoustic processing library to remove noise from audio data and convert it to text. For text data, it extracts important keywords using natural language processing tools. For image and video data, it analyzes them using an anomaly detection algorithm. In addition, it evaluates the patient's emotional state using the quantified results of the emotion engine. Based on the data processed in this way, a generative AI model performs an integrated analysis. This model has the ability to comprehensively analyze diverse data and, further considering emotional states, generate diagnostic information and treatment options.

[0172] Ultimately, the generated diagnostic information, treatment options, and emotion-based feedback are sent to the terminal via prompt messages. The user reviews this information and, in consultation with healthcare professionals, decides on a course of treatment. This system enables communication that takes the patient's emotions into account, and is expected to improve the quality of medical care.

[0173] As a concrete example, consider supporting the treatment of a patient in a high-stress state. The terminal evaluates the patient's voice and facial expressions, and the emotion engine recognizes this state as "high stress." Based on this information, the server recommends treatment strategies that require special attention and communication approaches aimed at stress reduction. For this purpose, prompts such as "We recommend treatment methods that take stress reduction into consideration" are used. This allows the patient to receive treatment with peace of mind, and improves the effectiveness of the treatment.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The terminal receives audio, text, image, and video data from healthcare professionals. Specifically, healthcare professionals input audio data using a microphone, retrieve text data from the electronic medical record system, and capture image and video data using a camera or medical device. The input data is treated as raw data to be prepared for subsequent analysis.

[0177] Step 2:

[0178] The device uses an emotion engine to analyze the intonation of received audio data and facial expressions in video data. Specifically, pitch, speed, and volume are detected from the audio data, and facial muscle movements are analyzed from the video data. This process involves data processing to quantify the user's emotional state. As output, an emotion score is generated.

[0179] Step 3:

[0180] The device encrypts all data before sending it to the server. This encryption process uses security protocols to protect the data. Inputs include processed audio, text, image, and video data, and output is encrypted data packets.

[0181] Step 4:

[0182] The server decrypts the received encrypted data and preprocesses it according to its format. For audio data, it uses an acoustic processing library to remove noise and converts it to text data. For text data, it extracts important keywords using natural language processing tools. For image and video data, it runs anomaly detection algorithms to extract features. The output consists of preprocessed text, a keyword list, and image feature data.

[0183] Step 5:

[0184] The server uses a generative AI model to comprehensively analyze preprocessed data and generate diagnostic information and treatment options. Sentiment scores are also taken into consideration during this analysis. Specifically, the AI ​​algorithm makes the optimal diagnosis based on information from multiple data sources. The output includes specific treatment suggestions and prompt messages.

[0185] Step 6:

[0186] The server sends the generated diagnostic information, treatment options, and prompt messages to the terminal. The prompt messages include emotion-based feedback. The output consists of treatment guidance and feedback information presented to the user. The user uses this information to consult with a healthcare professional and decide on the optimal treatment plan.

[0187] (Application Example 2)

[0188] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0189] In modern care settings, accurately understanding the emotional state of users and providing appropriate care accordingly is a challenging task. Responding to changes in users' emotions is essential for providing higher quality care services and improving user satisfaction.

[0190] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0191] In this invention, the server includes means for analyzing the user's emotional state in real time and providing emotion-based feedback, means for generating support options using a machine learning model, and means for securely transmitting each piece of information to the user's terminal. This enables care workers to quickly provide appropriate responses in accordance with the user's emotions.

[0192] "Sound" refers to the waveform of sound produced by human speech, and is a format used as a medium for transmitting information.

[0193] "Characters" are a set of symbols used to represent linguistic information, and are communicated visually as text.

[0194] An "image" is a two-dimensional representation of visual information, and includes forms such as photographs and illustrations.

[0195] A "video" is a media format that expresses movement by displaying a series of images in accordance with the passage of time.

[0196] "User" refers to the entity that operates or uses the system, and in this context, it may also refer to the person receiving care.

[0197] A "machine learning model" is the structure of an algorithm that automatically discovers patterns and rules based on data and makes predictions and decisions.

[0198] "Support options" refers to a series of countermeasures and action guidelines presented according to the user's situation and needs.

[0199] "User terminal" refers to a device that displays the final generated information and is accessible to the user, including hardware such as smartphones and tablets.

[0200] "Real-time" refers to a mode of operation in which processing and responses are performed simultaneously at the moment an event occurs.

[0201] "Secure transmission" refers to the act of ensuring reliable transmission of data by using encryption and authentication processes to prevent unauthorized access and information leakage during data transfer.

[0202] In implementing this invention, the server utilizes a machine learning model to analyze diverse user data and provide feedback based on emotional state. Specifically, it integrates the user's voice, text, image, and video data on a single platform. This data is collected in real time through user devices such as smart glasses and tablets.

[0203] The server uses Google Cloud's Speech-to-Text API to convert speech data into text and AWS Rekognition to analyze image and video data. Furthermore, a machine learning model using TensorFlow analyzes and quantifies the user's emotional state in real time.

[0204] The analyzed data is securely encrypted and transmitted to the user's device, presenting specific support options to enable caregivers to provide appropriate care based on the user's emotional state. This feedback is visualized on the user's device, supporting the caregivers' activities.

[0205] This technology can improve the quality of care by quickly detecting when a user shows signs of anxiety during a meal and prompting staff to take specific actions in response to those emotions, such as speaking to them in a calm voice.

[0206] Examples of prompt statements for the generated AI model are as follows:

[0207] "This user has been particularly prone to stress lately. Therefore, what would be the best approach when talking to them? Please suggest steps to help them reach a state of maximum relaxation."

[0208] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0209] Step 1:

[0210] The server receives audio data acquired from the user's device. This audio data is input into Google Cloud's Speech-to-Text API, which converts the audio signal into text information. This converted text information is output as initial data for diagnosis.

[0211] Step 2:

[0212] The server receives image and video data collected from the user's device. This visual data is input into AWS Rekognition, where image recognition technology is used to analyze people's facial expressions and movements. This analysis generates numerical data of emotional states, which is then used as input for the next processing stage.

[0213] Step 3:

[0214] The server integrates text data and emotional state data and inputs it into a TensorFlow-based machine learning model. This model matches each data point and analyzes the user's state in detail. The output is predicted support options and specific care approaches.

[0215] Step 4:

[0216] The server sends the generated support options and care approaches to the user's terminal. This displays specific actions and points of caution recommended for the user on the caregiver's terminal screen.

[0217] Step 5:

[0218] Users perform appropriate care actions based on the information presented on their device. This improves the quality of care and enables individualized support tailored to the user's emotional state.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] This invention provides a system that comprehensively collects and analyzes data in various formats, including audio, text, images, and video data, to support diagnosis and treatment planning in medical settings. The processing of this system's program is described below in natural language.

[0236] First, the terminal receives patient data entered by healthcare professionals. This data includes recorded voice interviews from consultations, textual information from patient charts, X-ray and MRI images, and video data from endoscopic examinations. This data is encrypted and transmitted to the server via secure communication.

[0237] Next, the server preprocesses the received data according to a defined format. For audio data, noise is removed and the data is converted to text using speech recognition technology. For text data, important sections are extracted using text mining technology. Image data is analyzed based on computer vision technology to detect any anomalies. For video data, frame by frame analysis is performed.

[0238] Once preprocessing is complete, the server inputs this data into an artificial intelligence model and begins analysis. Based on the trained medical dataset, the AI ​​evaluates the interpretation of symptoms and the likelihood of disease, generating diagnostic information. Furthermore, treatment options are suggested, and appropriate treatment plans are indicated.

[0239] The generated diagnostic information and treatment suggestions are sent to the user's terminal and made accessible to healthcare professionals. The user, together with a specialist, evaluates the provided information and sends feedback to the server. This feedback is used to determine the next steps in the patient's treatment.

[0240] As a concrete example, consider a case where a patient visits a clinic complaining of abdominal pain. The user records an audio interview and simultaneously acquires image data from a CT scan, inputting it into the terminal. The server analyzes this data, understanding the patient's symptoms from the audio and detecting abnormalities from the CT images. Based on this information, the AI ​​diagnoses a high probability of acute appendicitis and generates a treatment plan recommending immediate surgical intervention. The user can then proceed with treatment quickly based on this information.

[0241] In this way, this system efficiently handles diverse data and supports rapid and accurate diagnosis and treatment planning in medical settings.

[0242] The following describes the processing flow.

[0243] Step 1:

[0244] The terminal collects patient voice, text, image, and video data entered by healthcare professionals. The data is encrypted for privacy reasons and transmitted to the server via a secure connection.

[0245] Step 2:

[0246] The server receives various types of data sent from the terminal. It performs preprocessing according to the data format, removes background noise from audio data, and converts it into text format using speech recognition.

[0247] Step 3:

[0248] The server performs text mining on the text data. Using natural language processing techniques, it extracts important keywords and organizes information related to symptoms and medical history.

[0249] Step 4:

[0250] The server analyzes image data and uses computer vision algorithms to detect abnormal areas and characteristic patterns. It utilizes specialized deep learning models to identify abnormalities such as tumors and inflammation.

[0251] Step 5:

[0252] The server divides the video data frame by frame and performs similar image analysis on each frame. It highlights any dynamic changes and generates information that is useful for detecting anomalies and monitoring treatment.

[0253] Step 6:

[0254] The server integrates the pre-processed data and performs analysis using an artificial intelligence model. It compares the results with similar past cases, lists potential diagnoses, and presents treatment options.

[0255] Step 7:

[0256] The server sends the generated diagnostic information and treatment suggestions to the user's terminal. Specifically, it provides a report containing an overview of the diagnostic results and details of recommended treatment methods.

[0257] Step 8:

[0258] Users view information from the server on their devices and collaborate with specialists to review the diagnosis. They can send additional feedback to the server as needed for further data analysis and optimization of treatment plans.

[0259] Step 9:

[0260] The user will perform specific medical procedures based on the established treatment plan, including explaining the treatment plan to the patient and arranging the procedures.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] In recent years, with the increasing volume of diverse data in medical settings, there has been a growing need for more efficient information integration and analysis. However, current systems struggle to effectively process data in different formats, such as audio, text, images, and video, and to quickly determine diagnoses and treatment plans. This challenge needs to be addressed.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for receiving diverse information from experts in the form of audio, text data, still images, and video; means for processing the received information according to each format; and means for interpreting the processed information using an automated knowledge model to generate diagnostic results and treatment plans. This enables rapid and efficient analysis of diverse data in different formats, allowing for accurate diagnosis and determination of optimal treatment plans in medical settings.

[0266] "Sound" refers to information emitted by humans that travels through the air as sound waves, and is used as data through recording and playback.

[0267] "Text data" refers to information written in language that is represented in digital format, and is a collection of characters and symbols.

[0268] A "still image" is a representation of a single moment of visual information on a flat surface, and is treated as digital data.

[0269] A "moving image format" is a data format that visually represents movement by playing a series of images over time.

[0270] The term "expert" refers to a professional who possesses specialized knowledge and skills in a particular field.

[0271] "Information" refers to meaningful data and knowledge, especially content that provides value to others.

[0272] "Means of receiving" refers to methods or devices used to acquire signals or data from external sources.

[0273] "Means of data processing" refers to methods or techniques used in the process of analyzing received information and converting it into a format suitable for a specific purpose.

[0274] An "automated knowledge model" refers to a virtual model built using artificial intelligence or machine learning techniques, which is a system that automatically makes decisions and inferences based on the input information.

[0275] "Interpreting" refers to the act of understanding the meaning based on given information and deriving relevant conclusions or implications.

[0276] "Diagnosis" refers to a medical evaluation or judgment provided based on analyzed data and expert knowledge.

[0277] A "treatment plan" is a proposed medical response or treatment plan for a specific symptom or disease.

[0278] This invention is a system that integrates and processes various data formats, providing immediacy and accuracy in medical settings. Embodiments are shown below.

[0279] The terminal receives patient information in the form of audio, text data, still images, and video provided by healthcare professionals. Examples include recordings of consultations, medical record information, X-ray and MRI images, and endoscopic examination videos. The received information is encrypted to maintain data confidentiality and security. Common secure communication protocols (e.g., SSL / TLS) are used for encryption.

[0280] Next, the server receives this information and performs preprocessing according to each data format. Audio data is converted to text using a speech recognition engine (e.g., widely known speech recognition technologies) after applying noise reduction techniques. Text data is then processed using natural language processing libraries (e.g., NLTK, spaCy) to extract important information. Image data is analyzed and anomaly detection is performed using computer vision technologies (e.g., OpenCV or deep learning models). Video data is analyzed frame by frame, allowing for the extraction of important scenes.

[0281] The processed information is comprehensively analyzed by an automated knowledge model (generative AI model) within the server. This model can comprehensively evaluate a patient's symptoms based on a wide range of medical datasets and generate diagnostic results and treatment plans. Through this process, it is possible to improve the quality of medical treatment in medical settings where rapid response is required.

[0282] As a specific example, there may be a case where a patient comes to the hospital complaining of abdominal pain. The user inputs the voice data of the medical interview and the CT scan images into the terminal. The server analyzes these comprehensively and derives a diagnostic result suggesting the possibility of acute appendicitis, and presents a treatment plan recommending immediate surgical treatment.

[0283] As an example of a prompt sentence, an instruction such as "Please generate a diagnostic result and a treatment plan based on the voice data of the medical interview and the CT scan image data of a patient complaining of abdominal pain" can be considered. This system strongly supports rapid and accurate diagnosis and treatment planning in the medical field.

[0284] The flow of the specific process in Example 1 will be described using FIG. 11.

[0285] Step 1:

[0286] The terminal receives information in the form of voice, text data, still images, and moving images provided by medical staff. This input includes voice recordings related to patient examinations, texturized medical record information, examination images (X-rays, MRIs), and videos of endoscopic examinations. As a specific operation, the terminal encrypts this data and prepares to handle it while maintaining security.

[0287] Step 2:

[0288] The terminal transmits received data to the server using a secure communication protocol. The input here is encrypted patient data, and the output is the secure transmission of data to the server. Specifically, the transfer is performed using a communication protocol (e.g., SSL / TLS) to ensure data integrity and privacy.

[0289] Step 3:

[0290] The server preprocesses received patient data according to its format. Audio data is subjected to a noise reduction algorithm and converted to text using speech recognition technology. Text data is analyzed through text mining techniques to extract important keywords. Image data is analyzed using computer vision technology (e.g., OpenCV) to identify signs of abnormality. Video is processed frame by frame, and critical scenes are extracted. Input is raw data in various formats, and output is data in an analyzable format.

[0291] Step 4:

[0292] The server inputs pre-processed data into an automated knowledge model and performs complex analysis. All input data is pre-processed, and based on this, the generating AI model compares it with the patient's symptoms to generate diagnostic results and treatment suggestions. Specifically, the AI ​​uses pattern recognition based on past medical data to improve the accuracy of the diagnosis. The output is a highly accurate diagnostic result and suggestions based on the analysis.

[0293] Step 5:

[0294] The diagnostic results and treatment plan generated by the server are sent back to the user terminal in a secure format. The input is the diagnostic results and treatment plan, and the output is the receipt to the user's device. The terminal prepares to display this information, allowing healthcare professionals and specialists to properly evaluate it.

[0295] Step 6:

[0296] The user consults with a specialist and evaluates the diagnostic results and treatment plan displayed on the terminal. Specifically, this evaluation is sent to the server as feedback data, which is then used for subsequent medical responses and decisions. The input in this step is the displayed diagnostic information, and the output is the feedback data sent to the server.

[0297] (Application Example 1)

[0298] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0299] In recent years, with the advancement of an aging society, the increasing burden on caregivers has become a social problem. In particular, there is a need to constantly monitor the health status of residents and respond quickly and appropriately, but in reality, it is difficult to achieve this with limited resources. Therefore, there is a need for a system that can more efficiently monitor health status and immediately detect and respond to abnormalities.

[0300] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0301] In this invention, the server includes means for receiving diverse information from a user in the form of voice, text, images, and video; means for processing the received information according to each format; means for analyzing the pre-processed information using an artificial intelligence algorithm to generate information for diagnosis and treatment options; means for monitoring health status from voice, video, and image data collected during daily activities and detecting abnormalities; and means for sending notifications to facilitate a rapid response when an abnormality is detected. This enables efficient monitoring of the health status of residents in care settings and allows for a rapid response in the event of an abnormality.

[0302] "Voice" refers to all spoken language and acoustic information input by the user, and data processing and analysis are performed based on this.

[0303] "Text" refers to information in the form of documents or text, and targets the written or electronic data input by users for analysis.

[0304] "Image" refers to photos or scanned visual information provided by users and is used for information processing.

[0305] "Video" refers to dynamic video information, and analysis and monitoring are performed based on the videos provided by users.

[0306] "Diverse information" is a general term for all different data forms such as audio, text, images, videos, etc., and is necessary to enable comprehensive information processing.

[0307] "Means for receiving" refers to the mechanisms or devices for acquiring various forms of user information and providing it for subsequent processing.

[0308] "Means for data processing" refers to the technologies or methods for preprocessing the received information and converting it into a form easy to analyze.

[0309] "Artificial intelligence algorithm" refers to the computational methods used to analyze the received information and generate optimal diagnostic information and treatment options.

[0310] "Information for diagnosis" refers to the information including the possibility of diseases and the interpretation of symptoms generated as a result of analysis.

[0311] "Options for treatment" refers to the countermeasures recommended for specific health problems and the implementation of which is desired.

[0312] "Means for monitoring health status" refers to the technologies or mechanisms for observing the daily activities of users and contributing to the early detection of abnormalities.

[0313] "Means for detecting anomalies" refers to a system that identifies unusual conditions or events, enabling a rapid response.

[0314] "Means of sending notifications to facilitate a rapid response" refers to communication technologies and methods for quickly transmitting information to relevant parties when an anomaly is detected.

[0315] To implement this invention, a system is used that receives diverse information from the user in the form of audio, text, images, and video. The server processes the received information according to its respective format. In this invention, received audio data is denoised and converted into text data using speech recognition technology. Text information is analyzed using text mining technology to extract important parts. Image data is analyzed using computer vision technology to detect anomalies. Video data is analyzed frame by frame.

[0316] The program uses Python for processing, a third-party API for speech recognition, and machine learning libraries such as TensorFlow for image analysis. The user terminal receives the analysis results and displays the outputted diagnostic information and treatment options for evaluation by medical or care professionals. Based on the information provided, the user can send feedback to the server and update the treatment plan.

[0317] As a concrete example, in nursing homes, robots monitor the environment within a room and collect audio and video data to monitor residents' daily activities. This allows for rapid notification if any abnormalities in health conditions are detected. By using a prompt such as, "Analyze the audio data and propose an appropriate care plan if the resident shows discomfort," the AI ​​can suggest the optimal care plan.

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] The terminal receives audio, text, image, and video information provided by the user. This data is classified and encoded according to its intended use and sent to the server. Input is user data, and output is classified and encoded data sent to the server.

[0321] Step 2:

[0322] The server preprocesses the received data according to its format. Audio data is denoised and converted into text data using a speech recognition API. Text information is then extracted using text mining techniques. Image data is analyzed using computer vision techniques to detect anomalies. Video data is analyzed by dividing it into frames. The input is encoded data, and the output is preprocessed, analyzable data.

[0323] Step 3:

[0324] The server uses the preprocessed data as input to a generating AI model and performs analysis. Based on the trained dataset, the server generates diagnostic information and treatment options. The input is preprocessed data, and the output is diagnostic information and treatment options.

[0325] Step 4:

[0326] The server sends the generated diagnostic information and treatment options to the user terminal. The terminal visualizes the received information, and the user (a medical or care professional) evaluates it. The input is the diagnostic information and treatment options, and the output is the display of the information by the terminal.

[0327] Step 5:

[0328] Users use their devices to view diagnostic information and treatment options, input feedback, and send it to the server. This feedback is used to update treatment plans. The input is the user's feedback, and the output is the updated treatment plan.

[0329] Step 6:

[0330] The server analyzes audio, video, and image data collected through daily activities and monitors for anomalies. If an anomaly is detected, it sends a notification to the terminal to enable prompt action. The input is data from daily activities, and the output is anomaly notifications.

[0331] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0332] This invention provides a medical support system that incorporates an emotion engine that recognizes the user's emotions along with voice, text, image, and video data. The program processing of this system is described below in natural language.

[0333] First, the terminal receives voice data entered by healthcare professionals, text information from patient medical records, medical images, and video data. In addition, an emotion engine analyzes the user's emotional state from voice intonation and facial expression data. This data is encrypted and securely transmitted to the server.

[0334] Based on the received data, the server preprocesses each data format according to its type. Audio data is de-noised and converted to text. Text data is searched for important keywords, and image and video data are subjected to algorithmic detection for anomalies. Simultaneously, the emotion engine quantifies the user's emotional state and provides the results in a format easily understood by healthcare professionals.

[0335] Once the pre-processed data is available, the server performs integrated analysis using an artificial intelligence model to generate diagnostic information and treatment options. The AI ​​compares multiple data points and proposes the optimal treatment plan, while also taking the user's emotional state into consideration. This allows for the provision of information in a format that is more easily accepted by the user.

[0336] The generated diagnostic information, treatment options, and emotion-based feedback are sent to the user's device. The user reviews this information and consults with a specialist to determine the final treatment plan. This feedback process leads to improved communication that takes the patient's emotional state into account.

[0337] For example, if a patient experiencing high stress levels receives treatment, the user's device evaluates the patient's voice and facial expressions, and the emotion engine recognizes this as a high-stress state. In response, the server can recommend a less stressful communication approach and provide treatment plans that require particular attention. As a result, the patient may feel more at ease during treatment, potentially leading to improved treatment outcomes.

[0338] This system makes it possible to improve the quality of medical care through integrated analysis of diverse data and evaluation of users' emotional states.

[0339] The following describes the processing flow.

[0340] Step 1:

[0341] The terminal collects voice data, text data, image data, and video data entered by healthcare professionals. This includes recordings of patient interviews, entries in electronic medical records, images from CT and MRI scans, and endoscopic images. The terminal also uses sensors to capture the patient's voice tone and facial expressions to acquire data necessary for the emotion engine.

[0342] Step 2:

[0343] The device encrypts the collected data and sends it to the server. Secure communication methods such as SSL / TLS are used to ensure data privacy and security.

[0344] Step 3:

[0345] The server preprocesses each received data according to its format. Noise is removed from audio data, and it is converted into text data using speech recognition technology. Important information is extracted from the text data using natural language processing. Image data is analyzed using deep learning models to detect anomalies. Video data is analyzed frame by frame, and dynamic anomalies are also detected.

[0346] Step 4:

[0347] The server uses an emotion engine to analyze received audio and facial expression data and evaluate the user's emotional state. Emotional states are classified into basic emotional categories such as joy, sadness, anger, fear, surprise, and disgust.

[0348] Step 5:

[0349] The server integrates pre-processed medical data and emotional information, and performs analysis using an artificial intelligence model. Based on the patient's medical history and current data, the AI ​​generates diagnostic information and treatment options. In doing so, it also takes the user's emotional state into consideration and proposes an emotionally sensitive approach to treatment.

[0350] Step 6:

[0351] The server transmits generated diagnostic information, treatment options, and emotional information to the user's terminal. This information is presented in a format that healthcare professionals can use to communicate with patients.

[0352] Step 7:

[0353] The user reviews the received information and works with a specialist to determine the treatment plan. The user also takes emotionally charged information into consideration and adjusts treatment methods and communication approaches accordingly.

[0354] Step 8:

[0355] The user implements specific medical procedures based on the decided treatment plan. They explain the treatment plan to the patient and plan the treatment procedure accordingly.

[0356] (Example 2)

[0357] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0358] Conventional medical support systems have been unable to consider the user's emotional state when generating diagnostic information and treatment options, leading to challenges in patient communication and acceptance of treatment plans. Furthermore, there is a lack of means for healthcare professionals to safely and efficiently handle diverse data, highlighting the need for effective methods to improve the quality of medical care.

[0359] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0360] In this invention, the server includes means for encrypting and securely transmitting data, means for quantifying and analyzing emotions using an emotion engine, and means for comprehensively analyzing data using a generative AI model to generate diagnostic information and treatment options that take emotions into consideration. This enables the provision of information that takes the user's emotions into consideration and improves the quality of medical care.

[0361] A "terminal" is a device that receives data in the form of voice, text, images, and video, and is equipped with the function to analyze the user's emotions.

[0362] "Encryption" is the process of converting information from plaintext to ciphertext in order to transmit data securely.

[0363] A "server" is a system that processes received data, generates diagnostic information and treatment options, and transmits them to the user's terminal.

[0364] An "emotion engine" is a program that analyzes a user's emotional state from audio and video and quantifies it.

[0365] A "generative AI model" is an artificial intelligence algorithm that comprehensively analyzes received data to generate diagnostic information and treatment options.

[0366] A "prompt statement" is an instruction statement used to present generated information to the user.

[0367] "Treatment policy" refers to the plan or policy for diagnosis and treatment that healthcare professionals implement for a patient.

[0368] This medical support system aims to generate diagnostic information and treatment options by integrating and analyzing diverse data using terminals, servers, and generative AI models. A specific implementation is shown below.

[0369] The terminal has the ability to receive data in the form of voice, text, images, and video. Healthcare workers input voice data through a microphone and provide text information through the electronic medical record system. Medical devices and cameras are used to acquire patient images and video data. Furthermore, the terminal has an emotion engine that analyzes voice intonation and facial expressions captured in videos to evaluate the emotional state of patients and healthcare workers. The analyzed emotion data is quantified, encrypted for security, and sent to the server.

[0370] The server performs format-appropriate preprocessing on the received data. Specifically, it uses an acoustic processing library to remove noise from audio data and convert it to text. For text data, it extracts important keywords using natural language processing tools. For image and video data, it analyzes them using an anomaly detection algorithm. In addition, it evaluates the patient's emotional state using the quantified results of the emotion engine. Based on the data processed in this way, a generative AI model performs an integrated analysis. This model has the ability to comprehensively analyze diverse data and, further considering emotional states, generate diagnostic information and treatment options.

[0371] Ultimately, the generated diagnostic information, treatment options, and emotion-based feedback are sent to the terminal via prompt messages. The user reviews this information and, in consultation with healthcare professionals, decides on a course of treatment. This system enables communication that takes the patient's emotions into account, and is expected to improve the quality of medical care.

[0372] As a concrete example, consider supporting the treatment of a patient in a high-stress state. The terminal evaluates the patient's voice and facial expressions, and the emotion engine recognizes this state as "high stress." Based on this information, the server recommends treatment strategies that require special attention and communication approaches aimed at stress reduction. For this purpose, prompts such as "We recommend treatment methods that take stress reduction into consideration" are used. This allows the patient to receive treatment with peace of mind, and improves the effectiveness of the treatment.

[0373] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0374] Step 1:

[0375] The terminal receives audio, text, image, and video data from healthcare professionals. Specifically, healthcare professionals input audio data using a microphone, retrieve text data from the electronic medical record system, and capture image and video data using a camera or medical device. The input data is treated as raw data to be prepared for subsequent analysis.

[0376] Step 2:

[0377] The device uses an emotion engine to analyze the intonation of received audio data and facial expressions in video data. Specifically, pitch, speed, and volume are detected from the audio data, and facial muscle movements are analyzed from the video data. This process involves data processing to quantify the user's emotional state. As output, an emotion score is generated.

[0378] Step 3:

[0379] The device encrypts all data before sending it to the server. This encryption process uses security protocols to protect the data. Inputs include processed audio, text, image, and video data, and output is encrypted data packets.

[0380] Step 4:

[0381] The server decrypts the received encrypted data and preprocesses it according to its format. For audio data, it uses an acoustic processing library to remove noise and converts it to text data. For text data, it extracts important keywords using natural language processing tools. For image and video data, it runs anomaly detection algorithms to extract features. The output consists of preprocessed text, a keyword list, and image feature data.

[0382] Step 5:

[0383] The server uses a generative AI model to comprehensively analyze preprocessed data and generate diagnostic information and treatment options. Sentiment scores are also taken into consideration during this analysis. Specifically, the AI ​​algorithm makes the optimal diagnosis based on information from multiple data sources. The output includes specific treatment suggestions and prompt messages.

[0384] Step 6:

[0385] The server sends the generated diagnostic information, treatment options, and prompt messages to the terminal. The prompt messages include emotion-based feedback. The output consists of treatment guidance and feedback information presented to the user. The user uses this information to consult with a healthcare professional and decide on the optimal treatment plan.

[0386] (Application Example 2)

[0387] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0388] In modern care settings, accurately understanding the emotional state of users and providing appropriate care accordingly is a challenging task. Responding to changes in users' emotions is essential for providing higher quality care services and improving user satisfaction.

[0389] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0390] In this invention, the server includes means for analyzing the user's emotional state in real time and providing emotion-based feedback, means for generating support options using a machine learning model, and means for securely transmitting each piece of information to the user's terminal. This enables care workers to quickly provide appropriate responses in accordance with the user's emotions.

[0391] "Sound" refers to the waveform of sound produced by human speech, and is a format used as a medium for transmitting information.

[0392] "Characters" are a set of symbols used to represent linguistic information, and are communicated visually as text.

[0393] An "image" is a two-dimensional representation of visual information, and includes forms such as photographs and illustrations.

[0394] A "video" is a media format that expresses movement by displaying a series of images in accordance with the passage of time.

[0395] "User" refers to the entity that operates or uses the system, and in this context, it may also refer to the person receiving care.

[0396] A "machine learning model" is the structure of an algorithm that automatically discovers patterns and rules based on data and makes predictions and decisions.

[0397] "Support options" refers to a series of countermeasures and action guidelines presented according to the user's situation and needs.

[0398] "User terminal" refers to a device that displays the final generated information and is accessible to the user, including hardware such as smartphones and tablets.

[0399] "Real-time" refers to a mode of operation in which processing and responses are performed simultaneously at the moment an event occurs.

[0400] "Secure transmission" refers to the act of ensuring reliable transmission of data by using encryption and authentication processes to prevent unauthorized access and information leakage during data transfer.

[0401] In implementing this invention, the server utilizes a machine learning model to analyze diverse user data and provide feedback based on emotional state. Specifically, it integrates the user's voice, text, image, and video data on a single platform. This data is collected in real time through user devices such as smart glasses and tablets.

[0402] The server uses Google Cloud's Speech-to-Text API to convert audio data into text and AWS Rekognition to analyze image and video data. Furthermore, a machine learning model using TensorFlow analyzes and quantifies the user's emotional state in real time.

[0403] The analyzed data is securely encrypted and transmitted to the user's device, presenting specific support options to enable caregivers to provide appropriate care based on the user's emotional state. This feedback is visualized on the user's device, supporting the caregivers' activities.

[0404] This technology can improve the quality of care by quickly detecting when a user shows signs of anxiety during a meal and prompting staff to take specific actions in response to those emotions, such as speaking to them in a calm voice.

[0405] Examples of prompt statements for the generated AI model are as follows:

[0406] "This user has been particularly prone to stress lately. Therefore, what would be the best approach when talking to them? Please suggest steps to help them reach a state of maximum relaxation."

[0407] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0408] Step 1:

[0409] The server receives audio data acquired from the user's device. This audio data is input into Google Cloud's Speech-to-Text API, which converts the audio signal into text information. This converted text information is output as initial data for diagnosis.

[0410] Step 2:

[0411] The server receives image and video data collected from the user's device. This visual data is input into AWS Rekognition, where image recognition technology is used to analyze people's facial expressions and movements. This analysis generates numerical data of emotional states, which is then used as input for the next processing stage.

[0412] Step 3:

[0413] The server integrates text data and emotional state data and inputs it into a TensorFlow-based machine learning model. This model matches each data point and analyzes the user's state in detail. The output is predicted support options and specific care approaches.

[0414] Step 4:

[0415] The server sends the generated support options and care approaches to the user's terminal. This displays specific actions and points of caution recommended for the user on the caregiver's terminal screen.

[0416] Step 5:

[0417] Users perform appropriate care actions based on the information presented on their device. This improves the quality of care and enables individualized support tailored to the user's emotional state.

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0419] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0421] [Third Embodiment]

[0422] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0423] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0424] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0426] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0428] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0429] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0430] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0433] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0434] This invention provides a system that comprehensively collects and analyzes data in various formats, including audio, text, images, and video data, to support diagnosis and treatment planning in medical settings. The processing of this system's program is described below in natural language.

[0435] First, the terminal receives patient data entered by healthcare professionals. This data includes recorded voice interviews from consultations, textual information from patient charts, X-ray and MRI images, and video data from endoscopic examinations. This data is encrypted and transmitted to the server via secure communication.

[0436] Next, the server preprocesses the received data according to a defined format. For audio data, noise is removed and the data is converted to text using speech recognition technology. For text data, important sections are extracted using text mining technology. Image data is analyzed based on computer vision technology to detect any anomalies. For video data, frame by frame analysis is performed.

[0437] Once preprocessing is complete, the server inputs this data into an artificial intelligence model and begins analysis. Based on the trained medical dataset, the AI ​​evaluates the interpretation of symptoms and the likelihood of disease, generating diagnostic information. Furthermore, treatment options are suggested, and appropriate treatment plans are indicated.

[0438] The generated diagnostic information and treatment suggestions are sent to the user's terminal and made accessible to healthcare professionals. The user, together with a specialist, evaluates the provided information and sends feedback to the server. This feedback is used to determine the next steps in the patient's treatment.

[0439] As a concrete example, consider a case where a patient visits a clinic complaining of abdominal pain. The user records an audio interview and simultaneously acquires image data from a CT scan, inputting it into the terminal. The server analyzes this data, understanding the patient's symptoms from the audio and detecting abnormalities from the CT images. Based on this information, the AI ​​diagnoses a high probability of acute appendicitis and generates a treatment plan recommending immediate surgical intervention. The user can then proceed with treatment quickly based on this information.

[0440] In this way, this system efficiently handles diverse data and supports rapid and accurate diagnosis and treatment planning in medical settings.

[0441] The following describes the processing flow.

[0442] Step 1:

[0443] The terminal collects patient voice, text, image, and video data entered by healthcare professionals. The data is encrypted for privacy reasons and transmitted to the server via a secure connection.

[0444] Step 2:

[0445] The server receives various types of data sent from the terminal. It performs preprocessing according to the data format, removes background noise from audio data, and converts it into text format using speech recognition.

[0446] Step 3:

[0447] The server performs text mining on the text data. Using natural language processing techniques, it extracts important keywords and organizes information related to symptoms and medical history.

[0448] Step 4:

[0449] The server analyzes image data and uses computer vision algorithms to detect abnormal areas and characteristic patterns. It utilizes specialized deep learning models to identify abnormalities such as tumors and inflammation.

[0450] Step 5:

[0451] The server divides the video data frame by frame and performs similar image analysis on each frame. It highlights any dynamic changes and generates information that is useful for detecting anomalies and monitoring treatment.

[0452] Step 6:

[0453] The server integrates the pre-processed data and performs analysis using an artificial intelligence model. It compares the results with similar past cases, lists potential diagnoses, and presents treatment options.

[0454] Step 7:

[0455] The server sends the generated diagnostic information and treatment suggestions to the user's terminal. Specifically, it provides a report containing an overview of the diagnostic results and details of recommended treatment methods.

[0456] Step 8:

[0457] Users view information from the server on their devices and collaborate with specialists to review the diagnosis. They can send additional feedback to the server as needed for further data analysis and optimization of treatment plans.

[0458] Step 9:

[0459] The user will perform specific medical procedures based on the established treatment plan, including explaining the treatment plan to the patient and arranging the procedures.

[0460] (Example 1)

[0461] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] In recent years, with the increasing volume of diverse data in medical settings, there has been a growing need for more efficient information integration and analysis. However, current systems struggle to effectively process data in different formats, such as audio, text, images, and video, and to quickly determine diagnoses and treatment plans. This challenge needs to be addressed.

[0463] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0464] In this invention, the server includes means for receiving diverse information from experts in the form of audio, text data, still images, and video; means for processing the received information according to each format; and means for interpreting the processed information using an automated knowledge model to generate diagnostic results and treatment plans. This enables rapid and efficient analysis of diverse data in different formats, allowing for accurate diagnosis and determination of optimal treatment plans in medical settings.

[0465] "Sound" refers to information emitted by humans that travels through the air as sound waves, and is used as data through recording and playback.

[0466] "Text data" refers to information written in language that is represented in digital format, and is a collection of characters and symbols.

[0467] A "still image" is a representation of a single moment of visual information on a flat surface, and is treated as digital data.

[0468] A "moving image format" is a data format that visually represents movement by playing a series of images over time.

[0469] The term "expert" refers to a professional who possesses specialized knowledge and skills in a particular field.

[0470] "Information" refers to meaningful data and knowledge, especially content that provides value to others.

[0471] "Means of receiving" refers to methods or devices used to acquire signals or data from external sources.

[0472] "Means of data processing" refers to methods or techniques used in the process of analyzing received information and converting it into a format suitable for a specific purpose.

[0473] An "automated knowledge model" refers to a virtual model built using artificial intelligence or machine learning techniques, which is a system that automatically makes decisions and inferences based on the input information.

[0474] "Interpreting" refers to the act of understanding the meaning based on given information and deriving relevant conclusions or implications.

[0475] "Diagnosis" refers to a medical evaluation or judgment provided based on analyzed data and expert knowledge.

[0476] A "treatment plan" is a proposed medical response or treatment plan for a specific symptom or disease.

[0477] This invention is a system that integrates and processes various data formats, providing immediacy and accuracy in medical settings. Embodiments are shown below.

[0478] The terminal receives patient information in the form of audio, text data, still images, and video provided by healthcare professionals. Examples include recordings of consultations, medical record information, X-ray and MRI images, and endoscopic examination videos. The received information is encrypted to maintain data confidentiality and security. Common secure communication protocols (e.g., SSL / TLS) are used for encryption.

[0479] Next, the server receives this information and performs preprocessing according to each data format. Audio data is converted to text using a speech recognition engine (e.g., widely known speech recognition technologies) after applying noise reduction techniques. Text data is then processed using natural language processing libraries (e.g., NLTK, spaCy) to extract important information. Image data is analyzed and anomaly detection is performed using computer vision technologies (e.g., OpenCV or deep learning models). Video data is analyzed frame by frame, allowing for the extraction of important scenes.

[0480] The processed information is comprehensively analyzed by an automated knowledge model (generative AI model) on the server. Based on a wide range of medical datasets, this model can comprehensively evaluate a patient's symptoms and generate diagnostic results and treatment plans. This process can improve the quality of medical care in healthcare settings where rapid response is crucial.

[0481] As a concrete example, suppose a patient comes to the hospital complaining of abdominal pain. The user inputs audio data from the medical interview and CT scan images into the terminal. The server analyzes this data comprehensively, derives a diagnosis suggesting the possibility of acute appendicitis, and presents a treatment plan recommending immediate surgical intervention.

[0482] An example of a prompt message might be, "Generate a diagnosis and treatment plan based on the audio interview data and CT scan image data of a patient complaining of abdominal pain." This system strongly supports rapid and highly accurate diagnosis and treatment planning in medical settings.

[0483] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0484] Step 1:

[0485] The terminal receives information in the form of audio, text data, still images, and video provided by healthcare professionals. This input includes audio recordings of patient examinations, transcribed medical record information, examination images (X-rays and MRIs), and videos of endoscopic examinations. Specifically, the terminal encrypts this data to prepare it for secure handling.

[0486] Step 2:

[0487] The terminal transmits received data to the server using a secure communication protocol. The input here is encrypted patient data, and the output is the secure transmission of data to the server. Specifically, the transfer is performed using a communication protocol (e.g., SSL / TLS) to ensure data integrity and privacy.

[0488] Step 3:

[0489] The server preprocesses received patient data according to its format. Audio data is subjected to a noise reduction algorithm and converted to text using speech recognition technology. Text data is analyzed through text mining techniques to extract important keywords. Image data is analyzed using computer vision technology (e.g., OpenCV) to identify signs of abnormality. Video is processed frame by frame, and critical scenes are extracted. Input is raw data in various formats, and output is data in an analyzable format.

[0490] Step 4:

[0491] The server inputs pre-processed data into an automated knowledge model and performs complex analysis. All input data is pre-processed, and based on this, the generating AI model compares it with the patient's symptoms to generate diagnostic results and treatment suggestions. Specifically, the AI ​​uses pattern recognition based on past medical data to improve the accuracy of the diagnosis. The output is a highly accurate diagnostic result and suggestions based on the analysis.

[0492] Step 5:

[0493] The diagnostic results and treatment plan generated by the server are sent back to the user terminal in a secure format. The input is the diagnostic results and treatment plan, and the output is the receipt to the user's device. The terminal prepares to display this information, allowing healthcare professionals and specialists to properly evaluate it.

[0494] Step 6:

[0495] The user consults with a specialist and evaluates the diagnostic results and treatment plan displayed on the terminal. Specifically, this evaluation is sent to the server as feedback data, which is then used for subsequent medical responses and decisions. The input in this step is the displayed diagnostic information, and the output is the feedback data sent to the server.

[0496] (Application Example 1)

[0497] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0498] In recent years, with the advancement of an aging society, the increasing burden on caregivers has become a social problem. In particular, there is a need to constantly monitor the health status of residents and respond quickly and appropriately, but in reality, it is difficult to achieve this with limited resources. Therefore, there is a need for a system that can more efficiently monitor health status and immediately detect and respond to abnormalities.

[0499] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0500] In this invention, the server includes means for receiving diverse information from a user in the form of voice, text, images, and video; means for processing the received information according to each format; means for analyzing the pre-processed information using an artificial intelligence algorithm to generate information for diagnosis and treatment options; means for monitoring health status from voice, video, and image data collected during daily activities and detecting abnormalities; and means for sending notifications to facilitate a rapid response when an abnormality is detected. This enables efficient monitoring of the health status of residents in care settings and allows for a rapid response in the event of an abnormality.

[0501] "Voice" refers to all spoken language and acoustic information input by the user, and data processing and analysis are performed based on this.

[0502] "Characters" refers to information in document or text format, and the analysis targets are written and electronic data entered by users.

[0503] "Images" refer to photographs or scanned visual information provided by users, and are used for information processing.

[0504] "Video" refers to dynamic visual information, and analysis and monitoring are performed based on the video provided by the user.

[0505] "Diverse information" refers to all different data formats, such as audio, text, images, and video, and is necessary to enable comprehensive information processing.

[0506] "Means of receiving" refers to mechanisms and devices that acquire user information in various forms and use it for subsequent processing.

[0507] "Means of data processing" refers to the technologies and methods used to preprocess received information and convert it into a format that is easy to analyze.

[0508] An "artificial intelligence algorithm" refers to a computational method used to analyze received information and generate optimal diagnostic information and treatment options.

[0509] "Diagnostic information" refers to information that includes the possibility of disease and interpretation of symptoms generated as a result of analysis.

[0510] "Treatment options" refer to recommended countermeasures for a specific health problem, and their implementation is desirable.

[0511] "Means of monitoring health status" refers to technologies and mechanisms that observe the user's daily activities and contribute to the early detection of abnormalities.

[0512] "Means for detecting anomalies" refers to a system that identifies unusual conditions or events, enabling a rapid response.

[0513] "Means of sending notifications to facilitate a rapid response" refers to communication technologies and methods for quickly transmitting information to relevant parties when an anomaly is detected.

[0514] To implement this invention, a system is used that receives diverse information from the user in the form of audio, text, images, and video. The server processes the received information according to its respective format. In this invention, received audio data is denoised and converted into text data using speech recognition technology. Text information is analyzed using text mining technology to extract important parts. Image data is analyzed using computer vision technology to detect anomalies. Video data is analyzed frame by frame.

[0515] The program uses Python for processing, a third-party API for speech recognition, and machine learning libraries such as TensorFlow for image analysis. The user terminal receives the analysis results and displays the outputted diagnostic information and treatment options for evaluation by medical or care professionals. Based on the information provided, the user can send feedback to the server and update the treatment plan.

[0516] As a concrete example, in nursing homes, robots monitor the environment within a room and collect audio and video data to monitor residents' daily activities. This allows for rapid notification if any abnormalities in health conditions are detected. By using a prompt such as, "Analyze the audio data and propose an appropriate care plan if the resident shows discomfort," the AI ​​can suggest the optimal care plan.

[0517] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0518] Step 1:

[0519] The terminal receives audio, text, image, and video information provided by the user. This data is classified and encoded according to its intended use and sent to the server. Input is user data, and output is classified and encoded data sent to the server.

[0520] Step 2:

[0521] The server preprocesses the received data according to its format. Audio data is denoised and converted into text data using a speech recognition API. Text information is then extracted using text mining techniques. Image data is analyzed using computer vision techniques to detect anomalies. Video data is analyzed by dividing it into frames. The input is encoded data, and the output is preprocessed, analyzable data.

[0522] Step 3:

[0523] The server uses the preprocessed data as input to a generating AI model and performs analysis. Based on the trained dataset, the server generates diagnostic information and treatment options. The input is preprocessed data, and the output is diagnostic information and treatment options.

[0524] Step 4:

[0525] The server sends the generated diagnostic information and treatment options to the user terminal. The terminal visualizes the received information, and the user (a medical or care professional) evaluates it. The input is the diagnostic information and treatment options, and the output is the display of the information by the terminal.

[0526] Step 5:

[0527] Users use their devices to view diagnostic information and treatment options, input feedback, and send it to the server. This feedback is used to update treatment plans. The input is the user's feedback, and the output is the updated treatment plan.

[0528] Step 6:

[0529] The server analyzes audio, video, and image data collected through daily activities and monitors for anomalies. If an anomaly is detected, it sends a notification to the terminal to enable prompt action. The input is data from daily activities, and the output is anomaly notifications.

[0530] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0531] This invention provides a medical support system that incorporates an emotion engine that recognizes the user's emotions along with voice, text, image, and video data. The program processing of this system is described below in natural language.

[0532] First, the terminal receives voice data entered by healthcare professionals, text information from patient medical records, medical images, and video data. In addition, an emotion engine analyzes the user's emotional state from voice intonation and facial expression data. This data is encrypted and securely transmitted to the server.

[0533] Based on the received data, the server preprocesses each data format according to its type. Audio data is de-noised and converted to text. Text data is searched for important keywords, and image and video data are subjected to algorithmic detection for anomalies. Simultaneously, the emotion engine quantifies the user's emotional state and provides the results in a format easily understood by healthcare professionals.

[0534] Once the pre-processed data is available, the server performs integrated analysis using an artificial intelligence model to generate diagnostic information and treatment options. The AI ​​compares multiple data points and proposes the optimal treatment plan, while also taking the user's emotional state into consideration. This allows for the provision of information in a format that is more easily accepted by the user.

[0535] The generated diagnostic information, treatment options, and emotion-based feedback are sent to the user's device. The user reviews this information and consults with a specialist to determine the final treatment plan. This feedback process leads to improved communication that takes the patient's emotional state into account.

[0536] For example, if a patient experiencing high stress levels receives treatment, the user's device evaluates the patient's voice and facial expressions, and the emotion engine recognizes this as a high-stress state. In response, the server can recommend a less stressful communication approach and provide treatment plans that require particular attention. As a result, the patient may feel more at ease during treatment, potentially leading to improved treatment outcomes.

[0537] This system makes it possible to improve the quality of medical care through integrated analysis of diverse data and evaluation of users' emotional states.

[0538] The following describes the processing flow.

[0539] Step 1:

[0540] The terminal collects voice data, text data, image data, and video data entered by healthcare professionals. This includes recordings of patient interviews, entries in electronic medical records, images from CT and MRI scans, and endoscopic images. The terminal also uses sensors to capture the patient's voice tone and facial expressions to acquire data necessary for the emotion engine.

[0541] Step 2:

[0542] The device encrypts the collected data and sends it to the server. Secure communication methods such as SSL / TLS are used to ensure data privacy and security.

[0543] Step 3:

[0544] The server preprocesses each received data according to its format. Noise is removed from audio data, and it is converted into text data using speech recognition technology. Important information is extracted from the text data using natural language processing. Image data is analyzed using deep learning models to detect anomalies. Video data is analyzed frame by frame, and dynamic anomalies are also detected.

[0545] Step 4:

[0546] The server uses an emotion engine to analyze received audio and facial expression data and evaluate the user's emotional state. Emotional states are classified into basic emotional categories such as joy, sadness, anger, fear, surprise, and disgust.

[0547] Step 5:

[0548] The server integrates pre-processed medical data and emotional information, and performs analysis using an artificial intelligence model. Based on the patient's medical history and current data, the AI ​​generates diagnostic information and treatment options. In doing so, it also takes the user's emotional state into consideration and proposes an emotionally sensitive approach to treatment.

[0549] Step 6:

[0550] The server transmits generated diagnostic information, treatment options, and emotional information to the user's terminal. This information is presented in a format that healthcare professionals can use to communicate with patients.

[0551] Step 7:

[0552] The user reviews the received information and works with a specialist to determine the treatment plan. The user also takes emotionally charged information into consideration and adjusts treatment methods and communication approaches accordingly.

[0553] Step 8:

[0554] The user implements specific medical procedures based on the decided treatment plan. They explain the treatment plan to the patient and plan the treatment procedure accordingly.

[0555] (Example 2)

[0556] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0557] Conventional medical support systems have been unable to consider the user's emotional state when generating diagnostic information and treatment options, leading to challenges in patient communication and acceptance of treatment plans. Furthermore, there is a lack of means for healthcare professionals to safely and efficiently handle diverse data, highlighting the need for effective methods to improve the quality of medical care.

[0558] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0559] In this invention, the server includes means for encrypting and securely transmitting data, means for quantifying and analyzing emotions using an emotion engine, and means for comprehensively analyzing data using a generative AI model to generate diagnostic information and treatment options that take emotions into consideration. This enables the provision of information that takes the user's emotions into consideration and improves the quality of medical care.

[0560] A "terminal" is a device that receives data in the form of voice, text, images, and video, and is equipped with the function to analyze the user's emotions.

[0561] "Encryption" is the process of converting information from plaintext to ciphertext in order to transmit data securely.

[0562] A "server" is a system that processes received data, generates diagnostic information and treatment options, and transmits them to the user's terminal.

[0563] An "emotion engine" is a program that analyzes a user's emotional state from audio and video and quantifies it.

[0564] A "generative AI model" is an artificial intelligence algorithm that comprehensively analyzes received data to generate diagnostic information and treatment options.

[0565] A "prompt statement" is an instruction statement used to present generated information to the user.

[0566] "Treatment policy" refers to the plan or policy for diagnosis and treatment that healthcare professionals implement for a patient.

[0567] This medical support system aims to generate diagnostic information and treatment options by integrating and analyzing diverse data using terminals, servers, and generative AI models. A specific implementation is shown below.

[0568] The terminal has the ability to receive data in the form of voice, text, images, and video. Healthcare workers input voice data through a microphone and provide text information through the electronic medical record system. Medical devices and cameras are used to acquire patient images and video data. Furthermore, the terminal has an emotion engine that analyzes voice intonation and facial expressions captured in videos to evaluate the emotional state of patients and healthcare workers. The analyzed emotion data is quantified, encrypted for security, and sent to the server.

[0569] The server performs format-appropriate preprocessing on the received data. Specifically, it uses an acoustic processing library to remove noise from audio data and convert it to text. For text data, it extracts important keywords using natural language processing tools. For image and video data, it analyzes them using an anomaly detection algorithm. In addition, it evaluates the patient's emotional state using the quantified results of the emotion engine. Based on the data processed in this way, a generative AI model performs an integrated analysis. This model has the ability to comprehensively analyze diverse data and, further considering emotional states, generate diagnostic information and treatment options.

[0570] Ultimately, the generated diagnostic information, treatment options, and emotion-based feedback are sent to the terminal via prompt messages. The user reviews this information and, in consultation with healthcare professionals, decides on a course of treatment. This system enables communication that takes the patient's emotions into account, and is expected to improve the quality of medical care.

[0571] As a concrete example, consider supporting the treatment of a patient in a high-stress state. The terminal evaluates the patient's voice and facial expressions, and the emotion engine recognizes this state as "high stress." Based on this information, the server recommends treatment strategies that require special attention and communication approaches aimed at stress reduction. For this purpose, prompts such as "We recommend treatment methods that take stress reduction into consideration" are used. This allows the patient to receive treatment with peace of mind, and improves the effectiveness of the treatment.

[0572] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0573] Step 1:

[0574] The terminal receives audio, text, image, and video data from healthcare professionals. Specifically, healthcare professionals input audio data using a microphone, retrieve text data from the electronic medical record system, and capture image and video data using a camera or medical device. The input data is treated as raw data to be prepared for subsequent analysis.

[0575] Step 2:

[0576] The device uses an emotion engine to analyze the intonation of received audio data and facial expressions in video data. Specifically, pitch, speed, and volume are detected from the audio data, and facial muscle movements are analyzed from the video data. This process involves data processing to quantify the user's emotional state. As output, an emotion score is generated.

[0577] Step 3:

[0578] The device encrypts all data before sending it to the server. This encryption process uses security protocols to protect the data. Inputs include processed audio, text, image, and video data, and output is encrypted data packets.

[0579] Step 4:

[0580] The server decrypts the received encrypted data and preprocesses it according to its format. For audio data, it uses an acoustic processing library to remove noise and converts it to text data. For text data, it extracts important keywords using natural language processing tools. For image and video data, it runs anomaly detection algorithms to extract features. The output consists of preprocessed text, a keyword list, and image feature data.

[0581] Step 5:

[0582] The server uses a generative AI model to comprehensively analyze preprocessed data and generate diagnostic information and treatment options. Sentiment scores are also taken into consideration during this analysis. Specifically, the AI ​​algorithm makes the optimal diagnosis based on information from multiple data sources. The output includes specific treatment suggestions and prompt messages.

[0583] Step 6:

[0584] The server sends the generated diagnostic information, treatment options, and prompt messages to the terminal. The prompt messages include emotion-based feedback. The output consists of treatment guidance and feedback information presented to the user. The user uses this information to consult with a healthcare professional and decide on the optimal treatment plan.

[0585] (Application Example 2)

[0586] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0587] In modern care settings, accurately understanding the emotional state of users and providing appropriate care accordingly is a challenging task. Responding to changes in users' emotions is essential for providing higher quality care services and improving user satisfaction.

[0588] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0589] In this invention, the server includes means for analyzing the user's emotional state in real time and providing emotion-based feedback, means for generating support options using a machine learning model, and means for securely transmitting each piece of information to the user's terminal. This enables care workers to quickly provide appropriate responses in accordance with the user's emotions.

[0590] "Sound" refers to the waveform of sound produced by human speech, and is a format used as a medium for transmitting information.

[0591] "Characters" are a set of symbols used to represent linguistic information, and are communicated visually as text.

[0592] An "image" is a two-dimensional representation of visual information, and includes forms such as photographs and illustrations.

[0593] A "video" is a media format that expresses movement by displaying a series of images in accordance with the passage of time.

[0594] "User" refers to the entity that operates or uses the system, and in this context, it may also refer to the person receiving care.

[0595] A "machine learning model" is the structure of an algorithm that automatically discovers patterns and rules based on data and makes predictions and decisions.

[0596] "Support options" refers to a series of countermeasures and action guidelines presented according to the user's situation and needs.

[0597] "User terminal" refers to a device that displays the final generated information and is accessible to the user, including hardware such as smartphones and tablets.

[0598] "Real-time" refers to a mode of operation in which processing and responses are performed simultaneously at the moment an event occurs.

[0599] "Secure transmission" refers to the act of ensuring reliable transmission of data by using encryption and authentication processes to prevent unauthorized access and information leakage during data transfer.

[0600] In implementing this invention, the server utilizes a machine learning model to analyze diverse user data and provide feedback based on emotional state. Specifically, it integrates the user's voice, text, image, and video data on a single platform. This data is collected in real time through user devices such as smart glasses and tablets.

[0601] The server uses Google Cloud's Speech-to-Text API to convert audio data into text and AWS Rekognition to analyze image and video data. Furthermore, a machine learning model using TensorFlow analyzes and quantifies the user's emotional state in real time.

[0602] The analyzed data is securely encrypted and transmitted to the user's device, presenting specific support options to enable caregivers to provide appropriate care based on the user's emotional state. This feedback is visualized on the user's device, supporting the caregivers' activities.

[0603] This technology can improve the quality of care by quickly detecting when a user shows signs of anxiety during a meal and prompting staff to take specific actions in response to those emotions, such as speaking to them in a calm voice.

[0604] Examples of prompt statements for the generated AI model are as follows:

[0605] "This user has been particularly prone to stress lately. Therefore, what would be the best approach when talking to them? Please suggest steps to help them reach a state of maximum relaxation."

[0606] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0607] Step 1:

[0608] The server receives audio data acquired from the user's device. This audio data is input into Google Cloud's Speech-to-Text API, which converts the audio signal into text information. This converted text information is output as initial data for diagnosis.

[0609] Step 2:

[0610] The server receives image and video data collected from the user's device. This visual data is input into AWS Rekognition, where image recognition technology is used to analyze people's facial expressions and movements. This analysis generates numerical data of emotional states, which is then used as input for the next processing stage.

[0611] Step 3:

[0612] The server integrates text data and emotional state data and inputs it into a TensorFlow-based machine learning model. This model matches each data point and analyzes the user's state in detail. The output is predicted support options and specific care approaches.

[0613] Step 4:

[0614] The server sends the generated support options and care approaches to the user's terminal. This displays specific actions and points of caution recommended for the user on the caregiver's terminal screen.

[0615] Step 5:

[0616] Users perform appropriate care actions based on the information presented on their device. This improves the quality of care and enables individualized support tailored to the user's emotional state.

[0617] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0618] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0620] [Fourth Embodiment]

[0621] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0622] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0623] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0625] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0627] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0628] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0629] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0630] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0631] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0632] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0633] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] This invention provides a system that comprehensively collects and analyzes data in various formats, including audio, text, images, and video data, to support diagnosis and treatment planning in medical settings. The processing of this system's program is described below in natural language.

[0635] First, the terminal receives patient data entered by healthcare professionals. This data includes recorded voice interviews from consultations, textual information from patient charts, X-ray and MRI images, and video data from endoscopic examinations. This data is encrypted and transmitted to the server via secure communication.

[0636] Next, the server preprocesses the received data according to a defined format. For audio data, noise is removed and the data is converted to text using speech recognition technology. For text data, important sections are extracted using text mining technology. Image data is analyzed based on computer vision technology to detect any anomalies. For video data, frame by frame analysis is performed.

[0637] Once preprocessing is complete, the server inputs this data into an artificial intelligence model and begins analysis. Based on the trained medical dataset, the AI ​​evaluates the interpretation of symptoms and the likelihood of disease, generating diagnostic information. Furthermore, treatment options are suggested, and appropriate treatment plans are indicated.

[0638] The generated diagnostic information and treatment suggestions are sent to the user's terminal and made accessible to healthcare professionals. The user, together with a specialist, evaluates the provided information and sends feedback to the server. This feedback is used to determine the next steps in the patient's treatment.

[0639] As a concrete example, consider a case where a patient visits a clinic complaining of abdominal pain. The user records an audio interview and simultaneously acquires image data from a CT scan, inputting it into the terminal. The server analyzes this data, understanding the patient's symptoms from the audio and detecting abnormalities from the CT images. Based on this information, the AI ​​diagnoses a high probability of acute appendicitis and generates a treatment plan recommending immediate surgical intervention. The user can then proceed with treatment quickly based on this information.

[0640] In this way, this system efficiently handles diverse data and supports rapid and accurate diagnosis and treatment planning in medical settings.

[0641] The following describes the processing flow.

[0642] Step 1:

[0643] The terminal collects patient voice, text, image, and video data entered by healthcare professionals. The data is encrypted for privacy reasons and transmitted to the server via a secure connection.

[0644] Step 2:

[0645] The server receives various types of data sent from the terminal. It performs preprocessing according to the data format, removes background noise from audio data, and converts it into text format using speech recognition.

[0646] Step 3:

[0647] The server performs text mining on the text data. Using natural language processing techniques, it extracts important keywords and organizes information related to symptoms and medical history.

[0648] Step 4:

[0649] The server analyzes image data and uses computer vision algorithms to detect abnormal areas and characteristic patterns. It utilizes specialized deep learning models to identify abnormalities such as tumors and inflammation.

[0650] Step 5:

[0651] The server divides the video data frame by frame and performs similar image analysis on each frame. It highlights any dynamic changes and generates information that is useful for detecting anomalies and monitoring treatment.

[0652] Step 6:

[0653] The server integrates the pre-processed data and performs analysis using an artificial intelligence model. It compares the results with similar past cases, lists potential diagnoses, and presents treatment options.

[0654] Step 7:

[0655] The server sends the generated diagnostic information and treatment suggestions to the user's terminal. Specifically, it provides a report containing an overview of the diagnostic results and details of recommended treatment methods.

[0656] Step 8:

[0657] Users view information from the server on their devices and collaborate with specialists to review the diagnosis. They can send additional feedback to the server as needed for further data analysis and optimization of treatment plans.

[0658] Step 9:

[0659] The user will perform specific medical procedures based on the established treatment plan, including explaining the treatment plan to the patient and arranging the procedures.

[0660] (Example 1)

[0661] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0662] In recent years, with the increasing volume of diverse data in medical settings, there has been a growing need for more efficient information integration and analysis. However, current systems struggle to effectively process data in different formats, such as audio, text, images, and video, and to quickly determine diagnoses and treatment plans. This challenge needs to be addressed.

[0663] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0664] In this invention, the server includes means for receiving diverse information from experts in the form of audio, text data, still images, and video; means for processing the received information according to each format; and means for interpreting the processed information using an automated knowledge model to generate diagnostic results and treatment plans. This enables rapid and efficient analysis of diverse data in different formats, allowing for accurate diagnosis and determination of optimal treatment plans in medical settings.

[0665] "Sound" refers to information emitted by humans that travels through the air as sound waves, and is used as data through recording and playback.

[0666] "Text data" refers to information written in language that is represented in digital format, and is a collection of characters and symbols.

[0667] A "still image" is a representation of a single moment of visual information on a flat surface, and is treated as digital data.

[0668] A "moving image format" is a data format that visually represents movement by playing a series of images over time.

[0669] The term "expert" refers to a professional who possesses specialized knowledge and skills in a particular field.

[0670] "Information" refers to meaningful data and knowledge, especially content that provides value to others.

[0671] "Means of receiving" refers to methods or devices used to acquire signals or data from external sources.

[0672] "Means of data processing" refers to methods or techniques used in the process of analyzing received information and converting it into a format suitable for a specific purpose.

[0673] An "automated knowledge model" refers to a virtual model built using artificial intelligence or machine learning techniques, which is a system that automatically makes decisions and inferences based on the input information.

[0674] "Interpreting" refers to the act of understanding the meaning based on given information and deriving relevant conclusions or implications.

[0675] "Diagnosis" refers to a medical evaluation or judgment provided based on analyzed data and expert knowledge.

[0676] A "treatment plan" is a proposed medical response or treatment plan for a specific symptom or disease.

[0677] This invention is a system that integrates and processes various data formats, providing immediacy and accuracy in medical settings. Embodiments are shown below.

[0678] The terminal receives patient information in the form of audio, text data, still images, and video provided by healthcare professionals. Examples include recordings of consultations, medical record information, X-ray and MRI images, and endoscopic examination videos. The received information is encrypted to maintain data confidentiality and security. Common secure communication protocols (e.g., SSL / TLS) are used for encryption.

[0679] Next, the server receives this information and performs preprocessing according to each data format. Audio data is converted to text using a speech recognition engine (e.g., widely known speech recognition technologies) after applying noise reduction techniques. Text data is then processed using natural language processing libraries (e.g., NLTK, spaCy) to extract important information. Image data is analyzed and anomaly detection is performed using computer vision technologies (e.g., OpenCV or deep learning models). Video data is analyzed frame by frame, allowing for the extraction of important scenes.

[0680] The processed information is comprehensively analyzed by an automated knowledge model (generative AI model) on the server. Based on a wide range of medical datasets, this model can comprehensively evaluate a patient's symptoms and generate diagnostic results and treatment plans. This process can improve the quality of medical care in healthcare settings where rapid response is crucial.

[0681] As a concrete example, suppose a patient comes to the hospital complaining of abdominal pain. The user inputs audio data from the medical interview and CT scan images into the terminal. The server analyzes this data comprehensively, derives a diagnosis suggesting the possibility of acute appendicitis, and presents a treatment plan recommending immediate surgical intervention.

[0682] An example of a prompt message might be, "Generate a diagnosis and treatment plan based on the audio interview data and CT scan image data of a patient complaining of abdominal pain." This system strongly supports rapid and highly accurate diagnosis and treatment planning in medical settings.

[0683] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0684] Step 1:

[0685] The terminal receives information in the form of audio, text data, still images, and video provided by healthcare professionals. This input includes audio recordings of patient examinations, transcribed medical record information, examination images (X-rays and MRIs), and videos of endoscopic examinations. Specifically, the terminal encrypts this data to prepare it for secure handling.

[0686] Step 2:

[0687] The terminal transmits received data to the server using a secure communication protocol. The input here is encrypted patient data, and the output is the secure transmission of data to the server. Specifically, the transfer is performed using a communication protocol (e.g., SSL / TLS) to ensure data integrity and privacy.

[0688] Step 3:

[0689] The server preprocesses received patient data according to its format. Audio data is subjected to a noise reduction algorithm and converted to text using speech recognition technology. Text data is analyzed through text mining techniques to extract important keywords. Image data is analyzed using computer vision technology (e.g., OpenCV) to identify signs of abnormality. Video is processed frame by frame, and critical scenes are extracted. Input is raw data in various formats, and output is data in an analyzable format.

[0690] Step 4:

[0691] The server inputs pre-processed data into an automated knowledge model and performs complex analysis. All input data is pre-processed, and based on this, the generating AI model compares it with the patient's symptoms to generate diagnostic results and treatment suggestions. Specifically, the AI ​​uses pattern recognition based on past medical data to improve the accuracy of the diagnosis. The output is a highly accurate diagnostic result and suggestions based on the analysis.

[0692] Step 5:

[0693] The diagnostic results and treatment plan generated by the server are sent back to the user terminal in a secure format. The input is the diagnostic results and treatment plan, and the output is the receipt to the user's device. The terminal prepares to display this information, allowing healthcare professionals and specialists to properly evaluate it.

[0694] Step 6:

[0695] The user consults with a specialist and evaluates the diagnostic results and treatment plan displayed on the terminal. Specifically, this evaluation is sent to the server as feedback data, which is then used for subsequent medical responses and decisions. The input in this step is the displayed diagnostic information, and the output is the feedback data sent to the server.

[0696] (Application Example 1)

[0697] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0698] In recent years, with the advancement of an aging society, the increasing burden on caregivers has become a social problem. In particular, there is a need to constantly monitor the health status of residents and respond quickly and appropriately, but in reality, it is difficult to achieve this with limited resources. Therefore, there is a need for a system that can more efficiently monitor health status and immediately detect and respond to abnormalities.

[0699] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0700] In this invention, the server includes means for receiving diverse information from a user in the form of voice, text, images, and video; means for processing the received information according to each format; means for analyzing the pre-processed information using an artificial intelligence algorithm to generate information for diagnosis and treatment options; means for monitoring health status from voice, video, and image data collected during daily activities and detecting abnormalities; and means for sending notifications to facilitate a rapid response when an abnormality is detected. This enables efficient monitoring of the health status of residents in care settings and allows for a rapid response in the event of an abnormality.

[0701] "Voice" refers to all spoken language and acoustic information input by the user, and data processing and analysis are performed based on this.

[0702] "Characters" refers to information in document or text format, and the analysis targets are written and electronic data entered by users.

[0703] "Images" refer to photographs or scanned visual information provided by users, and are used for information processing.

[0704] "Video" refers to dynamic visual information, and analysis and monitoring are performed based on the video provided by the user.

[0705] "Diverse information" refers to all different data formats, such as audio, text, images, and video, and is necessary to enable comprehensive information processing.

[0706] "Means of receiving" refers to mechanisms and devices that acquire user information in various forms and use it for subsequent processing.

[0707] "Means of data processing" refers to the technologies and methods used to preprocess received information and convert it into a format that is easy to analyze.

[0708] An "artificial intelligence algorithm" refers to a computational method used to analyze received information and generate optimal diagnostic information and treatment options.

[0709] "Diagnostic information" refers to information that includes the possibility of disease and interpretation of symptoms generated as a result of analysis.

[0710] "Treatment options" refer to recommended countermeasures for a specific health problem, and their implementation is desirable.

[0711] "Means of monitoring health status" refers to technologies and mechanisms that observe the user's daily activities and contribute to the early detection of abnormalities.

[0712] "Means for detecting anomalies" refers to a system that identifies unusual conditions or events, enabling a rapid response.

[0713] "Means of sending notifications to facilitate a rapid response" refers to communication technologies and methods for quickly transmitting information to relevant parties when an anomaly is detected.

[0714] To implement this invention, a system is used that receives diverse information from the user in the form of audio, text, images, and video. The server processes the received information according to its respective format. In this invention, received audio data is denoised and converted into text data using speech recognition technology. Text information is analyzed using text mining technology to extract important parts. Image data is analyzed using computer vision technology to detect anomalies. Video data is analyzed frame by frame.

[0715] The program uses Python for processing, a third-party API for speech recognition, and machine learning libraries such as TensorFlow for image analysis. The user terminal receives the analysis results and displays the outputted diagnostic information and treatment options for evaluation by medical or care professionals. Based on the information provided, the user can send feedback to the server and update the treatment plan.

[0716] As a concrete example, in nursing homes, robots monitor the environment within a room and collect audio and video data to monitor residents' daily activities. This allows for rapid notification if any abnormalities in health conditions are detected. By using a prompt such as, "Analyze the audio data and propose an appropriate care plan if the resident shows discomfort," the AI ​​can suggest the optimal care plan.

[0717] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0718] Step 1:

[0719] The terminal receives audio, text, image, and video information provided by the user. This data is classified and encoded according to its intended use and sent to the server. Input is user data, and output is classified and encoded data sent to the server.

[0720] Step 2:

[0721] The server preprocesses the received data according to its format. Audio data is denoised and converted into text data using a speech recognition API. Text information is then extracted using text mining techniques. Image data is analyzed using computer vision techniques to detect anomalies. Video data is analyzed by dividing it into frames. The input is encoded data, and the output is preprocessed, analyzable data.

[0722] Step 3:

[0723] The server uses the preprocessed data as input to a generating AI model and performs analysis. Based on the trained dataset, the server generates diagnostic information and treatment options. The input is preprocessed data, and the output is diagnostic information and treatment options.

[0724] Step 4:

[0725] The server sends the generated diagnostic information and treatment options to the user terminal. The terminal visualizes the received information, and the user (a medical or care professional) evaluates it. The input is the diagnostic information and treatment options, and the output is the display of the information by the terminal.

[0726] Step 5:

[0727] Users use their devices to view diagnostic information and treatment options, input feedback, and send it to the server. This feedback is used to update treatment plans. The input is the user's feedback, and the output is the updated treatment plan.

[0728] Step 6:

[0729] The server analyzes audio, video, and image data collected through daily activities and monitors for anomalies. If an anomaly is detected, it sends a notification to the terminal to enable prompt action. The input is data from daily activities, and the output is anomaly notifications.

[0730] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0731] This invention provides a medical support system that incorporates an emotion engine that recognizes the user's emotions along with voice, text, image, and video data. The program processing of this system is described below in natural language.

[0732] First, the terminal receives voice data entered by healthcare professionals, text information from patient medical records, medical images, and video data. In addition, an emotion engine analyzes the user's emotional state from voice intonation and facial expression data. This data is encrypted and securely transmitted to the server.

[0733] Based on the received data, the server preprocesses each data format according to its type. Audio data is de-noised and converted to text. Text data is searched for important keywords, and image and video data are subjected to algorithmic detection for anomalies. Simultaneously, the emotion engine quantifies the user's emotional state and provides the results in a format easily understood by healthcare professionals.

[0734] Once the pre-processed data is available, the server performs integrated analysis using an artificial intelligence model to generate diagnostic information and treatment options. The AI ​​compares multiple data points and proposes the optimal treatment plan, while also taking the user's emotional state into consideration. This allows for the provision of information in a format that is more easily accepted by the user.

[0735] The generated diagnostic information, treatment options, and emotion-based feedback are sent to the user's device. The user reviews this information and consults with a specialist to determine the final treatment plan. This feedback process leads to improved communication that takes the patient's emotional state into account.

[0736] For example, if a patient experiencing high stress levels receives treatment, the user's device evaluates the patient's voice and facial expressions, and the emotion engine recognizes this as a high-stress state. In response, the server can recommend a less stressful communication approach and provide treatment plans that require particular attention. As a result, the patient may feel more at ease during treatment, potentially leading to improved treatment outcomes.

[0737] This system makes it possible to improve the quality of medical care through integrated analysis of diverse data and evaluation of users' emotional states.

[0738] The following describes the processing flow.

[0739] Step 1:

[0740] The terminal collects voice data, text data, image data, and video data entered by healthcare professionals. This includes recordings of patient interviews, entries in electronic medical records, images from CT and MRI scans, and endoscopic images. The terminal also uses sensors to capture the patient's voice tone and facial expressions to acquire data necessary for the emotion engine.

[0741] Step 2:

[0742] The device encrypts the collected data and sends it to the server. Secure communication methods such as SSL / TLS are used to ensure data privacy and security.

[0743] Step 3:

[0744] The server preprocesses each received data according to its format. Noise is removed from audio data, and it is converted into text data using speech recognition technology. Important information is extracted from the text data using natural language processing. Image data is analyzed using deep learning models to detect anomalies. Video data is analyzed frame by frame, and dynamic anomalies are also detected.

[0745] Step 4:

[0746] The server uses an emotion engine to analyze received audio and facial expression data and evaluate the user's emotional state. Emotional states are classified into basic emotional categories such as joy, sadness, anger, fear, surprise, and disgust.

[0747] Step 5:

[0748] The server integrates pre-processed medical data and emotional information, and performs analysis using an artificial intelligence model. Based on the patient's medical history and current data, the AI ​​generates diagnostic information and treatment options. In doing so, it also takes the user's emotional state into consideration and proposes an emotionally sensitive approach to treatment.

[0749] Step 6:

[0750] The server transmits generated diagnostic information, treatment options, and emotional information to the user's terminal. This information is presented in a format that healthcare professionals can use to communicate with patients.

[0751] Step 7:

[0752] The user reviews the received information and works with a specialist to determine the treatment plan. The user also takes emotionally charged information into consideration and adjusts treatment methods and communication approaches accordingly.

[0753] Step 8:

[0754] The user implements specific medical procedures based on the decided treatment plan. They explain the treatment plan to the patient and plan the treatment procedure accordingly.

[0755] (Example 2)

[0756] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0757] Conventional medical support systems have been unable to consider the user's emotional state when generating diagnostic information and treatment options, leading to challenges in patient communication and acceptance of treatment plans. Furthermore, there is a lack of means for healthcare professionals to safely and efficiently handle diverse data, highlighting the need for effective methods to improve the quality of medical care.

[0758] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0759] In this invention, the server includes means for encrypting and securely transmitting data, means for quantifying and analyzing emotions using an emotion engine, and means for comprehensively analyzing data using a generative AI model to generate diagnostic information and treatment options that take emotions into consideration. This enables the provision of information that takes the user's emotions into consideration and improves the quality of medical care.

[0760] A "terminal" is a device that receives data in the form of voice, text, images, and video, and is equipped with the function to analyze the user's emotions.

[0761] "Encryption" is the process of converting information from plaintext to ciphertext in order to transmit data securely.

[0762] A "server" is a system that processes received data, generates diagnostic information and treatment options, and transmits them to the user's terminal.

[0763] An "emotion engine" is a program that analyzes a user's emotional state from audio and video and quantifies it.

[0764] A "generative AI model" is an artificial intelligence algorithm that comprehensively analyzes received data to generate diagnostic information and treatment options.

[0765] A "prompt statement" is an instruction statement used to present generated information to the user.

[0766] "Treatment policy" refers to the plan or policy for diagnosis and treatment that healthcare professionals implement for a patient.

[0767] This medical support system aims to generate diagnostic information and treatment options by integrating and analyzing diverse data using terminals, servers, and generative AI models. A specific implementation is shown below.

[0768] The terminal has the ability to receive data in the form of voice, text, images, and video. Healthcare workers input voice data through a microphone and provide text information through the electronic medical record system. Medical devices and cameras are used to acquire patient images and video data. Furthermore, the terminal has an emotion engine that analyzes voice intonation and facial expressions captured in videos to evaluate the emotional state of patients and healthcare workers. The analyzed emotion data is quantified, encrypted for security, and sent to the server.

[0769] The server performs format-appropriate preprocessing on the received data. Specifically, it uses an acoustic processing library to remove noise from audio data and convert it to text. For text data, it extracts important keywords using natural language processing tools. For image and video data, it analyzes them using an anomaly detection algorithm. In addition, it evaluates the patient's emotional state using the quantified results of the emotion engine. Based on the data processed in this way, a generative AI model performs an integrated analysis. This model has the ability to comprehensively analyze diverse data and, further considering emotional states, generate diagnostic information and treatment options.

[0770] Ultimately, the generated diagnostic information, treatment options, and emotion-based feedback are sent to the terminal via prompt messages. The user reviews this information and, in consultation with healthcare professionals, decides on a course of treatment. This system enables communication that takes the patient's emotions into account, and is expected to improve the quality of medical care.

[0771] As a concrete example, consider supporting the treatment of a patient in a high-stress state. The terminal evaluates the patient's voice and facial expressions, and the emotion engine recognizes this state as "high stress." Based on this information, the server recommends treatment strategies that require special attention and communication approaches aimed at stress reduction. For this purpose, prompts such as "We recommend treatment methods that take stress reduction into consideration" are used. This allows the patient to receive treatment with peace of mind, and improves the effectiveness of the treatment.

[0772] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0773] Step 1:

[0774] The terminal receives audio, text, image, and video data from healthcare professionals. Specifically, healthcare professionals input audio data using a microphone, retrieve text data from the electronic medical record system, and capture image and video data using a camera or medical device. The input data is treated as raw data to be prepared for subsequent analysis.

[0775] Step 2:

[0776] The device uses an emotion engine to analyze the intonation of received audio data and facial expressions in video data. Specifically, pitch, speed, and volume are detected from the audio data, and facial muscle movements are analyzed from the video data. This process involves data processing to quantify the user's emotional state. As output, an emotion score is generated.

[0777] Step 3:

[0778] The device encrypts all data before sending it to the server. This encryption process uses security protocols to protect the data. Inputs include processed audio, text, image, and video data, and output is encrypted data packets.

[0779] Step 4:

[0780] The server decrypts the received encrypted data and preprocesses it according to its format. For audio data, it uses an acoustic processing library to remove noise and converts it to text data. For text data, it extracts important keywords using natural language processing tools. For image and video data, it runs anomaly detection algorithms to extract features. The output consists of preprocessed text, a keyword list, and image feature data.

[0781] Step 5:

[0782] The server uses a generative AI model to comprehensively analyze preprocessed data and generate diagnostic information and treatment options. Sentiment scores are also taken into consideration during this analysis. Specifically, the AI ​​algorithm makes the optimal diagnosis based on information from multiple data sources. The output includes specific treatment suggestions and prompt messages.

[0783] Step 6:

[0784] The server sends the generated diagnostic information, treatment options, and prompt messages to the terminal. The prompt messages include emotion-based feedback. The output consists of treatment guidance and feedback information presented to the user. The user uses this information to consult with a healthcare professional and decide on the optimal treatment plan.

[0785] (Application Example 2)

[0786] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0787] In modern care settings, accurately understanding the emotional state of users and providing appropriate care accordingly is a challenging task. Responding to changes in users' emotions is essential for providing higher quality care services and improving user satisfaction.

[0788] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0789] In this invention, the server includes means for analyzing the user's emotional state in real time and providing emotion-based feedback, means for generating support options using a machine learning model, and means for securely transmitting each piece of information to the user's terminal. This enables care workers to quickly provide appropriate responses in accordance with the user's emotions.

[0790] "Sound" refers to the waveform of sound produced by human speech, and is a format used as a medium for transmitting information.

[0791] "Characters" are a set of symbols used to represent linguistic information, and are communicated visually as text.

[0792] An "image" is a two-dimensional representation of visual information, and includes forms such as photographs and illustrations.

[0793] A "video" is a media format that expresses movement by displaying a series of images in accordance with the passage of time.

[0794] "User" refers to the entity that operates or uses the system, and in this context, it may also refer to the person receiving care.

[0795] A "machine learning model" is the structure of an algorithm that automatically discovers patterns and rules based on data and makes predictions and decisions.

[0796] "Support options" refers to a series of countermeasures and action guidelines presented according to the user's situation and needs.

[0797] "User terminal" refers to a device that displays the final generated information and is accessible to the user, including hardware such as smartphones and tablets.

[0798] "Real-time" refers to a mode of operation in which processing and responses are performed simultaneously at the moment an event occurs.

[0799] "Secure transmission" refers to the act of ensuring reliable transmission of data by using encryption and authentication processes to prevent unauthorized access and information leakage during data transfer.

[0800] In implementing this invention, the server utilizes a machine learning model to analyze diverse user data and provide feedback based on emotional state. Specifically, it integrates the user's voice, text, image, and video data on a single platform. This data is collected in real time through user devices such as smart glasses and tablets.

[0801] The server uses Google Cloud's Speech-to-Text API to convert audio data into text and AWS Rekognition to analyze image and video data. Furthermore, a machine learning model using TensorFlow analyzes and quantifies the user's emotional state in real time.

[0802] The analyzed data is securely encrypted and transmitted to the user's device, presenting specific support options to enable caregivers to provide appropriate care based on the user's emotional state. This feedback is visualized on the user's device, supporting the caregivers' activities.

[0803] This technology can improve the quality of care by quickly detecting when a user shows signs of anxiety during a meal and prompting staff to take specific actions in response to those emotions, such as speaking to them in a calm voice.

[0804] Examples of prompt statements for the generated AI model are as follows:

[0805] "This user has been particularly prone to stress lately. Therefore, what would be the best approach when talking to them? Please suggest steps to help them reach a state of maximum relaxation."

[0806] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0807] Step 1:

[0808] The server receives audio data acquired from the user's device. This audio data is input into Google Cloud's Speech-to-Text API, which converts the audio signal into text information. This converted text information is output as initial data for diagnosis.

[0809] Step 2:

[0810] The server receives image and video data collected from the user's device. This visual data is input into AWS Rekognition, where image recognition technology is used to analyze people's facial expressions and movements. This analysis generates numerical data of emotional states, which is then used as input for the next processing stage.

[0811] Step 3:

[0812] The server integrates text data and emotional state data and inputs it into a TensorFlow-based machine learning model. This model matches each data point and analyzes the user's state in detail. The output is predicted support options and specific care approaches.

[0813] Step 4:

[0814] The server sends the generated support options and care approaches to the user's terminal. This displays specific actions and points of caution recommended for the user on the caregiver's terminal screen.

[0815] Step 5:

[0816] Users perform appropriate care actions based on the information presented on their device. This improves the quality of care and enables individualized support tailored to the user's emotional state.

[0817] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0818] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0819] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0820] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0821] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0822] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0823] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0824] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0825] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0826] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0827] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0828] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0829] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0830] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0831] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0832] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0833] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0834] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0835] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0836] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0837] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0838] The following is further disclosed regarding the embodiments described above.

[0839] (Claim 1)

[0840] A means of receiving diverse data from users in the form of audio, text, images, and video,

[0841] A means for preprocessing the received data according to its respective format,

[0842] A means for analyzing preprocessed data using an artificial intelligence model to generate diagnostic information and treatment options,

[0843] A means for transmitting generated diagnostic information and treatment options to the user terminal,

[0844] A means of receiving feedback from experts and updating treatment policies,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, wherein the artificial intelligence model has the function of selecting and presenting data that is highly relevant to different users.

[0848] (Claim 3)

[0849] The system according to claim 1, which includes a process for integrating the latest academic research results and data from past case databases to generate diagnostic information.

[0850] "Example 1"

[0851] (Claim 1)

[0852] Means for receiving diverse information from experts in audio, text data, still images, and video formats,

[0853] A means for processing the received information according to each format,

[0854] A means for interpreting processed information using an automated knowledge model to generate diagnostic results and treatment plans,

[0855] A means for notifying the user device of the generated diagnostic results and treatment plan,

[0856] Receiving evaluations from knowledgeable individuals provides a means to improve medical policies.

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, wherein the automated knowledge model has the function of selecting and displaying information that is highly relevant to each expert.

[0860] (Claim 3)

[0861] The system according to claim 1, which includes a process for generating diagnostic results that combines information from the latest academic research findings and past medical case data.

[0862] "Application Example 1"

[0863] (Claim 1)

[0864] A means of receiving diverse information from users in the form of audio, text, images, and video,

[0865] A means for processing the received information according to its respective format,

[0866] A means for analyzing pre-processed information using an artificial intelligence algorithm to generate information for diagnosis and treatment options,

[0867] A means for transmitting generated diagnostic information and treatment options to the user terminal,

[0868] A means of receiving expert opinions and updating treatment policies,

[0869] A means for monitoring health status and detecting abnormalities from audio, video, and image data collected during daily activities,

[0870] A means of sending notifications to facilitate a rapid response when an anomaly is detected,

[0871] A system that includes this.

[0872] (Claim 2)

[0873] The system according to claim 1, wherein the artificial intelligence algorithm has the function of selecting and presenting information that is highly relevant to different users.

[0874] (Claim 3)

[0875] The system according to claim 1, which includes a process for integrating the latest academic research results and information from a database of past cases in order to generate information for diagnosis.

[0876] "Example 2 of combining an emotion engine"

[0877] (Claim 1)

[0878] A terminal means equipped with means for receiving data in the form of audio, text, images, and video, and means for analyzing the user's emotions,

[0879] A means of encrypting received data and securely sending it to the server,

[0880] A preprocessing means for removing noise, extracting keywords, and detecting anomalies in the received data according to its format, and a means for quantifying emotions.

[0881] A means for analyzing pre-processed data using a generative AI model to generate diagnostic information and treatment options along with emotional evaluations,

[0882] A means of transmitting generated diagnostic information, treatment options, and emotion-based feedback to a terminal, thereby promoting the provision of information that takes emotions into consideration.

[0883] A means of receiving feedback from experts and updating treatment plans based on emotional evaluation,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, wherein the generative AI model has the function of selecting data that is highly relevant to different users and presenting it in a manner that takes into account their emotional state.

[0887] (Claim 3)

[0888] The system according to claim 1, which includes a process for generating diagnostic information that integrates the latest academic research results and data from past case databases and takes into account the user's emotional state.

[0889] "Application example 2 when combining with an emotional engine"

[0890] (Claim 1)

[0891] A means of receiving diverse information from users in the form of audio, text, images, and video,

[0892] A means for preprocessing the received information according to its respective format,

[0893] A means for analyzing pre-processed information using a machine learning model to generate diagnostic information and support options,

[0894] A means for transmitting generated diagnostic information and support options to the user's terminal,

[0895] A means of receiving feedback from experts and updating decision-making policies,

[0896] A means of analyzing the user's emotional state in real time and providing emotion-based feedback,

[0897] A system that includes this.

[0898] (Claim 2)

[0899] The system according to claim 1, wherein the machine learning model has the function of selecting and presenting information that is highly relevant to different users.

[0900] (Claim 3)

[0901] The system according to claim 1, which includes a process for integrating the latest academic research results and information from a database of past cases in order to generate diagnostic information. [Explanation of Symbols]

[0902] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving diverse information from users in the form of audio, text, images, and video, A means for processing the received information according to its respective format, A means for analyzing pre-processed information using an artificial intelligence algorithm to generate information for diagnosis and treatment options, A means for transmitting generated diagnostic information and treatment options to the user terminal, A means of receiving expert opinions and updating treatment policies, A means for monitoring health status and detecting abnormalities from audio, video, and image data collected during daily activities, A means of sending notifications to facilitate a rapid response when an anomaly is detected, A system that includes this.

2. The system according to claim 1, wherein the artificial intelligence algorithm has the function of selecting and presenting information that is highly relevant to different users.

3. The system according to claim 1, which includes a process for integrating the latest academic research results and information from a database of past cases in order to generate information for diagnosis.

Images