System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting medical data of a patient; means for inputting medical images of the patient; means for pre-processing the inputted medical data and medical images; means for inputting physician comments and instructions based on the pre-processed data and images; means for using a generative artificial intelligence model to analyze the inputted comments and instructions, medical data, and medical images and convert them into a form that is easy for the patient to understand; and means for displaying the converted data in the generated form.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Medical images and medical data, such as CT scans and MRI scans, are difficult for ordinary patients without specialized knowledge to understand. Doctors have limited consultation time, making it difficult for them to provide sufficient explanations to patients. As a result, patients often do not accurately understand their own health conditions, which often hinders appropriate treatment and self-management. To solve this problem, there is a need for a system that can convert medical data and images into an easy-to-understand format, making it easier for patients to understand their own health conditions. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a means for inputting patient medical data, a means for inputting medical images, a means for preprocessing the input data and images, a means for inputting physician comments and instructions, a means for using a generative artificial intelligence model to analyze the data and convert it into a format that is easy for patients to understand, and a means for displaying the converted data. The generated data is presented in a format that is easy for patients to understand, such as a format that highlights lesions, 3D images, or videos, allowing patients to accurately understand their health status and take appropriate treatment and self-management measures. The system also includes a history management means for reanalyzing and converting stored data, supporting effective communication with medical professionals.

[0006] "Patient medical data" refers to medical information such as a patient's diagnosis, measurements, test results, and treatment history.

[0007] "Medical images" refers to images taken with medical equipment such as CT scans, MRIs, X-rays, and ultrasounds.

[0008] "Preprocessing" refers to processing such as format matching, noise removal, resolution adjustment, and contrast enhancement of input medical data and medical images.

[0009] "Doctor's comments and instructions" refers to the explanations and instructions entered by the doctor based on the patient's diagnosis and treatment plan.

[0010] A "generative artificial intelligence model" refers to an artificial intelligence technology that analyzes input medical data, medical images, and doctor's comments and instructions, and converts them into a format that is easy for patients to understand.

[0011] "Display means" refers to a device or interface for visually presenting generated photographs, 3D images, videos, etc. to the patient.

[0012] A "database" refers to a system for centrally storing and managing information such as medical data, medical images, comments, and instructions.

[0013] "History management means" refers to a function for reanalyzing or converting data based on the saved data.

[0014] "Terminal" refers to electronic devices such as computers, tablets, and smartphones that doctors and patients use to input data and view display results. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] A specific embodiment of the present invention will be described below. The purpose of this system is to convert a patient's medical data and medical images into a format that is easy for the patient to understand, thereby facilitating communication between doctors and patients.

[0037] System Configuration

[0038] The system mainly includes the following elements:

[0039] 1. Data entry method

[0040] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[0041] 2. Pretreatment Methods

[0042] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[0043] 3. Comment and instruction input means

[0044] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[0045] 4. Generative AI Models

[0046] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for patients to understand.

[0047] 5. Display means

[0048] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[0049] 6. Database

[0050] The server has a database that centrally stores medical data, medical images, comments and instructions.

[0051] 7. History Management Methods

[0052] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[0053] Natural language description of the program

[0054] Data Entry and Preprocessing

[0055] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[0056] 2. The device sends the selected file to the server.

[0057] 3. The server receives the data and images and performs preprocessing, such as noise reduction, resolution adjustment, and contrast enhancement. It also standardizes the format and units of medical data.

[0058] Entering comments and instructions

[0059] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[0060] 2. The device sends comments and instructions to the server.

[0061] 3. The server stores the comments and instructions in a database.

[0062] Data analysis and transformation

[0063] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[0064] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[0065] 3. The server receives the generated data and stores it in a database.

[0066] Providing results

[0067] 1. The server sends the generated visualization data to the device.

[0068] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[0069] 3. The user (patient) checks the visualized data and understands their own health condition.

[0070] Specific examples

[0071] For a shoulder fracture diagnosis:

[0072] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[0073] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[0074] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[0075] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[0076] 5. The server sends the generated 3D model to the device.

[0077] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[0078] In this way, this system converts patients' medical data and medical images into easy-to-understand information, supporting communication between doctors and patients and promoting patient health management.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] The user (doctor) selects the patient's medical data and medical images on the terminal. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[0082] Step 2:

[0083] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[0084] Step 3:

[0085] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[0086] Step 4:

[0087] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[0088] Step 5:

[0089] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[0090] Step 6:

[0091] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[0092] Step 7:

[0093] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[0094] Step 8:

[0095] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[0096] Step 9:

[0097] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[0098] Step 10:

[0099] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[0100] Step 11:

[0101] The server stores the generated visualization data in a database, where the update history is also managed.

[0102] Step 12:

[0103] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[0104] Step 13:

[0105] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[0106] Step 14:

[0107] Users (patients) can view the visualized data and understand the details of their health condition and treatment. Through the interface, patients can ask questions to their doctor or request additional explanations.

[0108] The above is the specific flow of program processing according to the present invention.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] In the medical field, it is important to convert medical data and medical images into a format that is easy for patients to understand, but this requires a lot of manual work and specialized knowledge. Furthermore, poor communication between doctors and patients can negatively impact the quality of medical care. The present invention aims to solve these problems by automatically converting medical data and medical images into an easily understandable format, thereby improving communication between doctors and patients.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes: a means for inputting patient medical data, a means for inputting patient medical images, a means for preprocessing the input medical data and medical images, a means for inputting doctor comments and instructions based on the preprocessed data and images, a means for using a generative artificial intelligence model that integrates and analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for patients to understand (e.g., photos, 3D images, videos), a means for visualizing and interactively displaying the generated data, and a means for having a database that centrally stores patient medical data, medical images, comments, and instructions and for managing their history. This automatically converts medical data and medical images into easy-to-understand formats, facilitating communication between doctors and patients and improving the quality of medical care.

[0114] "Patient Medical Data" means quantitative or qualitative medical information about a patient, such as a patient's diagnostic results, measurements, medical history, and prescription information.

[0115] "Patient medical images" refers to image data that visually captures the internal state of a patient's body, such as CT scans, MRIs, X-rays, and ultrasound images.

[0116] "Preprocessing means" refers to a process for improving the quality of input medical data and medical images by performing processes such as noise removal, resolution adjustment, contrast enhancement, and format matching.

[0117] "Doctor's comments and instructions" refers to text information entered by a doctor regarding the patient's condition and treatment, and includes a diagnosis, treatment plan, and precautions.

[0118] A "generative artificial intelligence model" refers to an algorithm or program that uses machine learning and deep learning technologies to analyze input medical data and medical images and convert them into a format that is easy for patients to understand.

[0119] "Means for interactive display" refers to an interface that displays the generated visualization data in a form that allows the user to manipulate it and enables detailed confirmation and observation.

[0120] "Database" means an information system for centrally storing, efficiently managing, and making accessible medical data, medical images, comments, and instructions.

[0121] "Means for managing history" refers to the process of recording and managing the history of past processing and operations in order to track, analyze, and reuse stored medical data and medical images, as well as comments and instructions.

[0122] This invention is a system that automatically converts a patient's medical data and medical images into a format that is easy for the patient to understand. This aims to improve communication between doctors and patients and enhance the quality of medical care. The system of this invention is mainly composed of three main elements: a server, a terminal, and a user.

[0123] System Configuration

[0124] 1. Data entry method

[0125] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading. The device sends the selected files to the server. For example, the device transfers the files using the HTTP or HTTPS protocol.

[0126] 2. Pretreatment Methods

[0127] The server performs preprocessing such as noise removal, resolution adjustment, and contrast enhancement on the medical data and medical images it receives. Specifically, it uses the OpenCV library to perform image filtering, and the Pandas library to align the format of the medical data and standardize units.

[0128] 3. Comment and instruction input means

[0129] The user (doctor) enters comments about the patient's condition and treatment instructions on the device. The device sends the comments and instructions to the server. For example, the data is sent using JSON format, and the server converts it into a format that can be properly analyzed.

[0130] 4. Generative AI Models

[0131] The server integrates the preprocessed data and images with the doctor's comments and converts them into a format that is easy for the patient to understand (e.g., photos, 3D images, videos) using a generative AI model. Specifically, the generative AI model is invoked using the Scikit-learn and TENSORFLOW® libraries.

[0132] 5. Display means

[0133] The server sends the generated visualization data to the terminal, which provides an interactive display where the user (patient) can view the transformed data. For example, D3.js or Three.js can be used to interactively display the data.

[0134] 6. Database

[0135] The server has a database for centrally storing patient medical data, medical images, comments, and instructions, and manages the history. Data is stored using MongoDB or MySQL (registered trademark), and past data can be searched using SQL queries.

[0136] Specific examples

[0137] For a shoulder fracture diagnosis:

[0138] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[0139] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[0140] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of this shoulder."

[0141] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[0142] 5. The server sends the generated 3D model to the device.

[0143] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[0144] Prompt Sentence Examples

[0145] The following is an example of a specific prompt:

[0146] 1. "Enter a CT scan image and generate a 3D model highlighting the fracture."

[0147] 2. "Analyze the cardiac MRI images and mark any abnormalities."

[0148] In this way, the system of the present invention converts patient medical data and medical images into easy-to-understand information, thereby supporting communication between doctors and patients and promoting health management for patients.

[0149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0150] Step 1: Data entry

[0151] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[0152] Input: Medical data files (e.g., CSV, Excel), medical image files (e.g., CT scans, MRI images)

[0153] Specific operations: The user operates the interface on the device, selects the required file, and clicks the "Upload" button.

[0154] Output: The selected files are sent to the server.

[0155] 2. The device sends the selected file to the server.

[0156] Input: A file selected by the user

[0157] Data processing: Transfer files securely using HTTP or HTTPS protocols.

[0158] Specific operation: The device temporarily saves the file and sends it to the specified API endpoint on the server.

[0159] Output: The server receives the file.

[0160] Step 2: Preprocessing

[0161] 1. The server receives the data and images.

[0162] Input: File sent from terminal

[0163] Specific operation: The server saves the file in the specified directory and records the file's metadata (source, date and time of sending, etc.).

[0164] Output: Saved medical data and medical image files

[0165] 2. The server performs noise reduction, resolution adjustment, and contrast enhancement on the medical images.

[0166] Input: Saved medical image files

[0167] Data processing: Image noise removal, resolution adjustment, and contrast enhancement are performed using the OpenCV library.

[0168] Specific operation: Executes OpenCV functions (e.g., cv2.fastNlMeansDenoising(), cv2.resize(), cv2.equalizeHist()) sequentially.

[0169] Output: Preprocessed medical image files

[0170] 3. The server harmonizes the format and standardizes the units of medical data.

[0171] Input: Stored medical data file

[0172] Data processing: Use the Pandas library to clean the data, align the format, and standardize units.

[0173] Specific operation: Executes Pandas functions (e.g., pd.read_csv(), df.fillna(), df.apply()) sequentially.

[0174] Output: Preprocessed medical data file

[0175] Step 3: Enter comments and instructions

[0176] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[0177] Input: Comments about the patient's condition and treatment instructions

[0178] Specific operation: The user enters a comment or instruction into the interface on the device and clicks the "Send" button.

[0179] Output: Comments and instructions entered

[0180] 2. The device sends comments and instructions to the server.

[0181] Input: Comments or instructions entered

[0182] Data processing: Convert to JSON format and send to the server.

[0183] Specific operation: The terminal converts the input data into JSON format and calls an API to send it to the server.

[0184] Output: The server receives comments and instructions.

[0185] Step 4: Analyze and transform the data

[0186] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[0187] Input: Preprocessed medical data files, medical image files, comments

[0188] Data processing: Integrating data and images into a format that can be input into a generative AI model.

[0189] Specific operation: Convert the data into the input format for the generative AI model and call the AI ​​model.

[0190] Output: Input data for the AI ​​model

[0191] 2. A generative AI model analyzes the data and translates it into a format that is easy for patients to understand.

[0192] Input: Input data for the AI ​​model

[0193] Data Computation: Analysis and transformation using machine learning algorithms and deep learning models.

[0194] Specific operation: Calls the model's prediction function and generates a 3D model and video that highlights the lesion area.

[0195] Output: Generated visualization data (e.g. 3D models, enhanced images)

[0196] 3. The server receives the generated data and stores it in a database.

[0197] Input: Generated visualization data

[0198] Data processing: Save the data in the appropriate format for the database.

[0199] Specific operation: Insert the generated data into a MongoDB or MySQL database.

[0200] Output: Visualization data stored in a database

[0201] Step 5: View and verify

[0202] 1. The server sends the generated visualization data to the device.

[0203] Input: Visualization data stored in a database

[0204] Data processing: Converting data into the format required for sending to the terminal.

[0205] Specific operation: Converts data into JSON or binary format and sends it to the terminal.

[0206] Output: Visualization data sent to the terminal

[0207] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[0208] Input: Visualization data received from the server

[0209] Specific behavior: Display data interactively using D3.js and Three.js.

[0210] Output: User-navigable visualization of data

[0211] 3. The user (patient) checks the visualized data and understands their own health condition.

[0212] Input: Interactive visualization data

[0213] Specific operation: The user drags the data, zooms, etc.

[0214] Output: The user visually understands their health status.

[0215] Step 6: History Management

[0216] 1. The server manages history based on centrally stored data.

[0217] Input: Stored medical data, medical images, comments and instructions

[0218] Data Processing: Organize historical data and make it quickly accessible when needed.

[0219] What you will do: Search and analyze historical data using SQL queries and log search tools.

[0220] Output: Archived data

[0221] 2. The server stores the operation history to facilitate re-analysis and conversion.

[0222] Input: Log data for each operation

[0223] Data processing: Analyze and save log data.

[0224] Specific behavior: Recording operations in log files and dedicated history databases.

[0225] Output: Saved operation history

[0226] (Application example 1)

[0227] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0228] Conventional medical data and medical image processing systems have the problem that it is difficult for patients to understand their own medical condition. Furthermore, the authentication data in facial recognition systems is difficult to understand visually, making it difficult to intuitively confirm authentication results. Therefore, there is a need to facilitate communication between doctors and patients, and between security managers and authentication targets, and to streamline authentication work.

[0229] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0230] In this invention, the server includes means for inputting patient medical data, means for inputting patient medical images, means for preprocessing the input medical data and medical images, means for inputting doctor comments and instructions based on the preprocessed data and images, means for using a generative artificial intelligence model that analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for the patient to understand, means for displaying the data converted into the generated format, means for inputting facial authentication data, means for preprocessing the input facial authentication data, means for using a generative artificial intelligence model that analyzes and visualizes the preprocessed facial authentication data, and means for interactively displaying the generated facial authentication data. This allows patients to intuitively understand their own medical data and medical images, and also enables the person to be authenticated to visually confirm their own authentication data.

[0231] "Medical data" refers to information such as a patient's health status, diagnosis results, and treatment information.

[0232] "Medical images" refers to image data obtained by medical equipment such as CT scans and MRIs.

[0233] "Preprocessing means" refers to techniques for performing processes such as noise removal and resolution adjustment on medical data, medical images, and facial recognition data.

[0234] "Means for inputting comments and instructions" refers to an interface that allows doctors and administrators to input specific diagnosis results and instructions to the system.

[0235] A "generative artificial intelligence model" refers to artificial intelligence that analyzes input data and converts it into a format that is easy for patients and those being certified to understand.

[0236] "Means for displaying data converted into a generated format" refers to an interface for visually displaying the parsed and converted data.

[0237] "Facial recognition data" refers to image data and feature data relating to an individual's face.

[0238] "Interactive display means" refers to a display method that is easy for users to operate and allows them to intuitively check the data.

[0239] A specific embodiment of the present invention will be described below. The purpose of this system is to convert patient medical data and medical images into a format that is easy for patients to understand, and to provide visually intuitive authentication results in a face recognition system.

[0240] System Configuration

[0241] The system mainly includes the following elements:

[0242] 1. Data entry method

[0243] The terminal provides an interface for inputting the patient's medical data, medical images, and facial recognition data.

[0244] 2. Pretreatment Methods

[0245] The server receives the input medical data, medical images, and facial recognition data, and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[0246] 3. Comment and instruction input means

[0247] The terminal provides an interface that accepts comments and instructions entered by a doctor or security administrator.

[0248] 4. Generative AI Models

[0249] The server combines the pre-processed data, medical images, facial recognition data, and comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for the patient or person being authenticated to understand.

[0250] 5. Display means

[0251] The terminal provides the generated visualization data and provides an interface that allows for interactive display.

[0252] 6. Database

[0253] The server has a database that centrally stores medical data, medical images, facial recognition data, comments and instructions.

[0254] 7. History Management Methods

[0255] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[0256] Hardware and software used

[0257] Smartphones (e.g., iPhone (registered trademark), ANDROID (registered trademark) devices)

[0258] Head-mounted display (HMD) (e.g., Oculus Quest 2, Microsoft® HoloLens® 2)

[0259] Server (e.g., AWS EC2 instance, Google Cloud Platform)

[0260] Databases (e.g. Amazon RDS, Google Cloud Firestore)

[0261] Generative AI models (e.g., models using TensorFlow or PyTorch)

[0262] Processing flow

[0263] Data Entry and Preprocessing

[0264] 1. The user (doctor or person to be authenticated) enters medical data, medical images, and facial recognition data using a smartphone or HMD and uploads them to the server.

[0265] 2. The server receives the data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[0266] Entering comments and instructions

[0267] 1. The user (doctor or administrator) enters analysis comments and instructions on the device and sends them to the server.

[0268] 2. The server stores the comments and instructions in a database.

[0269] Data analysis and transformation

[0270] 1. The server integrates the preprocessed data and comments and invokes the generative AI model.

[0271] 2. A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[0272] 3. The server receives the generated data and stores it in a database.

[0273] Providing results

[0274] 1. The server sends the generated visualization data to the device.

[0275] 2. The terminal provides an interactive display, allowing the user (patient or authentication subject) to review the content.

[0276] Specific examples

[0277] For a shoulder fracture diagnosis:

[0278] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system.

[0279] 2. The server preprocesses the CT scan images (adjusting resolution and removing noise).

[0280] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured shoulder."

[0281] 4. The generative AI model analyzes the data and generates a 3D model that highlights the fracture.

[0282] 5. The server sends the generated 3D model to the device.

[0283] 6. The user (patient) checks the 3D model on a smartphone or HMD to visually understand the condition of the shoulder fracture.

[0284] Prompt Sentence Examples

[0285] Target image: CT scan image of shoulder

[0286] Comment: Please emphasize the broken shoulder.

[0287] The following applies to the specific example of face recognition:

[0288] 1. The user (person to be authenticated) takes a photo of their face with their smartphone and uploads it to the authentication system.

[0289] 2. The server preprocesses the image and performs feature extraction.

[0290] 3. The administrator enters a comment saying, "Emphasis on the position of the eyes and the width of the mouth."

[0291] 4. The generative AI model analyzes the data and generates a highlighted 3D face model containing the authentication results.

[0292] 5. The server sends the generated 3D face model and authentication results to the device.

[0293] 6. The user checks the 3D model on a smartphone or HMD and intuitively understands the authentication results and highlighting.

[0294] Prompt Sentence Examples

[0295] Target image: Facial recognition image

[0296] Comment: Emphasize the position of the eyes and the width of the mouth

[0297] In this way, the embodiments for carrying out the present invention have been specifically described.

[0298] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0299] Step 1:

[0300] Users (doctors or individuals to be authenticated) enter medical data, medical images, and facial recognition data using a smartphone or HMD and upload them to the server.

[0301] Input: Patient medical data, medical images, or facial recognition data

[0302] Output: Raw data sent to the server

[0303] Specific action: The user captures data using the device's camera or scanning function and uploads it.

[0304] Step 2:

[0305] The server receives the uploaded data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[0306] Input: Uploaded data (medical data, medical images, facial recognition data)

[0307] Output: Preprocessed data

[0308] Specific operation: The server aligns the format of the medical data, removes noise, and adjusts the image resolution to generate preprocessed data.

[0309] Step 3:

[0310] The user (doctor or administrator) enters analysis comments and instructions on the terminal and sends them to the server.

[0311] Input: Comments and instructions

[0312] Output: Comments and instructions sent to the server

[0313] Specific operation: The user uses the input interface of the terminal to input diagnostic results and analysis instructions, and sends them to the server.

[0314] Step 4:

[0315] The server integrates the preprocessed data and comments and invokes the generative AI model.

[0316] Input: Preprocessed data and comments

[0317] Output: Input data to a generative AI model

[0318] Specific operation: The server integrates the preprocessed data with input instructions and comments to pass them to the generative AI model.

[0319] Step 5:

[0320] A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[0321] Input: Integrated data and comments

[0322] Output: Visualization data (highlighted photos, 3D images, videos)

[0323] What it does: The generative AI model analyzes the data and generates visualizations based on the prompt.

[0324] Step 6:

[0325] The server receives the generated data and stores it in a database.

[0326] Input: Generated visualization data

[0327] Output: Visualization data stored in a database

[0328] Specific operation: The server stores the generated data in a database and provides a history management function.

[0329] Step 7:

[0330] The server transmits the generated visualization data to the terminal.

[0331] Input: Visualization data stored in a database

[0332] Output: Visualization data sent to the terminal

[0333] Specific operation: The server retrieves visualization data from the database and sends it to the terminal.

[0334] Step 8:

[0335] The terminal provides an interactive display, and the user (patient or person being authenticated) can review the content.

[0336] Input: Visualization data sent to the device

[0337] Output: Visualization data reviewed by the user

[0338] Specific operation: The terminal displays the visualized data interactively, and the user can perform operations to check the data.

[0339] Through each of these steps, the system of the present invention can effectively process medical data and facial recognition data and present it to the user in an intuitive and easy-to-understand format.

[0340] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0341] A specific embodiment of the present invention is described below. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[0342] System Configuration

[0343] The system mainly includes the following elements:

[0344] 1. Data entry method

[0345] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[0346] 2. Pretreatment Methods

[0347] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[0348] 3. Comment and instruction input means

[0349] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[0350] 4. Generative AI Models

[0351] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[0352] 5. Display means

[0353] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[0354] 6. Database

[0355] The server has a database that centrally stores medical data, medical images, comments and instructions.

[0356] 7. History Management Methods

[0357] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[0358] 8. Emotion Engine

[0359] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[0360] Natural language description of the program

[0361] Data Entry and Preprocessing

[0362] 1. The user (doctor) selects the patient's medical data and medical images on the device and begins uploading.

[0363] 2. The device sends the selected file to the server.

[0364] 3. The server receives the data and images and performs preprocessing, including image noise reduction, resolution adjustment, and contrast enhancement. For medical data, processing such as unit unification and format conversion is also performed.

[0365] Entering comments and instructions

[0366] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[0367] 2. The device sends comments and instructions to the server.

[0368] 3. The server stores the comments and instructions in a database.

[0369] Data analysis and transformation

[0370] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[0371] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[0372] 3. The server receives the generated data and stores it in a database.

[0373] Emotion analysis and display adjustment

[0374] 1. The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time.

[0375] 2. The emotion engine recognizes the user's emotional state based on the analyzed data. For example, if the user is feeling anxious, it will display an option to immediately contact a call center staff member.

[0376] 3. The server dynamically adjusts the displayed data based on the results of the emotion engine, for example by reducing technical terms or emphasizing visuals to make it easier for users to understand.

[0377] Providing results

[0378] 1. The server sends the generated visualization data to the device.

[0379] 2. The terminal provides an interactive display where the user (patient) can view the converted data. The user can rotate the 3D model and zoom in and out to view the details.

[0380] 3. The user (patient) reviews the visualized data and understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface.

[0381] Specific examples

[0382] For a shoulder fracture diagnosis:

[0383] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[0384] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[0385] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[0386] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[0387] 5. The server sends the generated 3D model to the device.

[0388] 6. The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state.

[0389] 7. The server adapts the display to the patient's emotional state and provides additional information and explanations to reduce anxiety.

[0390] 8. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[0391] In this way, this system converts patients' medical data and medical images into easy-to-understand formats, and also takes into consideration the user's emotional state, thereby supporting communication between doctors and patients and promoting patient health management.

[0392] The processing flow will be explained below.

[0393] Step 1:

[0394] The user (doctor) selects the patient's medical data and medical images on the terminal and starts uploading. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[0395] Step 2:

[0396] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[0397] Step 3:

[0398] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[0399] Step 4:

[0400] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[0401] Step 5:

[0402] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[0403] Step 6:

[0404] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[0405] Step 7:

[0406] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[0407] Step 8:

[0408] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[0409] Step 9:

[0410] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[0411] Step 10:

[0412] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[0413] Step 11:

[0414] The server stores the generated visualization data in a database, where the update history is also managed.

[0415] Step 12:

[0416] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[0417] Step 13:

[0418] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[0419] Step 14:

[0420] The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state. For example, data is collected in real time using a camera or microphone, and emotion analysis is performed.

[0421] Step 15:

[0422] The server receives the analysis results of the emotion engine and dynamically adjusts the display content according to the patient's emotional state. For example, if the patient is highly anxious, it adds detailed explanations or displays options for asking questions to the doctor.

[0423] Step 16:

[0424] The user (patient) can view the tailored visualization data to understand the details of their health condition and treatment. Through the interface, the patient can ask questions to the doctor or request additional explanations.

[0425] In this way, the present invention facilitates communication between doctors and patients and promotes patient health management by visualizing medical data and images in an easy-to-understand manner and dynamically adjusting them according to the patient's emotional state.

[0426] Example 2

[0427] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0428] It is extremely difficult for patients to understand their own medical data and medical images. Poor communication between doctors and patients can lead to inadequate understanding of treatment and diagnosis. It is also important to appropriately adjust the content of explanations based on the patient's emotional state, but this has been difficult with conventional systems.

[0429] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0430] In this invention, the server includes means for integrating preprocessed data, images, and comments, calling a generative artificial intelligence model, and performing analysis; means for storing the generated data in a database; and means for displaying the generated data and receiving patient responses and questions on a terminal with an interactive display. This allows patients to intuitively understand their own medical data and medical images, enabling effective communication between doctors and patients. Furthermore, the display content can be dynamically adjusted according to the patient's emotional state, deepening the patient's understanding and reducing anxiety.

[0431] "Patient medical data" refers to information such as the patient's health condition, medical history, diagnosis, and treatment progress.

[0432] "Medical images" are image data, such as CT scans, MRIs, and X-rays, that visually show the internal structure and condition of a patient's body.

[0433] "Input means" refers to an interface or device that allows a user to provide medical data or medical images to the system.

[0434] A "server" is a central computer system that receives and processes input data.

[0435] "Preprocessing" refers to processes such as noise removal, resolution adjustment, and format conversion to improve the quality of input data.

[0436] "Doctor's comments and instructions" are written input of the doctor's professional opinions and instructions regarding the patient's condition and treatment plan.

[0437] A "database" is a data storage system for centrally managing and storing information about the system, such as medical data, medical images, comments, and instructions.

[0438] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[0439] An "interactive display" is an interface that allows users to manipulate and view the generated data.

[0440] The "means for accepting patient responses and questions" is an interface that allows patients to ask questions or give feedback on the displayed data.

[0441] A specific embodiment of the present invention will now be described. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and furthermore, dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[0442] System Configuration

[0443] The system mainly includes the following elements:

[0444] 1. Data entry method

[0445] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[0446] 2. Pretreatment Methods

[0447] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[0448] 3. Comment and instruction input means

[0449] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[0450] 4. Generative AI Models

[0451] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[0452] 5. Display means

[0453] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[0454] 6. Database

[0455] The server has a database that centrally stores medical data, medical images, comments and instructions.

[0456] 7. History Management Methods

[0457] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[0458] 8. Emotion Engine

[0459] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[0460] Data Processing

[0461] 1. Data Entry

[0462] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading them. The device then sends this data to the server.

[0463] 2. Pretreatment

[0464] The server preprocesses the medical data and images it receives. For image data, it performs noise removal, resolution adjustment, and contrast enhancement, and for medical data, it standardizes units and converts formats.

[0465] 3. Enter comments and instructions

[0466] The user (doctor) enters comments and instructions about the patient's condition and treatment into the terminal, which then sends this information to the server, which then stores this information in a database.

[0467] 4. Data Analysis and Transformation

[0468] The server combines the preprocessed medical data, medical images, and comments and invokes the generative AI model, which analyzes the data and converts it into a format that is easy for patients to understand (e.g., a 3D image highlighting the lesion).

[0469] 5. Emotion analysis and display adjustment

[0470] The device analyzes the user's (patient's) facial expressions and voice in real time, and the emotion engine uses this information to recognize the user's emotional state. The server dynamically adjusts the data displayed based on the emotion engine's results. For example, if the patient is feeling anxious, the server displays simplified explanations using technical terms.

[0471] 6. Interactive Display

[0472] The server sends the generated visualization data to the terminal, which then provides it to the patient through an interactive interface, allowing the patient to rotate, zoom in and out to view the data.

[0473] Specific examples

[0474] For a shoulder fracture diagnosis:

[0475] Example prompt sentence:

[0476] "Please analyze the CT scan of the patient's shoulder and generate a 3D model that highlights the fracture. The generated model should be intuitive for the patient."

[0477] By inputting this prompt into the generative AI model, a 3D image that is easy for patients to understand is generated, highlighting the fractured shoulder area, allowing patients to intuitively understand their condition and facilitating communication with their doctors.

[0478] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0479] Step 1:

[0480] A user (doctor) uses a terminal to select a patient's medical data (e.g., diagnosis results, measurements) and medical images (e.g., CT scans, MRI images) and initiates uploading to the system. The input at this time is the medical data and medical images selected by the user, and the output is the state in which these data have been uploaded to the terminal.

[0481] Step 2:

[0482] The terminal transmits the medical data and medical images selected by the user to the server. The input is the medical data and medical images on the terminal, and the output is the data transmitted to the server. HTTPS is used as the transmission protocol.

[0483] Step 3:

[0484] The server receives medical data and medical images sent from the terminal. The input is the data and images sent to the server, and the output is the data that has been received inside the server. The server reads this data for preprocessing.

[0485] Step 4:

[0486] The server preprocesses the medical data and medical images it receives. First, it performs noise removal, resolution adjustment, and contrast enhancement on the image data. Next, it standardizes the units and converts the format of the medical data. The input at this time is the raw data received by the server, and the output is the preprocessed medical data and medical images.

[0487] Step 5:

[0488] The user (doctor) inputs comments and instructions about the patient's condition and treatment into the terminal and sends the data to the system. The input is the comments and instructions entered by the doctor, and the output is the comments and instructions sent from the terminal to the server.

[0489] Step 6:

[0490] The terminal sends the entered comments and instructions to the server. The input here is the comments and instructions entered on the terminal, and the output is the data sent to the server.

[0491] Step 7:

[0492] The server integrates the preprocessed medical data, medical images, and comments and stores them in a database, where the inputs are the preprocessed data, images, and doctor's comments and instructions, and the output is the integrated data stored in the database.

[0493] Step 8:

[0494] The server generates a prompt to invoke the generative AI model. For example, it generates a prompt such as, "Analyze the CT scan image of the patient's shoulder and generate a 3D model that highlights the fractured area." The input is the integrated data and comments, and the output is the generated prompt.

[0495] Step 9:

[0496] The server sends data and prompts to the generative AI model for analysis and conversion. The input is the data integrated with the generated prompt, and the output is the data and prompt sent to the generative AI model.

[0497] Step 10:

[0498] The generative AI model analyzes the data and converts it into a format that is easy for the patient to understand (e.g., a photograph highlighting the lesion or a 3D image). The input to the generative AI model is the data and prompts it receives, and the output is the converted visualization data.

[0499] Step 11:

[0500] The server receives the analysis results from the generative AI model and stores them in a database. The input at this time is the analysis results from the generative AI model, and the output is the visualization data stored in the database.

[0501] Step 12:

[0502] The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time. The input is the patient's facial expressions, voice, and operation patterns, and the output is analyzed emotional data.

[0503] Step 13:

[0504] The server dynamically adjusts the displayed data based on the results of the emotion engine. For example, if a patient is feeling anxious, it displays a simplified explanation using technical terms. The input is the emotion engine's results, and the output is the adjusted display data.

[0505] Step 14:

[0506] The server sends the generated visualization data to the terminal, where the input is the visualization data in the server and the output is the visualization data sent to the terminal.

[0507] Step 15:

[0508] The device provides the generated visualization data to the patient through an interactive interface, allowing the patient to rotate the data and zoom in and out to view the details. The input is the visualization data displayed on the device, and the output is the interface operated by the patient.

[0509] Step 16:

[0510] The user (patient) understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface. The input is the patient's feedback obtained through the interface, and the output is the doctor's additional explanation or response.

[0511] (Application example 2)

[0512] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0513] Autonomous vehicles require systems that can recognize the driver's emotions and state and provide appropriate responses based on that information. However, existing in-vehicle systems have not been able to analyze and recognize the driver's emotions in real time, and dynamically adjust the display to provide appropriate information accordingly. In particular, there is a need for systems that can provide information on relaxation and rest when the driver is feeling stressed or fatigued.

[0514] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0515] In this invention, the server includes a data input means, a preprocessing means, and an analysis and display adjustment means, which allows the server to analyze the driver's emotions in real time and dynamically adjust the display and provide appropriate information accordingly.

[0516] "Data input means" means a device or method for inputting patient medical data and medical images into the system.

[0517] "Preprocessing means" refers to a device or method that processes input medical data or medical images, such as noise removal, resolution adjustment, and contrast enhancement, to convert them into a format that is easy to analyze.

[0518] "Means for inputting doctor's comments and instructions" refers to a device or method that allows a doctor to input comments about a patient's condition and treatment instructions into the system.

[0519] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[0520] The "display means" is a display or interface for visually showing the data converted into the generated format to the user.

[0521] "Emotion recognition means" is a technology that analyzes a user's facial expressions, voice, operation patterns, etc. to identify the user's emotional state.

[0522] A "display adjustment means" is a device or method that dynamically changes the content or format of displayed data based on recognized emotions.

[0523] A "database" is an information system for centrally storing patient medical data, medical images, comments, instructions, etc.

[0524] MODE FOR CARRYING OUT THE INVENTION

[0525] This invention relates to a system for autonomous vehicles that recognizes the driver's emotions in real time and provides appropriate information. Specifically, it analyzes the driver's emotions using a camera and microphone and dynamically adjusts the display content to improve the driver's comfort and safety.

[0526] System Configuration

[0527] The system mainly includes the following elements:

[0528] 1. Data entry method:

[0529] The camera and microphone provide an interface for inputting the driver's facial expressions and voice.

[0530] 2. Pretreatment methods:

[0531] The server receives the input data and performs preprocessing such as image noise removal, resolution adjustment, and audio filtering.

[0532] 3. Emotion recognition means:

[0533] Emotion recognition software analyzes the pre-processed data and recognizes the driver's emotions.

[0534] 4. Generative AI Models:

[0535] The server uses the recognized emotion data to use a generative AI model to provide appropriate information to the driver.

[0536] 5. Display means:

[0537] An in-car display provides the generated data to the driver.

[0538] 6. Database:

[0539] The server has a database that stores emotion data and display content.

[0540] 7. Display adjustment means:

[0541] It includes a software module that dynamically changes the content and format of the data displayed based on the recognized emotion.

[0542] Hardware and Software

[0543] Hardware:

[0544] Camera: Using "Logitech C920" as an example

[0545] Microphone: Using "Blue Yeti" as an example

[0546] In-vehicle display: Using the "Tesla Model 3 dashboard display" as an example

[0547] Server: A server for processing and storing data.

[0548] software:

[0549] OpenCV: A library for processing video data from cameras

[0550] EmotionRecognizer: A library for facial expression recognition

[0551] DriverAssistantDisplay: A module for managing the display content

[0552] Data processing and calculation

[0553] The server preprocesses the data acquired from the camera and microphone and analyzes the driver's emotions using facial and voice recognition technology. The analyzed emotional data is then input into a generative AI model to generate appropriate information content. The generated information is then displayed on the in-vehicle display. The displayed content changes dynamically depending on the recognized emotion. For example, if the driver is feeling stressed, the system can provide information about the nearest rest area or relaxing music.

[0554] Specific examples

[0555] Example: A driver gets into a car and the camera starts capturing their facial expressions. For example, if the driver's facial expression is recognized as "stressed," the dashboard display will offer options such as "Find the nearest rest stop here" or "Play relaxing music." As the driver selects a relaxation measure, the display will dynamically update.

[0556] Example prompt sentence:

[0557] If the driver is stressed, create a list of what information to exclude and what to provide instead, and generate an example script that dynamically adjusts the display accordingly.

[0558] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0559] Step 1:

[0560] When a user gets into a car, the device activates the camera and microphone. The camera captures the driver's face, and the microphone collects audio. The input is the camera video and audio data, which are sent to the server. The server receives these data for preprocessing.

[0561] Step 2:

[0562] The server performs noise reduction and resolution adjustment on the received video data, and filtering on the audio data. It uses OpenCV for noise reduction and resolution adjustment, and an audio processing library for audio filtering. The preprocessed data is output.

[0563] Step 3:

[0564] The server passes the preprocessed video data to EmotionRecognizer, which analyzes the driver's facial expressions. The analysis results output the driver's emotions (e.g., relaxed, stressed, urgent).

[0565] Step 4:

[0566] The server analyzes the pre-processed audio data and performs additional emotion analysis based on volume and tone. The results of the speech recognition and facial expression recognition are integrated to provide a highly accurate emotion assessment. Analyzed emotion data is generated.

[0567] Step 5:

[0568] The server inputs the emotional data into the generative AI model and generates appropriate information content. The data generated using the prompt sentence is output from the generative AI model. Specific information is generated, such as "display route information if the user is relaxed" or "display rest stop information if the user is stressed."

[0569] Step 6:

[0570] The server sends the generated information to the in-vehicle display, where it is displayed by the terminal. The display content is dynamically adjusted in real time according to the recognized emotion. For example, if the driver is judged to be in a "stressed state" in step 5, information such as "The nearest rest stop is here" will be displayed.

[0571] Step 7:

[0572] The device monitors user actions, allowing the driver to make selections and operations based on the displayed information. The selections are then sent back to the server, and in some cases additional information is generated using a generative AI model, and the displayed content is updated.

[0573] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0574] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0575] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0576] [Second embodiment]

[0577] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0578] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0579] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0580] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0581] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0582] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0583] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0584] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0585] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0586] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0587] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0588] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0589] A specific embodiment of the present invention will be described below. The purpose of this system is to convert a patient's medical data and medical images into a format that is easy for the patient to understand, thereby facilitating communication between doctors and patients.

[0590] System Configuration

[0591] The system mainly includes the following elements:

[0592] 1. Data entry method

[0593] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[0594] 2. Pretreatment Methods

[0595] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[0596] 3. Comment and instruction input means

[0597] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[0598] 4. Generative AI Models

[0599] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for patients to understand.

[0600] 5. Display means

[0601] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[0602] 6. Database

[0603] The server has a database that centrally stores medical data, medical images, comments and instructions.

[0604] 7. History Management Methods

[0605] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[0606] Natural language description of the program

[0607] Data Entry and Preprocessing

[0608] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[0609] 2. The device sends the selected file to the server.

[0610] 3. The server receives the data and images and performs preprocessing, such as noise reduction, resolution adjustment, and contrast enhancement. It also standardizes the format and units of medical data.

[0611] Entering comments and instructions

[0612] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[0613] 2. The device sends comments and instructions to the server.

[0614] 3. The server stores the comments and instructions in a database.

[0615] Data analysis and transformation

[0616] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[0617] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[0618] 3. The server receives the generated data and stores it in a database.

[0619] Providing results

[0620] 1. The server sends the generated visualization data to the device.

[0621] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[0622] 3. The user (patient) checks the visualized data and understands their own health condition.

[0623] Specific examples

[0624] For a shoulder fracture diagnosis:

[0625] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[0626] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[0627] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[0628] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[0629] 5. The server sends the generated 3D model to the device.

[0630] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[0631] In this way, this system converts patients' medical data and medical images into easy-to-understand information, supporting communication between doctors and patients and promoting patient health management.

[0632] The processing flow will be explained below.

[0633] Step 1:

[0634] The user (doctor) selects the patient's medical data and medical images on the terminal. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[0635] Step 2:

[0636] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[0637] Step 3:

[0638] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[0639] Step 4:

[0640] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[0641] Step 5:

[0642] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[0643] Step 6:

[0644] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[0645] Step 7:

[0646] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[0647] Step 8:

[0648] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[0649] Step 9:

[0650] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[0651] Step 10:

[0652] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[0653] Step 11:

[0654] The server stores the generated visualization data in a database, where the update history is also managed.

[0655] Step 12:

[0656] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[0657] Step 13:

[0658] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[0659] Step 14:

[0660] Users (patients) can view the visualized data and understand the details of their health condition and treatment. Through the interface, patients can ask questions to their doctor or request additional explanations.

[0661] The above is the specific flow of program processing according to the present invention.

[0662] Example 1

[0663] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0664] In the medical field, it is important to convert medical data and medical images into a format that is easy for patients to understand, but this requires a lot of manual work and specialized knowledge. Furthermore, poor communication between doctors and patients can negatively impact the quality of medical care. The present invention aims to solve these problems by automatically converting medical data and medical images into an easily understandable format, thereby improving communication between doctors and patients.

[0665] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0666] In this invention, the server includes: a means for inputting patient medical data, a means for inputting patient medical images, a means for preprocessing the input medical data and medical images, a means for inputting doctor comments and instructions based on the preprocessed data and images, a means for using a generative artificial intelligence model that integrates and analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for patients to understand (e.g., photos, 3D images, videos), a means for visualizing and interactively displaying the generated data, and a means for having a database that centrally stores patient medical data, medical images, comments, and instructions and for managing their history. This automatically converts medical data and medical images into easy-to-understand formats, facilitating communication between doctors and patients and improving the quality of medical care.

[0667] "Patient Medical Data" means quantitative or qualitative medical information about a patient, such as a patient's diagnostic results, measurements, medical history, and prescription information.

[0668] "Patient medical images" refers to image data that visually captures the internal state of a patient's body, such as CT scans, MRIs, X-rays, and ultrasound images.

[0669] "Preprocessing means" refers to a process for improving the quality of input medical data and medical images by performing processes such as noise removal, resolution adjustment, contrast enhancement, and format matching.

[0670] "Doctor's comments and instructions" refers to text information entered by a doctor regarding the patient's condition and treatment, and includes a diagnosis, treatment plan, and precautions.

[0671] A "generative artificial intelligence model" refers to an algorithm or program that uses machine learning and deep learning technologies to analyze input medical data and medical images and convert them into a format that is easy for patients to understand.

[0672] "Means for interactive display" refers to an interface that displays the generated visualization data in a form that allows the user to manipulate it and enables detailed confirmation and observation.

[0673] "Database" means an information system for centrally storing, efficiently managing, and making accessible medical data, medical images, comments, and instructions.

[0674] "Means for managing history" refers to the process of recording and managing the history of past processing and operations in order to track, analyze, and reuse stored medical data and medical images, as well as comments and instructions.

[0675] This invention is a system that automatically converts a patient's medical data and medical images into a format that is easy for the patient to understand. This aims to improve communication between doctors and patients and enhance the quality of medical care. The system of this invention is mainly composed of three main elements: a server, a terminal, and a user.

[0676] System Configuration

[0677] 1. Data entry method

[0678] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading. The device sends the selected files to the server. For example, the device transfers the files using the HTTP or HTTPS protocol.

[0679] 2. Pretreatment Methods

[0680] The server performs preprocessing such as noise removal, resolution adjustment, and contrast enhancement on the medical data and medical images it receives. Specifically, it uses the OpenCV library to perform image filtering, and the Pandas library to align the format of the medical data and standardize units.

[0681] 3. Comment and instruction input means

[0682] The user (doctor) enters comments about the patient's condition and treatment instructions on the device. The device sends the comments and instructions to the server. For example, the data is sent using JSON format, and the server converts it into a format that can be properly analyzed.

[0683] 4. Generative AI Models

[0684] The server combines the preprocessed data and images with the doctor's comments and converts them into a format that is easy for the patient to understand (e.g., photos, 3D images, videos) using a generative AI model, specifically using Scikit-learn and TensorFlow libraries.

[0685] 5. Display means

[0686] The server sends the generated visualization data to the terminal, which provides an interactive display where the user (patient) can view the transformed data. For example, D3.js or Three.js can be used to interactively display the data.

[0687] 6. Database

[0688] The server will have a database to centrally store patient medical data, medical images, comments, and instructions, and manage the history. MongoDB and MySQL will be used to store the data, and past data can be searched using SQL queries.

[0689] Specific examples

[0690] For a shoulder fracture diagnosis:

[0691] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[0692] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[0693] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of this shoulder."

[0694] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[0695] 5. The server sends the generated 3D model to the device.

[0696] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[0697] Prompt Sentence Examples

[0698] The following is an example of a specific prompt:

[0699] 1. "Enter a CT scan image and generate a 3D model highlighting the fracture."

[0700] 2. "Analyze the cardiac MRI images and mark any abnormalities."

[0701] In this way, the system of the present invention converts patient medical data and medical images into easy-to-understand information, thereby supporting communication between doctors and patients and promoting health management for patients.

[0702] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0703] Step 1: Data entry

[0704] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[0705] Input: Medical data files (e.g., CSV, Excel), medical image files (e.g., CT scans, MRI images)

[0706] Specific operations: The user operates the interface on the device, selects the required file, and clicks the "Upload" button.

[0707] Output: The selected files are sent to the server.

[0708] 2. The device sends the selected file to the server.

[0709] Input: A file selected by the user

[0710] Data processing: Transfer files securely using HTTP or HTTPS protocols.

[0711] Specific operation: The device temporarily saves the file and sends it to the specified API endpoint on the server.

[0712] Output: The server receives the file.

[0713] Step 2: Preprocessing

[0714] 1. The server receives the data and images.

[0715] Input: File sent from terminal

[0716] Specific operation: The server saves the file in the specified directory and records the file's metadata (source, date and time of sending, etc.).

[0717] Output: Saved medical data and medical image files

[0718] 2. The server performs noise reduction, resolution adjustment, and contrast enhancement on the medical images.

[0719] Input: Saved medical image files

[0720] Data processing: Image noise removal, resolution adjustment, and contrast enhancement are performed using the OpenCV library.

[0721] Specific operation: Executes OpenCV functions (e.g., cv2.fastNlMeansDenoising(), cv2.resize(), cv2.equalizeHist()) sequentially.

[0722] Output: Preprocessed medical image files

[0723] 3. The server harmonizes the format and standardizes the units of medical data.

[0724] Input: Stored medical data file

[0725] Data processing: Use the Pandas library to clean the data, align the format, and standardize units.

[0726] Specific operation: Executes Pandas functions (e.g., pd.read_csv(), df.fillna(), df.apply()) sequentially.

[0727] Output: Preprocessed medical data file

[0728] Step 3: Enter comments and instructions

[0729] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[0730] Input: Comments about the patient's condition and treatment instructions

[0731] Specific operation: The user enters a comment or instruction into the interface on the device and clicks the "Send" button.

[0732] Output: Comments and instructions entered

[0733] 2. The device sends comments and instructions to the server.

[0734] Input: Comments or instructions entered

[0735] Data processing: Convert to JSON format and send to the server.

[0736] Specific operation: The terminal converts the input data into JSON format and calls an API to send it to the server.

[0737] Output: The server receives comments and instructions.

[0738] Step 4: Analyze and transform the data

[0739] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[0740] Input: Preprocessed medical data files, medical image files, comments

[0741] Data processing: Integrating data and images into a format that can be input into a generative AI model.

[0742] Specific operation: Convert the data into the input format for the generative AI model and call the AI ​​model.

[0743] Output: Input data for the AI ​​model

[0744] 2. A generative AI model analyzes the data and translates it into a format that is easy for patients to understand.

[0745] Input: Input data for the AI ​​model

[0746] Data Computation: Analysis and transformation using machine learning algorithms and deep learning models.

[0747] Specific operation: Calls the model's prediction function and generates a 3D model and video that highlights the lesion area.

[0748] Output: Generated visualization data (e.g. 3D models, enhanced images)

[0749] 3. The server receives the generated data and stores it in a database.

[0750] Input: Generated visualization data

[0751] Data processing: Save the data in the appropriate format for the database.

[0752] Specific operation: Insert the generated data into a MongoDB or MySQL database.

[0753] Output: Visualization data stored in a database

[0754] Step 5: View and verify

[0755] 1. The server sends the generated visualization data to the device.

[0756] Input: Visualization data stored in a database

[0757] Data processing: Converting data into the format required for sending to the terminal.

[0758] Specific operation: Converts data into JSON or binary format and sends it to the terminal.

[0759] Output: Visualization data sent to the terminal

[0760] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[0761] Input: Visualization data received from the server

[0762] Specific behavior: Display data interactively using D3.js and Three.js.

[0763] Output: User-navigable visualization of data

[0764] 3. The user (patient) checks the visualized data and understands their own health condition.

[0765] Input: Interactive visualization data

[0766] Specific operation: The user drags the data, zooms, etc.

[0767] Output: The user visually understands their health status.

[0768] Step 6: History Management

[0769] 1. The server manages history based on centrally stored data.

[0770] Input: Stored medical data, medical images, comments and instructions

[0771] Data Processing: Organize historical data and make it quickly accessible when needed.

[0772] What you will do: Search and analyze historical data using SQL queries and log search tools.

[0773] Output: Archived data

[0774] 2. The server stores the operation history to facilitate re-analysis and conversion.

[0775] Input: Log data for each operation

[0776] Data processing: Analyze and save log data.

[0777] Specific behavior: Recording operations in log files and dedicated history databases.

[0778] Output: Saved operation history

[0779] (Application example 1)

[0780] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0781] Conventional medical data and medical image processing systems have the problem that it is difficult for patients to understand their own medical condition. Furthermore, the authentication data in facial recognition systems is difficult to understand visually, making it difficult to intuitively confirm authentication results. Therefore, there is a need to facilitate communication between doctors and patients, and between security managers and authentication targets, and to streamline authentication work.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0783] In this invention, the server includes means for inputting patient medical data, means for inputting patient medical images, means for preprocessing the input medical data and medical images, means for inputting doctor comments and instructions based on the preprocessed data and images, means for using a generative artificial intelligence model that analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for the patient to understand, means for displaying the data converted into the generated format, means for inputting facial authentication data, means for preprocessing the input facial authentication data, means for using a generative artificial intelligence model that analyzes and visualizes the preprocessed facial authentication data, and means for interactively displaying the generated facial authentication data. This allows patients to intuitively understand their own medical data and medical images, and also enables the person to be authenticated to visually confirm their own authentication data.

[0784] "Medical data" refers to information such as a patient's health status, diagnosis results, and treatment information.

[0785] "Medical images" refers to image data obtained by medical equipment such as CT scans and MRIs.

[0786] "Preprocessing means" refers to techniques for performing processes such as noise removal and resolution adjustment on medical data, medical images, and facial recognition data.

[0787] "Means for inputting comments and instructions" refers to an interface that allows doctors and administrators to input specific diagnosis results and instructions to the system.

[0788] A "generative artificial intelligence model" refers to artificial intelligence that analyzes input data and converts it into a format that is easy for patients and those being certified to understand.

[0789] "Means for displaying data converted into a generated format" refers to an interface for visually displaying the parsed and converted data.

[0790] "Facial recognition data" refers to image data and feature data relating to an individual's face.

[0791] "Interactive display means" refers to a display method that is easy for users to operate and allows them to intuitively check the data.

[0792] A specific embodiment of the present invention will be described below. The purpose of this system is to convert patient medical data and medical images into a format that is easy for patients to understand, and to provide visually intuitive authentication results in a face recognition system.

[0793] System Configuration

[0794] The system mainly includes the following elements:

[0795] 1. Data entry method

[0796] The terminal provides an interface for inputting the patient's medical data, medical images, and facial recognition data.

[0797] 2. Pretreatment Methods

[0798] The server receives the input medical data, medical images, and facial recognition data, and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[0799] 3. Comment and instruction input means

[0800] The terminal provides an interface that accepts comments and instructions entered by a doctor or security administrator.

[0801] 4. Generative AI Models

[0802] The server combines the pre-processed data, medical images, facial recognition data, and comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for the patient or person being authenticated to understand.

[0803] 5. Display means

[0804] The terminal provides the generated visualization data and provides an interface that allows for interactive display.

[0805] 6. Database

[0806] The server has a database that centrally stores medical data, medical images, facial recognition data, comments and instructions.

[0807] 7. History Management Methods

[0808] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[0809] Hardware and software used

[0810] Smartphones (e.g. iPhone, Android devices)

[0811] Head-mounted displays (HMDs) (e.g., Oculus Quest 2, Microsoft HoloLens 2)

[0812] Server (e.g. AWS EC2 instance, Google Cloud Platform)

[0813] Databases (e.g. Amazon RDS, Google Cloud Firestore)

[0814] Generative AI models (e.g., models using TensorFlow or PyTorch)

[0815] Processing flow

[0816] Data Entry and Preprocessing

[0817] 1. The user (doctor or person to be authenticated) enters medical data, medical images, and facial recognition data using a smartphone or HMD and uploads them to the server.

[0818] 2. The server receives the data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[0819] Entering comments and instructions

[0820] 1. The user (doctor or administrator) enters analysis comments and instructions on the device and sends them to the server.

[0821] 2. The server stores the comments and instructions in a database.

[0822] Data analysis and transformation

[0823] 1. The server integrates the preprocessed data and comments and invokes the generative AI model.

[0824] 2. A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[0825] 3. The server receives the generated data and stores it in a database.

[0826] Providing results

[0827] 1. The server sends the generated visualization data to the device.

[0828] 2. The terminal provides an interactive display, allowing the user (patient or authentication subject) to review the content.

[0829] Specific examples

[0830] For a shoulder fracture diagnosis:

[0831] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system.

[0832] 2. The server preprocesses the CT scan images (adjusting resolution and removing noise).

[0833] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured shoulder."

[0834] 4. The generative AI model analyzes the data and generates a 3D model that highlights the fracture.

[0835] 5. The server sends the generated 3D model to the device.

[0836] 6. The user (patient) checks the 3D model on a smartphone or HMD to visually understand the condition of the shoulder fracture.

[0837] Prompt Sentence Examples

[0838] Target image: CT scan image of shoulder

[0839] Comment: Please emphasize the broken shoulder.

[0840] The following applies to the specific example of face recognition:

[0841] 1. The user (person to be authenticated) takes a photo of their face with their smartphone and uploads it to the authentication system.

[0842] 2. The server preprocesses the image and performs feature extraction.

[0843] 3. The administrator enters a comment saying, "Emphasis on the position of the eyes and the width of the mouth."

[0844] 4. The generative AI model analyzes the data and generates a highlighted 3D face model containing the authentication results.

[0845] 5. The server sends the generated 3D face model and authentication results to the device.

[0846] 6. The user checks the 3D model on a smartphone or HMD and intuitively understands the authentication results and highlighting.

[0847] Prompt Sentence Examples

[0848] Target image: Facial recognition image

[0849] Comment: Emphasize the position of the eyes and the width of the mouth

[0850] In this way, the embodiments for carrying out the present invention have been specifically described.

[0851] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0852] Step 1:

[0853] Users (doctors or individuals to be authenticated) enter medical data, medical images, and facial recognition data using a smartphone or HMD and upload them to the server.

[0854] Input: Patient medical data, medical images, or facial recognition data

[0855] Output: Raw data sent to the server

[0856] Specific action: The user captures data using the device's camera or scanning function and uploads it.

[0857] Step 2:

[0858] The server receives the uploaded data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[0859] Input: Uploaded data (medical data, medical images, facial recognition data)

[0860] Output: Preprocessed data

[0861] Specific operation: The server aligns the format of the medical data, removes noise, and adjusts the image resolution to generate preprocessed data.

[0862] Step 3:

[0863] The user (doctor or administrator) enters analysis comments and instructions on the terminal and sends them to the server.

[0864] Input: Comments and instructions

[0865] Output: Comments and instructions sent to the server

[0866] Specific operation: The user uses the input interface of the terminal to input diagnostic results and analysis instructions, and sends them to the server.

[0867] Step 4:

[0868] The server integrates the preprocessed data and comments and invokes the generative AI model.

[0869] Input: Preprocessed data and comments

[0870] Output: Input data to a generative AI model

[0871] Specific operation: The server integrates the preprocessed data with input instructions and comments to pass them to the generative AI model.

[0872] Step 5:

[0873] A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[0874] Input: Integrated data and comments

[0875] Output: Visualization data (highlighted photos, 3D images, videos)

[0876] What it does: The generative AI model analyzes the data and generates visualizations based on the prompt.

[0877] Step 6:

[0878] The server receives the generated data and stores it in a database.

[0879] Input: Generated visualization data

[0880] Output: Visualization data stored in a database

[0881] Specific operation: The server stores the generated data in a database and provides a history management function.

[0882] Step 7:

[0883] The server transmits the generated visualization data to the terminal.

[0884] Input: Visualization data stored in a database

[0885] Output: Visualization data sent to the terminal

[0886] Specific operation: The server retrieves visualization data from the database and sends it to the terminal.

[0887] Step 8:

[0888] The terminal provides an interactive display, and the user (patient or person being authenticated) can review the content.

[0889] Input: Visualization data sent to the device

[0890] Output: Visualization data reviewed by the user

[0891] Specific operation: The terminal displays the visualized data interactively, and the user can perform operations to check the data.

[0892] Through each of these steps, the system of the present invention can effectively process medical data and facial recognition data and present it to the user in an intuitive and easy-to-understand format.

[0893] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0894] A specific embodiment of the present invention is described below. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[0895] System Configuration

[0896] The system mainly includes the following elements:

[0897] 1. Data entry method

[0898] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[0899] 2. Pretreatment Methods

[0900] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[0901] 3. Comment and instruction input means

[0902] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[0903] 4. Generative AI Models

[0904] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[0905] 5. Display means

[0906] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[0907] 6. Database

[0908] The server has a database that centrally stores medical data, medical images, comments and instructions.

[0909] 7. History Management Methods

[0910] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[0911] 8. Emotion Engine

[0912] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[0913] Natural language description of the program

[0914] Data Entry and Preprocessing

[0915] 1. The user (doctor) selects the patient's medical data and medical images on the device and begins uploading.

[0916] 2. The device sends the selected file to the server.

[0917] 3. The server receives the data and images and performs preprocessing, including image noise reduction, resolution adjustment, and contrast enhancement. For medical data, processing such as unit unification and format conversion is also performed.

[0918] Entering comments and instructions

[0919] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[0920] 2. The device sends comments and instructions to the server.

[0921] 3. The server stores the comments and instructions in a database.

[0922] Data analysis and transformation

[0923] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[0924] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[0925] 3. The server receives the generated data and stores it in a database.

[0926] Emotion analysis and display adjustment

[0927] 1. The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time.

[0928] 2. The emotion engine recognizes the user's emotional state based on the analyzed data. For example, if the user is feeling anxious, it will display an option to immediately contact a call center staff member.

[0929] 3. The server dynamically adjusts the displayed data based on the results of the emotion engine, for example by reducing technical terms or emphasizing visuals to make it easier for users to understand.

[0930] Providing results

[0931] 1. The server sends the generated visualization data to the device.

[0932] 2. The terminal provides an interactive display where the user (patient) can view the converted data. The user can rotate the 3D model and zoom in and out to view the details.

[0933] 3. The user (patient) reviews the visualized data and understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface.

[0934] Specific examples

[0935] For a shoulder fracture diagnosis:

[0936] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[0937] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[0938] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[0939] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[0940] 5. The server sends the generated 3D model to the device.

[0941] 6. The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state.

[0942] 7. The server adapts the display to the patient's emotional state and provides additional information and explanations to reduce anxiety.

[0943] 8. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[0944] In this way, this system converts patients' medical data and medical images into easy-to-understand formats, and also takes into consideration the user's emotional state, thereby supporting communication between doctors and patients and promoting patient health management.

[0945] The processing flow will be explained below.

[0946] Step 1:

[0947] The user (doctor) selects the patient's medical data and medical images on the terminal and starts uploading. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[0948] Step 2:

[0949] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[0950] Step 3:

[0951] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[0952] Step 4:

[0953] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[0954] Step 5:

[0955] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[0956] Step 6:

[0957] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[0958] Step 7:

[0959] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[0960] Step 8:

[0961] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[0962] Step 9:

[0963] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[0964] Step 10:

[0965] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[0966] Step 11:

[0967] The server stores the generated visualization data in a database, where the update history is also managed.

[0968] Step 12:

[0969] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[0970] Step 13:

[0971] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[0972] Step 14:

[0973] The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state. For example, data is collected in real time using a camera or microphone, and emotion analysis is performed.

[0974] Step 15:

[0975] The server receives the analysis results of the emotion engine and dynamically adjusts the display content according to the patient's emotional state. For example, if the patient is highly anxious, it adds detailed explanations or displays options for asking questions to the doctor.

[0976] Step 16:

[0977] The user (patient) can view the tailored visualization data to understand the details of their health condition and treatment. Through the interface, the patient can ask questions to the doctor or request additional explanations.

[0978] In this way, the present invention facilitates communication between doctors and patients and promotes patient health management by visualizing medical data and images in an easy-to-understand manner and dynamically adjusting them according to the patient's emotional state.

[0979] Example 2

[0980] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0981] It is extremely difficult for patients to understand their own medical data and medical images. Poor communication between doctors and patients can lead to inadequate understanding of treatment and diagnosis. It is also important to appropriately adjust the content of explanations based on the patient's emotional state, but this has been difficult with conventional systems.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0983] In this invention, the server includes means for integrating preprocessed data, images, and comments, calling a generative artificial intelligence model, and performing analysis; means for storing the generated data in a database; and means for displaying the generated data and receiving patient responses and questions on a terminal with an interactive display. This allows patients to intuitively understand their own medical data and medical images, enabling effective communication between doctors and patients. Furthermore, the display content can be dynamically adjusted according to the patient's emotional state, deepening the patient's understanding and reducing anxiety.

[0984] "Patient medical data" refers to information such as the patient's health condition, medical history, diagnosis, and treatment progress.

[0985] "Medical images" are image data, such as CT scans, MRIs, and X-rays, that visually show the internal structure and condition of a patient's body.

[0986] "Input means" refers to an interface or device that allows a user to provide medical data or medical images to the system.

[0987] A "server" is a central computer system that receives and processes input data.

[0988] "Preprocessing" refers to processes such as noise removal, resolution adjustment, and format conversion to improve the quality of input data.

[0989] "Doctor's comments and instructions" are written input of the doctor's professional opinions and instructions regarding the patient's condition and treatment plan.

[0990] A "database" is a data storage system for centrally managing and storing information about the system, such as medical data, medical images, comments, and instructions.

[0991] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[0992] An "interactive display" is an interface that allows users to manipulate and view the generated data.

[0993] The "means for accepting patient responses and questions" is an interface that allows patients to ask questions or give feedback on the displayed data.

[0994] A specific embodiment of the present invention will now be described. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and furthermore, dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[0995] System Configuration

[0996] The system mainly includes the following elements:

[0997] 1. Data entry method

[0998] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[0999] 2. Pretreatment Methods

[1000] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[1001] 3. Comment and instruction input means

[1002] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[1003] 4. Generative AI Models

[1004] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[1005] 5. Display means

[1006] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[1007] 6. Database

[1008] The server has a database that centrally stores medical data, medical images, comments and instructions.

[1009] 7. History Management Methods

[1010] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[1011] 8. Emotion Engine

[1012] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[1013] Data Processing

[1014] 1. Data Entry

[1015] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading them. The device then sends this data to the server.

[1016] 2. Pretreatment

[1017] The server preprocesses the medical data and images it receives. For image data, it performs noise removal, resolution adjustment, and contrast enhancement, and for medical data, it standardizes units and converts formats.

[1018] 3. Enter comments and instructions

[1019] The user (doctor) enters comments and instructions about the patient's condition and treatment into the terminal, which then sends this information to the server, which then stores this information in a database.

[1020] 4. Data Analysis and Transformation

[1021] The server combines the preprocessed medical data, medical images, and comments and invokes the generative AI model, which analyzes the data and converts it into a format that is easy for patients to understand (e.g., a 3D image highlighting the lesion).

[1022] 5. Emotion analysis and display adjustment

[1023] The device analyzes the user's (patient's) facial expressions and voice in real time, and the emotion engine uses this information to recognize the user's emotional state. The server dynamically adjusts the data displayed based on the emotion engine's results. For example, if the patient is feeling anxious, the server displays simplified explanations using technical terms.

[1024] 6. Interactive Display

[1025] The server sends the generated visualization data to the terminal, which then provides it to the patient through an interactive interface, allowing the patient to rotate, zoom in and out to view the data.

[1026] Specific examples

[1027] For a shoulder fracture diagnosis:

[1028] Example prompt sentence:

[1029] "Please analyze the CT scan of the patient's shoulder and generate a 3D model that highlights the fracture. The generated model should be intuitive for the patient."

[1030] By inputting this prompt into the generative AI model, a 3D image that is easy for patients to understand is generated, highlighting the fractured shoulder area, allowing patients to intuitively understand their condition and facilitating communication with their doctors.

[1031] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1032] Step 1:

[1033] A user (doctor) uses a terminal to select a patient's medical data (e.g., diagnosis results, measurements) and medical images (e.g., CT scans, MRI images) and initiates uploading to the system. The input at this time is the medical data and medical images selected by the user, and the output is the state in which these data have been uploaded to the terminal.

[1034] Step 2:

[1035] The terminal transmits the medical data and medical images selected by the user to the server. The input is the medical data and medical images on the terminal, and the output is the data transmitted to the server. HTTPS is used as the transmission protocol.

[1036] Step 3:

[1037] The server receives medical data and medical images sent from the terminal. The input is the data and images sent to the server, and the output is the data that has been received inside the server. The server reads this data for preprocessing.

[1038] Step 4:

[1039] The server preprocesses the medical data and medical images it receives. First, it performs noise removal, resolution adjustment, and contrast enhancement on the image data. Next, it standardizes the units and converts the format of the medical data. The input at this time is the raw data received by the server, and the output is the preprocessed medical data and medical images.

[1040] Step 5:

[1041] The user (doctor) inputs comments and instructions about the patient's condition and treatment into the terminal and sends the data to the system. The input is the comments and instructions entered by the doctor, and the output is the comments and instructions sent from the terminal to the server.

[1042] Step 6:

[1043] The terminal sends the entered comments and instructions to the server. The input here is the comments and instructions entered on the terminal, and the output is the data sent to the server.

[1044] Step 7:

[1045] The server integrates the preprocessed medical data, medical images, and comments and stores them in a database, where the inputs are the preprocessed data, images, and doctor's comments and instructions, and the output is the integrated data stored in the database.

[1046] Step 8:

[1047] The server generates a prompt to invoke the generative AI model. For example, it generates a prompt such as, "Analyze the CT scan image of the patient's shoulder and generate a 3D model that highlights the fractured area." The input is the integrated data and comments, and the output is the generated prompt.

[1048] Step 9:

[1049] The server sends data and prompts to the generative AI model for analysis and conversion. The input is the data integrated with the generated prompt, and the output is the data and prompt sent to the generative AI model.

[1050] Step 10:

[1051] The generative AI model analyzes the data and converts it into a format that is easy for the patient to understand (e.g., a photograph highlighting the lesion or a 3D image). The input to the generative AI model is the data and prompts it receives, and the output is the converted visualization data.

[1052] Step 11:

[1053] The server receives the analysis results from the generative AI model and stores them in a database. The input at this time is the analysis results from the generative AI model, and the output is the visualization data stored in the database.

[1054] Step 12:

[1055] The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time. The input is the patient's facial expressions, voice, and operation patterns, and the output is analyzed emotional data.

[1056] Step 13:

[1057] The server dynamically adjusts the displayed data based on the results of the emotion engine. For example, if a patient is feeling anxious, it displays a simplified explanation using technical terms. The input is the emotion engine's results, and the output is the adjusted display data.

[1058] Step 14:

[1059] The server sends the generated visualization data to the terminal, where the input is the visualization data in the server and the output is the visualization data sent to the terminal.

[1060] Step 15:

[1061] The device provides the generated visualization data to the patient through an interactive interface, allowing the patient to rotate the data and zoom in and out to view the details. The input is the visualization data displayed on the device, and the output is the interface operated by the patient.

[1062] Step 16:

[1063] The user (patient) understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface. The input is the patient's feedback obtained through the interface, and the output is the doctor's additional explanation or response.

[1064] (Application example 2)

[1065] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1066] Autonomous vehicles require systems that can recognize the driver's emotions and state and provide appropriate responses based on that information. However, existing in-vehicle systems have not been able to analyze and recognize the driver's emotions in real time, and dynamically adjust the display to provide appropriate information accordingly. In particular, there is a need for systems that can provide information on relaxation and rest when the driver is feeling stressed or fatigued.

[1067] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1068] In this invention, the server includes a data input means, a preprocessing means, and an analysis and display adjustment means, which allows the server to analyze the driver's emotions in real time and dynamically adjust the display and provide appropriate information accordingly.

[1069] "Data input means" means a device or method for inputting patient medical data and medical images into the system.

[1070] "Preprocessing means" refers to a device or method that processes input medical data or medical images, such as noise removal, resolution adjustment, and contrast enhancement, to convert them into a format that is easy to analyze.

[1071] "Means for inputting doctor's comments and instructions" refers to a device or method that allows a doctor to input comments about a patient's condition and treatment instructions into the system.

[1072] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[1073] The "display means" is a display or interface for visually showing the data converted into the generated format to the user.

[1074] "Emotion recognition means" is a technology that analyzes a user's facial expressions, voice, operation patterns, etc. to identify the user's emotional state.

[1075] A "display adjustment means" is a device or method that dynamically changes the content or format of displayed data based on recognized emotions.

[1076] A "database" is an information system for centrally storing patient medical data, medical images, comments, instructions, etc.

[1077] MODE FOR CARRYING OUT THE INVENTION

[1078] This invention relates to a system for autonomous vehicles that recognizes the driver's emotions in real time and provides appropriate information. Specifically, it analyzes the driver's emotions using a camera and microphone and dynamically adjusts the display content to improve the driver's comfort and safety.

[1079] System Configuration

[1080] The system mainly includes the following elements:

[1081] 1. Data entry method:

[1082] The camera and microphone provide an interface for inputting the driver's facial expressions and voice.

[1083] 2. Pretreatment methods:

[1084] The server receives the input data and performs preprocessing such as image noise removal, resolution adjustment, and audio filtering.

[1085] 3. Emotion recognition means:

[1086] Emotion recognition software analyzes the pre-processed data and recognizes the driver's emotions.

[1087] 4. Generative AI Models:

[1088] The server uses the recognized emotion data to use a generative AI model to provide appropriate information to the driver.

[1089] 5. Display means:

[1090] An in-car display provides the generated data to the driver.

[1091] 6. Database:

[1092] The server has a database that stores emotion data and display content.

[1093] 7. Display adjustment means:

[1094] It includes a software module that dynamically changes the content and format of the data displayed based on the recognized emotion.

[1095] Hardware and Software

[1096] Hardware:

[1097] Camera: Using "Logitech C920" as an example

[1098] Microphone: Using "Blue Yeti" as an example

[1099] In-vehicle display: Using the "Tesla Model 3 dashboard display" as an example

[1100] Server: A server for processing and storing data.

[1101] software:

[1102] OpenCV: A library for processing video data from cameras

[1103] EmotionRecognizer: A library for facial expression recognition

[1104] DriverAssistantDisplay: A module for managing the display content

[1105] Data processing and calculation

[1106] The server preprocesses the data acquired from the camera and microphone and analyzes the driver's emotions using facial and voice recognition technology. The analyzed emotional data is then input into a generative AI model to generate appropriate information content. The generated information is then displayed on the in-vehicle display. The displayed content changes dynamically depending on the recognized emotion. For example, if the driver is feeling stressed, the system can provide information about the nearest rest area or relaxing music.

[1107] Specific examples

[1108] Example: A driver gets into a car and the camera starts capturing their facial expressions. For example, if the driver's facial expression is recognized as "stressed," the dashboard display will offer options such as "Find the nearest rest stop here" or "Play relaxing music." As the driver selects a relaxation measure, the display will dynamically update.

[1109] Example prompt sentence:

[1110] If the driver is stressed, create a list of what information to exclude and what to provide instead, and generate an example script that dynamically adjusts the display accordingly.

[1111] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1112] Step 1:

[1113] When a user gets into a car, the device activates the camera and microphone. The camera captures the driver's face, and the microphone collects audio. The input is the camera video and audio data, which are sent to the server. The server receives these data for preprocessing.

[1114] Step 2:

[1115] The server performs noise reduction and resolution adjustment on the received video data, and filtering on the audio data. It uses OpenCV for noise reduction and resolution adjustment, and an audio processing library for audio filtering. The preprocessed data is output.

[1116] Step 3:

[1117] The server passes the preprocessed video data to EmotionRecognizer, which analyzes the driver's facial expressions. The analysis results output the driver's emotions (e.g., relaxed, stressed, urgent).

[1118] Step 4:

[1119] The server analyzes the pre-processed audio data and performs additional emotion analysis based on volume and tone. The results of the speech recognition and facial expression recognition are integrated to provide a highly accurate emotion assessment. Analyzed emotion data is generated.

[1120] Step 5:

[1121] The server inputs the emotional data into the generative AI model and generates appropriate information content. The data generated using the prompt sentence is output from the generative AI model. Specific information is generated, such as "display route information if the user is relaxed" or "display rest stop information if the user is stressed."

[1122] Step 6:

[1123] The server sends the generated information to the in-vehicle display, where it is displayed by the terminal. The display content is dynamically adjusted in real time according to the recognized emotion. For example, if the driver is judged to be in a "stressed state" in step 5, information such as "The nearest rest stop is here" will be displayed.

[1124] Step 7:

[1125] The device monitors user actions, allowing the driver to make selections and operations based on the displayed information. The selections are then sent back to the server, and in some cases additional information is generated using a generative AI model, and the displayed content is updated.

[1126] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1127] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1128] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1129] [Third embodiment]

[1130] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1131] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1132] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1133] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1134] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1135] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1136] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1137] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1138] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1139] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1140] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1141] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1142] A specific embodiment of the present invention will be described below. The purpose of this system is to convert a patient's medical data and medical images into a format that is easy for the patient to understand, thereby facilitating communication between doctors and patients.

[1143] System Configuration

[1144] The system mainly includes the following elements:

[1145] 1. Data entry method

[1146] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[1147] 2. Pretreatment Methods

[1148] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[1149] 3. Comment and instruction input means

[1150] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[1151] 4. Generative AI Models

[1152] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for patients to understand.

[1153] 5. Display means

[1154] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[1155] 6. Database

[1156] The server has a database that centrally stores medical data, medical images, comments and instructions.

[1157] 7. History Management Methods

[1158] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[1159] Natural language description of the program

[1160] Data Entry and Preprocessing

[1161] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[1162] 2. The device sends the selected file to the server.

[1163] 3. The server receives the data and images and performs preprocessing, such as noise reduction, resolution adjustment, and contrast enhancement. It also standardizes the format and units of medical data.

[1164] Entering comments and instructions

[1165] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[1166] 2. The device sends comments and instructions to the server.

[1167] 3. The server stores the comments and instructions in a database.

[1168] Data analysis and transformation

[1169] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[1170] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[1171] 3. The server receives the generated data and stores it in a database.

[1172] Providing results

[1173] 1. The server sends the generated visualization data to the device.

[1174] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[1175] 3. The user (patient) checks the visualized data and understands their own health condition.

[1176] Specific examples

[1177] For a shoulder fracture diagnosis:

[1178] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[1179] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[1180] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[1181] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[1182] 5. The server sends the generated 3D model to the device.

[1183] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[1184] In this way, this system converts patients' medical data and medical images into easy-to-understand information, supporting communication between doctors and patients and promoting patient health management.

[1185] The processing flow will be explained below.

[1186] Step 1:

[1187] The user (doctor) selects the patient's medical data and medical images on the terminal. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[1188] Step 2:

[1189] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[1190] Step 3:

[1191] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[1192] Step 4:

[1193] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[1194] Step 5:

[1195] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[1196] Step 6:

[1197] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[1198] Step 7:

[1199] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[1200] Step 8:

[1201] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[1202] Step 9:

[1203] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[1204] Step 10:

[1205] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[1206] Step 11:

[1207] The server stores the generated visualization data in a database, where the update history is also managed.

[1208] Step 12:

[1209] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[1210] Step 13:

[1211] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[1212] Step 14:

[1213] Users (patients) can view the visualized data and understand the details of their health condition and treatment. Through the interface, patients can ask questions to their doctor or request additional explanations.

[1214] The above is the specific flow of program processing according to the present invention.

[1215] Example 1

[1216] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1217] In the medical field, it is important to convert medical data and medical images into a format that is easy for patients to understand, but this requires a lot of manual work and specialized knowledge. Furthermore, poor communication between doctors and patients can negatively impact the quality of medical care. The present invention aims to solve these problems by automatically converting medical data and medical images into an easily understandable format, thereby improving communication between doctors and patients.

[1218] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1219] In this invention, the server includes: a means for inputting patient medical data, a means for inputting patient medical images, a means for preprocessing the input medical data and medical images, a means for inputting doctor comments and instructions based on the preprocessed data and images, a means for using a generative artificial intelligence model that integrates and analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for patients to understand (e.g., photos, 3D images, videos), a means for visualizing and interactively displaying the generated data, and a means for having a database that centrally stores patient medical data, medical images, comments, and instructions and for managing their history. This automatically converts medical data and medical images into easy-to-understand formats, facilitating communication between doctors and patients and improving the quality of medical care.

[1220] "Patient Medical Data" means quantitative or qualitative medical information about a patient, such as a patient's diagnostic results, measurements, medical history, and prescription information.

[1221] "Patient medical images" refers to image data that visually captures the internal state of a patient's body, such as CT scans, MRIs, X-rays, and ultrasound images.

[1222] "Preprocessing means" refers to a process for improving the quality of input medical data and medical images by performing processes such as noise removal, resolution adjustment, contrast enhancement, and format matching.

[1223] "Doctor's comments and instructions" refers to text information entered by a doctor regarding the patient's condition and treatment, and includes a diagnosis, treatment plan, and precautions.

[1224] A "generative artificial intelligence model" refers to an algorithm or program that uses machine learning and deep learning technologies to analyze input medical data and medical images and convert them into a format that is easy for patients to understand.

[1225] "Means for interactive display" refers to an interface that displays the generated visualization data in a form that allows the user to manipulate it and enables detailed confirmation and observation.

[1226] "Database" means an information system for centrally storing, efficiently managing, and making accessible medical data, medical images, comments, and instructions.

[1227] "Means for managing history" refers to the process of recording and managing the history of past processing and operations in order to track, analyze, and reuse stored medical data and medical images, as well as comments and instructions.

[1228] This invention is a system that automatically converts a patient's medical data and medical images into a format that is easy for the patient to understand. This aims to improve communication between doctors and patients and enhance the quality of medical care. The system of this invention is mainly composed of three main elements: a server, a terminal, and a user.

[1229] System Configuration

[1230] 1. Data entry method

[1231] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading. The device sends the selected files to the server. For example, the device transfers the files using the HTTP or HTTPS protocol.

[1232] 2. Pretreatment Methods

[1233] The server performs preprocessing such as noise removal, resolution adjustment, and contrast enhancement on the medical data and medical images it receives. Specifically, it uses the OpenCV library to perform image filtering, and the Pandas library to align the format of the medical data and standardize units.

[1234] 3. Comment and instruction input means

[1235] The user (doctor) enters comments about the patient's condition and treatment instructions on the device. The device sends the comments and instructions to the server. For example, the data is sent using JSON format, and the server converts it into a format that can be properly analyzed.

[1236] 4. Generative AI Models

[1237] The server combines the preprocessed data and images with the doctor's comments and converts them into a format that is easy for the patient to understand (e.g., photos, 3D images, videos) using a generative AI model, specifically using Scikit-learn and TensorFlow libraries.

[1238] 5. Display means

[1239] The server sends the generated visualization data to the terminal, which provides an interactive display where the user (patient) can view the transformed data. For example, D3.js or Three.js can be used to interactively display the data.

[1240] 6. Database

[1241] The server will have a database to centrally store patient medical data, medical images, comments, and instructions, and manage the history. MongoDB and MySQL will be used to store the data, and past data can be searched using SQL queries.

[1242] Specific examples

[1243] For a shoulder fracture diagnosis:

[1244] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[1245] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[1246] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of this shoulder."

[1247] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[1248] 5. The server sends the generated 3D model to the device.

[1249] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[1250] Prompt Sentence Examples

[1251] The following is an example of a specific prompt:

[1252] 1. "Enter a CT scan image and generate a 3D model highlighting the fracture."

[1253] 2. "Analyze the cardiac MRI images and mark any abnormalities."

[1254] In this way, the system of the present invention converts patient medical data and medical images into easy-to-understand information, thereby supporting communication between doctors and patients and promoting health management for patients.

[1255] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1256] Step 1: Data entry

[1257] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[1258] Input: Medical data files (e.g., CSV, Excel), medical image files (e.g., CT scans, MRI images)

[1259] Specific operations: The user operates the interface on the device, selects the required file, and clicks the "Upload" button.

[1260] Output: The selected files are sent to the server.

[1261] 2. The device sends the selected file to the server.

[1262] Input: A file selected by the user

[1263] Data processing: Transfer files securely using HTTP or HTTPS protocols.

[1264] Specific operation: The device temporarily saves the file and sends it to the specified API endpoint on the server.

[1265] Output: The server receives the file.

[1266] Step 2: Preprocessing

[1267] 1. The server receives the data and images.

[1268] Input: File sent from terminal

[1269] Specific operation: The server saves the file in the specified directory and records the file's metadata (source, date and time of sending, etc.).

[1270] Output: Saved medical data and medical image files

[1271] 2. The server performs noise reduction, resolution adjustment, and contrast enhancement on the medical images.

[1272] Input: Saved medical image files

[1273] Data processing: Image noise removal, resolution adjustment, and contrast enhancement are performed using the OpenCV library.

[1274] Specific operation: Executes OpenCV functions (e.g., cv2.fastNlMeansDenoising(), cv2.resize(), cv2.equalizeHist()) sequentially.

[1275] Output: Preprocessed medical image files

[1276] 3. The server harmonizes the format and standardizes the units of medical data.

[1277] Input: Stored medical data file

[1278] Data processing: Use the Pandas library to clean the data, align the format, and standardize units.

[1279] Specific operation: Executes Pandas functions (e.g., pd.read_csv(), df.fillna(), df.apply()) sequentially.

[1280] Output: Preprocessed medical data file

[1281] Step 3: Enter comments and instructions

[1282] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[1283] Input: Comments about the patient's condition and treatment instructions

[1284] Specific operation: The user enters a comment or instruction into the interface on the device and clicks the "Send" button.

[1285] Output: Comments and instructions entered

[1286] 2. The device sends comments and instructions to the server.

[1287] Input: Comments or instructions entered

[1288] Data processing: Convert to JSON format and send to the server.

[1289] Specific operation: The terminal converts the input data into JSON format and calls an API to send it to the server.

[1290] Output: The server receives comments and instructions.

[1291] Step 4: Analyze and transform the data

[1292] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[1293] Input: Preprocessed medical data files, medical image files, comments

[1294] Data processing: Integrating data and images into a format that can be input into a generative AI model.

[1295] Specific operation: Convert the data into the input format for the generative AI model and call the AI ​​model.

[1296] Output: Input data for the AI ​​model

[1297] 2. A generative AI model analyzes the data and translates it into a format that is easy for patients to understand.

[1298] Input: Input data for the AI ​​model

[1299] Data Computation: Analysis and transformation using machine learning algorithms and deep learning models.

[1300] Specific operation: Calls the model's prediction function and generates a 3D model and video that highlights the lesion area.

[1301] Output: Generated visualization data (e.g. 3D models, enhanced images)

[1302] 3. The server receives the generated data and stores it in a database.

[1303] Input: Generated visualization data

[1304] Data processing: Save the data in the appropriate format for the database.

[1305] Specific operation: Insert the generated data into a MongoDB or MySQL database.

[1306] Output: Visualization data stored in a database

[1307] Step 5: View and verify

[1308] 1. The server sends the generated visualization data to the device.

[1309] Input: Visualization data stored in a database

[1310] Data processing: Converting data into the format required for sending to the terminal.

[1311] Specific operation: Converts data into JSON or binary format and sends it to the terminal.

[1312] Output: Visualization data sent to the terminal

[1313] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[1314] Input: Visualization data received from the server

[1315] Specific behavior: Display data interactively using D3.js and Three.js.

[1316] Output: User-navigable visualization of data

[1317] 3. The user (patient) checks the visualized data and understands their own health condition.

[1318] Input: Interactive visualization data

[1319] Specific operation: The user drags the data, zooms, etc.

[1320] Output: The user visually understands their health status.

[1321] Step 6: History Management

[1322] 1. The server manages history based on centrally stored data.

[1323] Input: Stored medical data, medical images, comments and instructions

[1324] Data Processing: Organize historical data and make it quickly accessible when needed.

[1325] What you will do: Search and analyze historical data using SQL queries and log search tools.

[1326] Output: Archived data

[1327] 2. The server stores the operation history to facilitate re-analysis and conversion.

[1328] Input: Log data for each operation

[1329] Data processing: Analyze and save log data.

[1330] Specific behavior: Recording operations in log files and dedicated history databases.

[1331] Output: Saved operation history

[1332] (Application example 1)

[1333] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1334] Conventional medical data and medical image processing systems have the problem that it is difficult for patients to understand their own medical condition. Furthermore, the authentication data in facial recognition systems is difficult to understand visually, making it difficult to intuitively confirm authentication results. Therefore, there is a need to facilitate communication between doctors and patients, and between security managers and authentication targets, and to streamline authentication work.

[1335] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1336] In this invention, the server includes means for inputting patient medical data, means for inputting patient medical images, means for preprocessing the input medical data and medical images, means for inputting doctor comments and instructions based on the preprocessed data and images, means for using a generative artificial intelligence model that analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for the patient to understand, means for displaying the data converted into the generated format, means for inputting facial authentication data, means for preprocessing the input facial authentication data, means for using a generative artificial intelligence model that analyzes and visualizes the preprocessed facial authentication data, and means for interactively displaying the generated facial authentication data. This allows patients to intuitively understand their own medical data and medical images, and also enables the person to be authenticated to visually confirm their own authentication data.

[1337] "Medical data" refers to information such as a patient's health status, diagnosis results, and treatment information.

[1338] "Medical images" refers to image data obtained by medical equipment such as CT scans and MRIs.

[1339] "Preprocessing means" refers to techniques for performing processes such as noise removal and resolution adjustment on medical data, medical images, and facial recognition data.

[1340] "Means for inputting comments and instructions" refers to an interface that allows doctors and administrators to input specific diagnosis results and instructions to the system.

[1341] A "generative artificial intelligence model" refers to artificial intelligence that analyzes input data and converts it into a format that is easy for patients and those being certified to understand.

[1342] "Means for displaying data converted into a generated format" refers to an interface for visually displaying the parsed and converted data.

[1343] "Facial recognition data" refers to image data and feature data relating to an individual's face.

[1344] "Interactive display means" refers to a display method that is easy for users to operate and allows them to intuitively check the data.

[1345] A specific embodiment of the present invention will be described below. The purpose of this system is to convert patient medical data and medical images into a format that is easy for patients to understand, and to provide visually intuitive authentication results in a face recognition system.

[1346] System Configuration

[1347] The system mainly includes the following elements:

[1348] 1. Data entry method

[1349] The terminal provides an interface for inputting the patient's medical data, medical images, and facial recognition data.

[1350] 2. Pretreatment Methods

[1351] The server receives the input medical data, medical images, and facial recognition data, and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[1352] 3. Comment and instruction input means

[1353] The terminal provides an interface that accepts comments and instructions entered by a doctor or security administrator.

[1354] 4. Generative AI Models

[1355] The server combines the pre-processed data, medical images, facial recognition data, and comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for the patient or person being authenticated to understand.

[1356] 5. Display means

[1357] The terminal provides the generated visualization data and provides an interface that allows for interactive display.

[1358] 6. Database

[1359] The server has a database that centrally stores medical data, medical images, facial recognition data, comments and instructions.

[1360] 7. History Management Methods

[1361] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[1362] Hardware and software used

[1363] Smartphones (e.g. iPhone, Android devices)

[1364] Head-mounted displays (HMDs) (e.g., Oculus Quest 2, Microsoft HoloLens 2)

[1365] Server (e.g. AWS EC2 instance, Google Cloud Platform)

[1366] Databases (e.g. Amazon RDS, Google Cloud Firestore)

[1367] Generative AI models (e.g., models using TensorFlow or PyTorch)

[1368] Processing flow

[1369] Data Entry and Preprocessing

[1370] 1. The user (doctor or person to be authenticated) enters medical data, medical images, and facial recognition data using a smartphone or HMD and uploads them to the server.

[1371] 2. The server receives the data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[1372] Entering comments and instructions

[1373] 1. The user (doctor or administrator) enters analysis comments and instructions on the device and sends them to the server.

[1374] 2. The server stores the comments and instructions in a database.

[1375] Data analysis and transformation

[1376] 1. The server integrates the preprocessed data and comments and invokes the generative AI model.

[1377] 2. A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[1378] 3. The server receives the generated data and stores it in a database.

[1379] Providing results

[1380] 1. The server sends the generated visualization data to the device.

[1381] 2. The terminal provides an interactive display, allowing the user (patient or authentication subject) to review the content.

[1382] Specific examples

[1383] For a shoulder fracture diagnosis:

[1384] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system.

[1385] 2. The server preprocesses the CT scan images (adjusting resolution and removing noise).

[1386] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured shoulder."

[1387] 4. The generative AI model analyzes the data and generates a 3D model that highlights the fracture.

[1388] 5. The server sends the generated 3D model to the device.

[1389] 6. The user (patient) checks the 3D model on a smartphone or HMD to visually understand the condition of the shoulder fracture.

[1390] Prompt Sentence Examples

[1391] Target image: CT scan image of shoulder

[1392] Comment: Please emphasize the broken shoulder.

[1393] The following applies to the specific example of face recognition:

[1394] 1. The user (person to be authenticated) takes a photo of their face with their smartphone and uploads it to the authentication system.

[1395] 2. The server preprocesses the image and performs feature extraction.

[1396] 3. The administrator enters a comment saying, "Emphasis on the position of the eyes and the width of the mouth."

[1397] 4. The generative AI model analyzes the data and generates a highlighted 3D face model containing the authentication results.

[1398] 5. The server sends the generated 3D face model and authentication results to the device.

[1399] 6. The user checks the 3D model on a smartphone or HMD and intuitively understands the authentication results and highlighting.

[1400] Prompt Sentence Examples

[1401] Target image: Facial recognition image

[1402] Comment: Emphasize the position of the eyes and the width of the mouth

[1403] In this way, the embodiments for carrying out the present invention have been specifically described.

[1404] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1405] Step 1:

[1406] Users (doctors or individuals to be authenticated) enter medical data, medical images, and facial recognition data using a smartphone or HMD and upload them to the server.

[1407] Input: Patient medical data, medical images, or facial recognition data

[1408] Output: Raw data sent to the server

[1409] Specific action: The user captures data using the device's camera or scanning function and uploads it.

[1410] Step 2:

[1411] The server receives the uploaded data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[1412] Input: Uploaded data (medical data, medical images, facial recognition data)

[1413] Output: Preprocessed data

[1414] Specific operation: The server aligns the format of the medical data, removes noise, and adjusts the image resolution to generate preprocessed data.

[1415] Step 3:

[1416] The user (doctor or administrator) enters analysis comments and instructions on the terminal and sends them to the server.

[1417] Input: Comments and instructions

[1418] Output: Comments and instructions sent to the server

[1419] Specific operation: The user uses the input interface of the terminal to input diagnostic results and analysis instructions, and sends them to the server.

[1420] Step 4:

[1421] The server integrates the preprocessed data and comments and invokes the generative AI model.

[1422] Input: Preprocessed data and comments

[1423] Output: Input data to a generative AI model

[1424] Specific operation: The server integrates the preprocessed data with input instructions and comments to pass them to the generative AI model.

[1425] Step 5:

[1426] A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[1427] Input: Integrated data and comments

[1428] Output: Visualization data (highlighted photos, 3D images, videos)

[1429] What it does: The generative AI model analyzes the data and generates visualizations based on the prompt.

[1430] Step 6:

[1431] The server receives the generated data and stores it in a database.

[1432] Input: Generated visualization data

[1433] Output: Visualization data stored in a database

[1434] Specific operation: The server stores the generated data in a database and provides a history management function.

[1435] Step 7:

[1436] The server transmits the generated visualization data to the terminal.

[1437] Input: Visualization data stored in a database

[1438] Output: Visualization data sent to the terminal

[1439] Specific operation: The server retrieves visualization data from the database and sends it to the terminal.

[1440] Step 8:

[1441] The terminal provides an interactive display, and the user (patient or person being authenticated) can review the content.

[1442] Input: Visualization data sent to the device

[1443] Output: Visualization data reviewed by the user

[1444] Specific operation: The terminal displays the visualized data interactively, and the user can perform operations to check the data.

[1445] Through each of these steps, the system of the present invention can effectively process medical data and facial recognition data and present it to the user in an intuitive and easy-to-understand format.

[1446] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1447] A specific embodiment of the present invention is described below. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[1448] System Configuration

[1449] The system mainly includes the following elements:

[1450] 1. Data entry method

[1451] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[1452] 2. Pretreatment Methods

[1453] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[1454] 3. Comment and instruction input means

[1455] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[1456] 4. Generative AI Models

[1457] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[1458] 5. Display means

[1459] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[1460] 6. Database

[1461] The server has a database that centrally stores medical data, medical images, comments and instructions.

[1462] 7. History Management Methods

[1463] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[1464] 8. Emotion Engine

[1465] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[1466] Natural language description of the program

[1467] Data Entry and Preprocessing

[1468] 1. The user (doctor) selects the patient's medical data and medical images on the device and begins uploading.

[1469] 2. The device sends the selected file to the server.

[1470] 3. The server receives the data and images and performs preprocessing, including image noise reduction, resolution adjustment, and contrast enhancement. For medical data, processing such as unit unification and format conversion is also performed.

[1471] Entering comments and instructions

[1472] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[1473] 2. The device sends comments and instructions to the server.

[1474] 3. The server stores the comments and instructions in a database.

[1475] Data analysis and transformation

[1476] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[1477] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[1478] 3. The server receives the generated data and stores it in a database.

[1479] Emotion analysis and display adjustment

[1480] 1. The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time.

[1481] 2. The emotion engine recognizes the user's emotional state based on the analyzed data. For example, if the user is feeling anxious, it will display an option to immediately contact a call center staff member.

[1482] 3. The server dynamically adjusts the displayed data based on the results of the emotion engine, for example by reducing technical terms or emphasizing visuals to make it easier for users to understand.

[1483] Providing results

[1484] 1. The server sends the generated visualization data to the device.

[1485] 2. The terminal provides an interactive display where the user (patient) can view the converted data. The user can rotate the 3D model and zoom in and out to view the details.

[1486] 3. The user (patient) reviews the visualized data and understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface.

[1487] Specific examples

[1488] For a shoulder fracture diagnosis:

[1489] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[1490] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[1491] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[1492] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[1493] 5. The server sends the generated 3D model to the device.

[1494] 6. The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state.

[1495] 7. The server adapts the display to the patient's emotional state and provides additional information and explanations to reduce anxiety.

[1496] 8. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[1497] In this way, this system converts patients' medical data and medical images into easy-to-understand formats, and also takes into consideration the user's emotional state, thereby supporting communication between doctors and patients and promoting patient health management.

[1498] The processing flow will be explained below.

[1499] Step 1:

[1500] The user (doctor) selects the patient's medical data and medical images on the terminal and starts uploading. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[1501] Step 2:

[1502] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[1503] Step 3:

[1504] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[1505] Step 4:

[1506] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[1507] Step 5:

[1508] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[1509] Step 6:

[1510] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[1511] Step 7:

[1512] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[1513] Step 8:

[1514] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[1515] Step 9:

[1516] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[1517] Step 10:

[1518] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[1519] Step 11:

[1520] The server stores the generated visualization data in a database, where the update history is also managed.

[1521] Step 12:

[1522] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[1523] Step 13:

[1524] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[1525] Step 14:

[1526] The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state. For example, data is collected in real time using a camera or microphone, and emotion analysis is performed.

[1527] Step 15:

[1528] The server receives the analysis results of the emotion engine and dynamically adjusts the display content according to the patient's emotional state. For example, if the patient is highly anxious, it adds detailed explanations or displays options for asking questions to the doctor.

[1529] Step 16:

[1530] The user (patient) can view the tailored visualization data to understand the details of their health condition and treatment. Through the interface, the patient can ask questions to the doctor or request additional explanations.

[1531] In this way, the present invention facilitates communication between doctors and patients and promotes patient health management by visualizing medical data and images in an easy-to-understand manner and dynamically adjusting them according to the patient's emotional state.

[1532] Example 2

[1533] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1534] It is extremely difficult for patients to understand their own medical data and medical images. Poor communication between doctors and patients can lead to inadequate understanding of treatment and diagnosis. It is also important to appropriately adjust the content of explanations based on the patient's emotional state, but this has been difficult with conventional systems.

[1535] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1536] In this invention, the server includes means for integrating preprocessed data, images, and comments, calling a generative artificial intelligence model, and performing analysis; means for storing the generated data in a database; and means for displaying the generated data and receiving patient responses and questions on a terminal with an interactive display. This allows patients to intuitively understand their own medical data and medical images, enabling effective communication between doctors and patients. Furthermore, the display content can be dynamically adjusted according to the patient's emotional state, deepening the patient's understanding and reducing anxiety.

[1537] "Patient medical data" refers to information such as the patient's health condition, medical history, diagnosis, and treatment progress.

[1538] "Medical images" are image data, such as CT scans, MRIs, and X-rays, that visually show the internal structure and condition of a patient's body.

[1539] "Input means" refers to an interface or device that allows a user to provide medical data or medical images to the system.

[1540] A "server" is a central computer system that receives and processes input data.

[1541] "Preprocessing" refers to processes such as noise removal, resolution adjustment, and format conversion to improve the quality of input data.

[1542] "Doctor's comments and instructions" are written input of the doctor's professional opinions and instructions regarding the patient's condition and treatment plan.

[1543] A "database" is a data storage system for centrally managing and storing information about the system, such as medical data, medical images, comments, and instructions.

[1544] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[1545] An "interactive display" is an interface that allows users to manipulate and view the generated data.

[1546] The "means for accepting patient responses and questions" is an interface that allows patients to ask questions or give feedback on the displayed data.

[1547] A specific embodiment of the present invention will now be described. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and furthermore, dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[1548] System Configuration

[1549] The system mainly includes the following elements:

[1550] 1. Data entry method

[1551] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[1552] 2. Pretreatment Methods

[1553] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[1554] 3. Comment and instruction input means

[1555] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[1556] 4. Generative AI Models

[1557] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[1558] 5. Display means

[1559] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[1560] 6. Database

[1561] The server has a database that centrally stores medical data, medical images, comments and instructions.

[1562] 7. History Management Methods

[1563] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[1564] 8. Emotion Engine

[1565] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[1566] Data Processing

[1567] 1. Data Entry

[1568] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading them. The device then sends this data to the server.

[1569] 2. Pretreatment

[1570] The server preprocesses the medical data and images it receives. For image data, it performs noise removal, resolution adjustment, and contrast enhancement, and for medical data, it standardizes units and converts formats.

[1571] 3. Enter comments and instructions

[1572] The user (doctor) enters comments and instructions about the patient's condition and treatment into the terminal, which then sends this information to the server, which then stores this information in a database.

[1573] 4. Data Analysis and Transformation

[1574] The server combines the preprocessed medical data, medical images, and comments and invokes the generative AI model, which analyzes the data and converts it into a format that is easy for patients to understand (e.g., a 3D image highlighting the lesion).

[1575] 5. Emotion analysis and display adjustment

[1576] The device analyzes the user's (patient's) facial expressions and voice in real time, and the emotion engine uses this information to recognize the user's emotional state. The server dynamically adjusts the data displayed based on the emotion engine's results. For example, if the patient is feeling anxious, the server displays simplified explanations using technical terms.

[1577] 6. Interactive Display

[1578] The server sends the generated visualization data to the terminal, which then provides it to the patient through an interactive interface, allowing the patient to rotate, zoom in and out to view the data.

[1579] Specific examples

[1580] For a shoulder fracture diagnosis:

[1581] Example prompt sentence:

[1582] "Please analyze the CT scan of the patient's shoulder and generate a 3D model that highlights the fracture. The generated model should be intuitive for the patient."

[1583] By inputting this prompt into the generative AI model, a 3D image that is easy for patients to understand is generated, highlighting the fractured shoulder area, allowing patients to intuitively understand their condition and facilitating communication with their doctors.

[1584] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1585] Step 1:

[1586] A user (doctor) uses a terminal to select a patient's medical data (e.g., diagnosis results, measurements) and medical images (e.g., CT scans, MRI images) and initiates uploading to the system. The input at this time is the medical data and medical images selected by the user, and the output is the state in which these data have been uploaded to the terminal.

[1587] Step 2:

[1588] The terminal transmits the medical data and medical images selected by the user to the server. The input is the medical data and medical images on the terminal, and the output is the data transmitted to the server. HTTPS is used as the transmission protocol.

[1589] Step 3:

[1590] The server receives medical data and medical images sent from the terminal. The input is the data and images sent to the server, and the output is the data that has been received inside the server. The server reads this data for preprocessing.

[1591] Step 4:

[1592] The server preprocesses the medical data and medical images it receives. First, it performs noise removal, resolution adjustment, and contrast enhancement on the image data. Next, it standardizes the units and converts the format of the medical data. The input at this time is the raw data received by the server, and the output is the preprocessed medical data and medical images.

[1593] Step 5:

[1594] The user (doctor) inputs comments and instructions about the patient's condition and treatment into the terminal and sends the data to the system. The input is the comments and instructions entered by the doctor, and the output is the comments and instructions sent from the terminal to the server.

[1595] Step 6:

[1596] The terminal sends the entered comments and instructions to the server. The input here is the comments and instructions entered on the terminal, and the output is the data sent to the server.

[1597] Step 7:

[1598] The server integrates the preprocessed medical data, medical images, and comments and stores them in a database, where the inputs are the preprocessed data, images, and doctor's comments and instructions, and the output is the integrated data stored in the database.

[1599] Step 8:

[1600] The server generates a prompt to invoke the generative AI model. For example, it generates a prompt such as, "Analyze the CT scan image of the patient's shoulder and generate a 3D model that highlights the fractured area." The input is the integrated data and comments, and the output is the generated prompt.

[1601] Step 9:

[1602] The server sends data and prompts to the generative AI model for analysis and conversion. The input is the data integrated with the generated prompt, and the output is the data and prompt sent to the generative AI model.

[1603] Step 10:

[1604] The generative AI model analyzes the data and converts it into a format that is easy for the patient to understand (e.g., a photograph highlighting the lesion or a 3D image). The input to the generative AI model is the data and prompts it receives, and the output is the converted visualization data.

[1605] Step 11:

[1606] The server receives the analysis results from the generative AI model and stores them in a database. The input at this time is the analysis results from the generative AI model, and the output is the visualization data stored in the database.

[1607] Step 12:

[1608] The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time. The input is the patient's facial expressions, voice, and operation patterns, and the output is analyzed emotional data.

[1609] Step 13:

[1610] The server dynamically adjusts the displayed data based on the results of the emotion engine. For example, if a patient is feeling anxious, it displays a simplified explanation using technical terms. The input is the emotion engine's results, and the output is the adjusted display data.

[1611] Step 14:

[1612] The server sends the generated visualization data to the terminal, where the input is the visualization data in the server and the output is the visualization data sent to the terminal.

[1613] Step 15:

[1614] The device provides the generated visualization data to the patient through an interactive interface, allowing the patient to rotate the data and zoom in and out to view the details. The input is the visualization data displayed on the device, and the output is the interface operated by the patient.

[1615] Step 16:

[1616] The user (patient) understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface. The input is the patient's feedback obtained through the interface, and the output is the doctor's additional explanation or response.

[1617] (Application example 2)

[1618] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1619] Autonomous vehicles require systems that can recognize the driver's emotions and state and provide appropriate responses based on that information. However, existing in-vehicle systems have not been able to analyze and recognize the driver's emotions in real time, and dynamically adjust the display to provide appropriate information accordingly. In particular, there is a need for systems that can provide information on relaxation and rest when the driver is feeling stressed or fatigued.

[1620] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1621] In this invention, the server includes a data input means, a preprocessing means, and an analysis and display adjustment means, which allows the server to analyze the driver's emotions in real time and dynamically adjust the display and provide appropriate information accordingly.

[1622] "Data input means" means a device or method for inputting patient medical data and medical images into the system.

[1623] "Preprocessing means" refers to a device or method that processes input medical data or medical images, such as noise removal, resolution adjustment, and contrast enhancement, to convert them into a format that is easy to analyze.

[1624] "Means for inputting doctor's comments and instructions" refers to a device or method that allows a doctor to input comments about a patient's condition and treatment instructions into the system.

[1625] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[1626] The "display means" is a display or interface for visually showing the data converted into the generated format to the user.

[1627] "Emotion recognition means" is a technology that analyzes a user's facial expressions, voice, operation patterns, etc. to identify the user's emotional state.

[1628] A "display adjustment means" is a device or method that dynamically changes the content or format of displayed data based on recognized emotions.

[1629] A "database" is an information system for centrally storing patient medical data, medical images, comments, instructions, etc.

[1630] MODE FOR CARRYING OUT THE INVENTION

[1631] This invention relates to a system for autonomous vehicles that recognizes the driver's emotions in real time and provides appropriate information. Specifically, it analyzes the driver's emotions using a camera and microphone and dynamically adjusts the display content to improve the driver's comfort and safety.

[1632] System Configuration

[1633] The system mainly includes the following elements:

[1634] 1. Data entry method:

[1635] The camera and microphone provide an interface for inputting the driver's facial expressions and voice.

[1636] 2. Pretreatment methods:

[1637] The server receives the input data and performs preprocessing such as image noise removal, resolution adjustment, and audio filtering.

[1638] 3. Emotion recognition means:

[1639] Emotion recognition software analyzes the pre-processed data and recognizes the driver's emotions.

[1640] 4. Generative AI Models:

[1641] The server uses the recognized emotion data to use a generative AI model to provide appropriate information to the driver.

[1642] 5. Display means:

[1643] An in-car display provides the generated data to the driver.

[1644] 6. Database:

[1645] The server has a database that stores emotion data and display content.

[1646] 7. Display adjustment means:

[1647] It includes a software module that dynamically changes the content and format of the data displayed based on the recognized emotion.

[1648] Hardware and Software

[1649] Hardware:

[1650] Camera: Using "Logitech C920" as an example

[1651] Microphone: Using "Blue Yeti" as an example

[1652] In-vehicle display: Using the "Tesla Model 3 dashboard display" as an example

[1653] Server: A server for processing and storing data.

[1654] software:

[1655] OpenCV: A library for processing video data from cameras

[1656] EmotionRecognizer: A library for facial expression recognition

[1657] DriverAssistantDisplay: A module for managing the display content

[1658] Data processing and calculation

[1659] The server preprocesses the data acquired from the camera and microphone and analyzes the driver's emotions using facial and voice recognition technology. The analyzed emotional data is then input into a generative AI model to generate appropriate information content. The generated information is then displayed on the in-vehicle display. The displayed content changes dynamically depending on the recognized emotion. For example, if the driver is feeling stressed, the system can provide information about the nearest rest area or relaxing music.

[1660] Specific examples

[1661] Example: A driver gets into a car and the camera starts capturing their facial expressions. For example, if the driver's facial expression is recognized as "stressed," the dashboard display will offer options such as "Find the nearest rest stop here" or "Play relaxing music." As the driver selects a relaxation measure, the display will dynamically update.

[1662] Example prompt sentence:

[1663] If the driver is stressed, create a list of what information to exclude and what to provide instead, and generate an example script that dynamically adjusts the display accordingly.

[1664] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1665] Step 1:

[1666] When a user gets into a car, the device activates the camera and microphone. The camera captures the driver's face, and the microphone collects audio. The input is the camera video and audio data, which are sent to the server. The server receives these data for preprocessing.

[1667] Step 2:

[1668] The server performs noise reduction and resolution adjustment on the received video data, and filtering on the audio data. It uses OpenCV for noise reduction and resolution adjustment, and an audio processing library for audio filtering. The preprocessed data is output.

[1669] Step 3:

[1670] The server passes the preprocessed video data to EmotionRecognizer, which analyzes the driver's facial expressions. The analysis results output the driver's emotions (e.g., relaxed, stressed, urgent).

[1671] Step 4:

[1672] The server analyzes the pre-processed audio data and performs additional emotion analysis based on volume and tone. The results of the speech recognition and facial expression recognition are integrated to provide a highly accurate emotion assessment. Analyzed emotion data is generated.

[1673] Step 5:

[1674] The server inputs the emotional data into the generative AI model and generates appropriate information content. The data generated using the prompt sentence is output from the generative AI model. Specific information is generated, such as "display route information if the user is relaxed" or "display rest stop information if the user is stressed."

[1675] Step 6:

[1676] The server sends the generated information to the in-vehicle display, where it is displayed by the terminal. The display content is dynamically adjusted in real time according to the recognized emotion. For example, if the driver is judged to be in a "stressed state" in step 5, information such as "The nearest rest stop is here" will be displayed.

[1677] Step 7:

[1678] The device monitors user actions, allowing the driver to make selections and operations based on the displayed information. The selections are then sent back to the server, and in some cases additional information is generated using a generative AI model, and the displayed content is updated.

[1679] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1680] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1681] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1682] [Fourth embodiment]

[1683] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1684] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1685] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1686] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1687] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1688] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1689] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1690] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1691] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1692] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1693] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1694] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1695] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1696] A specific embodiment of the present invention will be described below. The purpose of this system is to convert a patient's medical data and medical images into a format that is easy for the patient to understand, thereby facilitating communication between doctors and patients.

[1697] System Configuration

[1698] The system mainly includes the following elements:

[1699] 1. Data entry method

[1700] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[1701] 2. Pretreatment Methods

[1702] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[1703] 3. Comment and instruction input means

[1704] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[1705] 4. Generative AI Models

[1706] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for patients to understand.

[1707] 5. Display means

[1708] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[1709] 6. Database

[1710] The server has a database that centrally stores medical data, medical images, comments and instructions.

[1711] 7. History Management Methods

[1712] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[1713] Natural language description of the program

[1714] Data Entry and Preprocessing

[1715] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[1716] 2. The device sends the selected file to the server.

[1717] 3. The server receives the data and images and performs preprocessing, such as noise reduction, resolution adjustment, and contrast enhancement. It also standardizes the format and units of medical data.

[1718] Entering comments and instructions

[1719] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[1720] 2. The device sends comments and instructions to the server.

[1721] 3. The server stores the comments and instructions in a database.

[1722] Data analysis and transformation

[1723] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[1724] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[1725] 3. The server receives the generated data and stores it in a database.

[1726] Providing results

[1727] 1. The server sends the generated visualization data to the device.

[1728] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[1729] 3. The user (patient) checks the visualized data and understands their own health condition.

[1730] Specific examples

[1731] For a shoulder fracture diagnosis:

[1732] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[1733] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[1734] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[1735] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[1736] 5. The server sends the generated 3D model to the device.

[1737] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[1738] In this way, this system converts patients' medical data and medical images into easy-to-understand information, supporting communication between doctors and patients and promoting patient health management.

[1739] The processing flow will be explained below.

[1740] Step 1:

[1741] The user (doctor) selects the patient's medical data and medical images on the terminal. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[1742] Step 2:

[1743] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[1744] Step 3:

[1745] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[1746] Step 4:

[1747] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[1748] Step 5:

[1749] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[1750] Step 6:

[1751] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[1752] Step 7:

[1753] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[1754] Step 8:

[1755] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[1756] Step 9:

[1757] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[1758] Step 10:

[1759] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[1760] Step 11:

[1761] The server stores the generated visualization data in a database, where the update history is also managed.

[1762] Step 12:

[1763] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[1764] Step 13:

[1765] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[1766] Step 14:

[1767] Users (patients) can view the visualized data and understand the details of their health condition and treatment. Through the interface, patients can ask questions to their doctor or request additional explanations.

[1768] The above is the specific flow of program processing according to the present invention.

[1769] Example 1

[1770] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1771] In the medical field, it is important to convert medical data and medical images into a format that is easy for patients to understand, but this requires a lot of manual work and specialized knowledge. Furthermore, poor communication between doctors and patients can negatively impact the quality of medical care. The present invention aims to solve these problems by automatically converting medical data and medical images into an easily understandable format, thereby improving communication between doctors and patients.

[1772] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1773] In this invention, the server includes: a means for inputting patient medical data, a means for inputting patient medical images, a means for preprocessing the input medical data and medical images, a means for inputting doctor comments and instructions based on the preprocessed data and images, a means for using a generative artificial intelligence model that integrates and analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for patients to understand (e.g., photos, 3D images, videos), a means for visualizing and interactively displaying the generated data, and a means for having a database that centrally stores patient medical data, medical images, comments, and instructions and for managing their history. This automatically converts medical data and medical images into easy-to-understand formats, facilitating communication between doctors and patients and improving the quality of medical care.

[1774] "Patient Medical Data" means quantitative or qualitative medical information about a patient, such as a patient's diagnostic results, measurements, medical history, and prescription information.

[1775] "Patient medical images" refers to image data that visually captures the internal state of a patient's body, such as CT scans, MRIs, X-rays, and ultrasound images.

[1776] "Preprocessing means" refers to a process for improving the quality of input medical data and medical images by performing processes such as noise removal, resolution adjustment, contrast enhancement, and format matching.

[1777] "Doctor's comments and instructions" refers to text information entered by a doctor regarding the patient's condition and treatment, and includes a diagnosis, treatment plan, and precautions.

[1778] A "generative artificial intelligence model" refers to an algorithm or program that uses machine learning and deep learning technologies to analyze input medical data and medical images and convert them into a format that is easy for patients to understand.

[1779] "Means for interactive display" refers to an interface that displays the generated visualization data in a form that allows the user to manipulate it and enables detailed confirmation and observation.

[1780] "Database" means an information system for centrally storing, efficiently managing, and making accessible medical data, medical images, comments, and instructions.

[1781] "Means for managing history" refers to the process of recording and managing the history of past processing and operations in order to track, analyze, and reuse stored medical data and medical images, as well as comments and instructions.

[1782] This invention is a system that automatically converts a patient's medical data and medical images into a format that is easy for the patient to understand. This aims to improve communication between doctors and patients and enhance the quality of medical care. The system of this invention is mainly composed of three main elements: a server, a terminal, and a user.

[1783] System Configuration

[1784] 1. Data entry method

[1785] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading. The device sends the selected files to the server. For example, the device transfers the files using the HTTP or HTTPS protocol.

[1786] 2. Pretreatment Methods

[1787] The server performs preprocessing such as noise removal, resolution adjustment, and contrast enhancement on the medical data and medical images it receives. Specifically, it uses the OpenCV library to perform image filtering, and the Pandas library to align the format of the medical data and standardize units.

[1788] 3. Comment and instruction input means

[1789] The user (doctor) enters comments about the patient's condition and treatment instructions on the device. The device sends the comments and instructions to the server. For example, the data is sent using JSON format, and the server converts it into a format that can be properly analyzed.

[1790] 4. Generative AI Models

[1791] The server combines the preprocessed data and images with the doctor's comments and converts them into a format that is easy for the patient to understand (e.g., photos, 3D images, videos) using a generative AI model, specifically using Scikit-learn and TensorFlow libraries.

[1792] 5. Display means

[1793] The server sends the generated visualization data to the terminal, which provides an interactive display where the user (patient) can view the transformed data. For example, D3.js or Three.js can be used to interactively display the data.

[1794] 6. Database

[1795] The server will have a database to centrally store patient medical data, medical images, comments, and instructions, and manage the history. MongoDB and MySQL will be used to store the data, and past data can be searched using SQL queries.

[1796] Specific examples

[1797] For a shoulder fracture diagnosis:

[1798] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[1799] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[1800] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of this shoulder."

[1801] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[1802] 5. The server sends the generated 3D model to the device.

[1803] 6. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[1804] Prompt Sentence Examples

[1805] The following is an example of a specific prompt:

[1806] 1. "Enter a CT scan image and generate a 3D model highlighting the fracture."

[1807] 2. "Analyze the cardiac MRI images and mark any abnormalities."

[1808] In this way, the system of the present invention converts patient medical data and medical images into easy-to-understand information, thereby supporting communication between doctors and patients and promoting health management for patients.

[1809] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1810] Step 1: Data entry

[1811] 1. The user (doctor) selects the patient's medical data and medical images on the device and starts uploading.

[1812] Input: Medical data files (e.g., CSV, Excel), medical image files (e.g., CT scans, MRI images)

[1813] Specific operations: The user operates the interface on the device, selects the required file, and clicks the "Upload" button.

[1814] Output: The selected files are sent to the server.

[1815] 2. The device sends the selected file to the server.

[1816] Input: A file selected by the user

[1817] Data processing: Transfer files securely using HTTP or HTTPS protocols.

[1818] Specific operation: The device temporarily saves the file and sends it to the specified API endpoint on the server.

[1819] Output: The server receives the file.

[1820] Step 2: Preprocessing

[1821] 1. The server receives the data and images.

[1822] Input: File sent from terminal

[1823] Specific operation: The server saves the file in the specified directory and records the file's metadata (source, date and time of sending, etc.).

[1824] Output: Saved medical data and medical image files

[1825] 2. The server performs noise reduction, resolution adjustment, and contrast enhancement on the medical images.

[1826] Input: Saved medical image files

[1827] Data processing: Image noise removal, resolution adjustment, and contrast enhancement are performed using the OpenCV library.

[1828] Specific operation: Executes OpenCV functions (e.g., cv2.fastNlMeansDenoising(), cv2.resize(), cv2.equalizeHist()) sequentially.

[1829] Output: Preprocessed medical image files

[1830] 3. The server harmonizes the format and standardizes the units of medical data.

[1831] Input: Stored medical data file

[1832] Data processing: Use the Pandas library to clean the data, align the format, and standardize units.

[1833] Specific operation: Executes Pandas functions (e.g., pd.read_csv(), df.fillna(), df.apply()) sequentially.

[1834] Output: Preprocessed medical data file

[1835] Step 3: Enter comments and instructions

[1836] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[1837] Input: Comments about the patient's condition and treatment instructions

[1838] Specific operation: The user enters a comment or instruction into the interface on the device and clicks the "Send" button.

[1839] Output: Comments and instructions entered

[1840] 2. The device sends comments and instructions to the server.

[1841] Input: Comments or instructions entered

[1842] Data processing: Convert to JSON format and send to the server.

[1843] Specific operation: The terminal converts the input data into JSON format and calls an API to send it to the server.

[1844] Output: The server receives comments and instructions.

[1845] Step 4: Analyze and transform the data

[1846] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[1847] Input: Preprocessed medical data files, medical image files, comments

[1848] Data processing: Integrating data and images into a format that can be input into a generative AI model.

[1849] Specific operation: Convert the data into the input format for the generative AI model and call the AI ​​model.

[1850] Output: Input data for the AI ​​model

[1851] 2. A generative AI model analyzes the data and translates it into a format that is easy for patients to understand.

[1852] Input: Input data for the AI ​​model

[1853] Data Computation: Analysis and transformation using machine learning algorithms and deep learning models.

[1854] Specific operation: Calls the model's prediction function and generates a 3D model and video that highlights the lesion area.

[1855] Output: Generated visualization data (e.g. 3D models, enhanced images)

[1856] 3. The server receives the generated data and stores it in a database.

[1857] Input: Generated visualization data

[1858] Data processing: Save the data in the appropriate format for the database.

[1859] Specific operation: Insert the generated data into a MongoDB or MySQL database.

[1860] Output: Visualization data stored in a database

[1861] Step 5: View and verify

[1862] 1. The server sends the generated visualization data to the device.

[1863] Input: Visualization data stored in a database

[1864] Data processing: Converting data into the format required for sending to the terminal.

[1865] Specific operation: Converts data into JSON or binary format and sends it to the terminal.

[1866] Output: Visualization data sent to the terminal

[1867] 2. The terminal provides an interactive display where the user (patient) can review the converted data.

[1868] Input: Visualization data received from the server

[1869] Specific behavior: Display data interactively using D3.js and Three.js.

[1870] Output: User-navigable visualization of data

[1871] 3. The user (patient) checks the visualized data and understands their own health condition.

[1872] Input: Interactive visualization data

[1873] Specific operation: The user drags the data, zooms, etc.

[1874] Output: The user visually understands their health status.

[1875] Step 6: History Management

[1876] 1. The server manages history based on centrally stored data.

[1877] Input: Stored medical data, medical images, comments and instructions

[1878] Data Processing: Organize historical data and make it quickly accessible when needed.

[1879] What you will do: Search and analyze historical data using SQL queries and log search tools.

[1880] Output: Archived data

[1881] 2. The server stores the operation history to facilitate re-analysis and conversion.

[1882] Input: Log data for each operation

[1883] Data processing: Analyze and save log data.

[1884] Specific behavior: Recording operations in log files and dedicated history databases.

[1885] Output: Saved operation history

[1886] (Application example 1)

[1887] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1888] Conventional medical data and medical image processing systems have the problem that it is difficult for patients to understand their own medical condition. Furthermore, the authentication data in facial recognition systems is difficult to understand visually, making it difficult to intuitively confirm authentication results. Therefore, there is a need to facilitate communication between doctors and patients, and between security managers and authentication targets, and to streamline authentication work.

[1889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1890] In this invention, the server includes means for inputting patient medical data, means for inputting patient medical images, means for preprocessing the input medical data and medical images, means for inputting doctor comments and instructions based on the preprocessed data and images, means for using a generative artificial intelligence model that analyzes the input comments and instructions, medical data, and medical images and converts them into a format that is easy for the patient to understand, means for displaying the data converted into the generated format, means for inputting facial authentication data, means for preprocessing the input facial authentication data, means for using a generative artificial intelligence model that analyzes and visualizes the preprocessed facial authentication data, and means for interactively displaying the generated facial authentication data. This allows patients to intuitively understand their own medical data and medical images, and also enables the person to be authenticated to visually confirm their own authentication data.

[1891] "Medical data" refers to information such as a patient's health status, diagnosis results, and treatment information.

[1892] "Medical images" refers to image data obtained by medical equipment such as CT scans and MRIs.

[1893] "Preprocessing means" refers to techniques for performing processes such as noise removal and resolution adjustment on medical data, medical images, and facial recognition data.

[1894] "Means for inputting comments and instructions" refers to an interface that allows doctors and administrators to input specific diagnosis results and instructions to the system.

[1895] A "generative artificial intelligence model" refers to artificial intelligence that analyzes input data and converts it into a format that is easy for patients and those being certified to understand.

[1896] "Means for displaying data converted into a generated format" refers to an interface for visually displaying the parsed and converted data.

[1897] "Facial recognition data" refers to image data and feature data relating to an individual's face.

[1898] "Interactive display means" refers to a display method that is easy for users to operate and allows them to intuitively check the data.

[1899] A specific embodiment of the present invention will be described below. The purpose of this system is to convert patient medical data and medical images into a format that is easy for patients to understand, and to provide visually intuitive authentication results in a face recognition system.

[1900] System Configuration

[1901] The system mainly includes the following elements:

[1902] 1. Data entry method

[1903] The terminal provides an interface for inputting the patient's medical data, medical images, and facial recognition data.

[1904] 2. Pretreatment Methods

[1905] The server receives the input medical data, medical images, and facial recognition data, and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[1906] 3. Comment and instruction input means

[1907] The terminal provides an interface that accepts comments and instructions entered by a doctor or security administrator.

[1908] 4. Generative AI Models

[1909] The server combines the pre-processed data, medical images, facial recognition data, and comments and instructions, and uses a generative AI model to convert them into a format (photos, 3D images, videos) that is easy for the patient or person being authenticated to understand.

[1910] 5. Display means

[1911] The terminal provides the generated visualization data and provides an interface that allows for interactive display.

[1912] 6. Database

[1913] The server has a database that centrally stores medical data, medical images, facial recognition data, comments and instructions.

[1914] 7. History Management Methods

[1915] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[1916] Hardware and software used

[1917] Smartphones (e.g. iPhone, Android devices)

[1918] Head-mounted displays (HMDs) (e.g., Oculus Quest 2, Microsoft HoloLens 2)

[1919] Server (e.g. AWS EC2 instance, Google Cloud Platform)

[1920] Databases (e.g. Amazon RDS, Google Cloud Firestore)

[1921] Generative AI models (e.g., models using TensorFlow or PyTorch)

[1922] Processing flow

[1923] Data Entry and Preprocessing

[1924] 1. The user (doctor or person to be authenticated) enters medical data, medical images, and facial recognition data using a smartphone or HMD and uploads them to the server.

[1925] 2. The server receives the data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[1926] Entering comments and instructions

[1927] 1. The user (doctor or administrator) enters analysis comments and instructions on the device and sends them to the server.

[1928] 2. The server stores the comments and instructions in a database.

[1929] Data analysis and transformation

[1930] 1. The server integrates the preprocessed data and comments and invokes the generative AI model.

[1931] 2. A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[1932] 3. The server receives the generated data and stores it in a database.

[1933] Providing results

[1934] 1. The server sends the generated visualization data to the device.

[1935] 2. The terminal provides an interactive display, allowing the user (patient or authentication subject) to review the content.

[1936] Specific examples

[1937] For a shoulder fracture diagnosis:

[1938] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system.

[1939] 2. The server preprocesses the CT scan images (adjusting resolution and removing noise).

[1940] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured shoulder."

[1941] 4. The generative AI model analyzes the data and generates a 3D model that highlights the fracture.

[1942] 5. The server sends the generated 3D model to the device.

[1943] 6. The user (patient) checks the 3D model on a smartphone or HMD to visually understand the condition of the shoulder fracture.

[1944] Prompt Sentence Examples

[1945] Target image: CT scan image of shoulder

[1946] Comment: Please emphasize the broken shoulder.

[1947] The following applies to the specific example of face recognition:

[1948] 1. The user (person to be authenticated) takes a photo of their face with their smartphone and uploads it to the authentication system.

[1949] 2. The server preprocesses the image and performs feature extraction.

[1950] 3. The administrator enters a comment saying, "Emphasis on the position of the eyes and the width of the mouth."

[1951] 4. The generative AI model analyzes the data and generates a highlighted 3D face model containing the authentication results.

[1952] 5. The server sends the generated 3D face model and authentication results to the device.

[1953] 6. The user checks the 3D model on a smartphone or HMD and intuitively understands the authentication results and highlighting.

[1954] Prompt Sentence Examples

[1955] Target image: Facial recognition image

[1956] Comment: Emphasize the position of the eyes and the width of the mouth

[1957] In this way, the embodiments for carrying out the present invention have been specifically described.

[1958] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1959] Step 1:

[1960] Users (doctors or individuals to be authenticated) enter medical data, medical images, and facial recognition data using a smartphone or HMD and upload them to the server.

[1961] Input: Patient medical data, medical images, or facial recognition data

[1962] Output: Raw data sent to the server

[1963] Specific action: The user captures data using the device's camera or scanning function and uploads it.

[1964] Step 2:

[1965] The server receives the uploaded data and performs pre-processing such as format matching, noise removal, and resolution adjustment.

[1966] Input: Uploaded data (medical data, medical images, facial recognition data)

[1967] Output: Preprocessed data

[1968] Specific operation: The server aligns the format of the medical data, removes noise, and adjusts the image resolution to generate preprocessed data.

[1969] Step 3:

[1970] The user (doctor or administrator) enters analysis comments and instructions on the terminal and sends them to the server.

[1971] Input: Comments and instructions

[1972] Output: Comments and instructions sent to the server

[1973] Specific operation: The user uses the input interface of the terminal to input diagnostic results and analysis instructions, and sends them to the server.

[1974] Step 4:

[1975] The server integrates the preprocessed data and comments and invokes the generative AI model.

[1976] Input: Preprocessed data and comments

[1977] Output: Input data to a generative AI model

[1978] Specific operation: The server integrates the preprocessed data with input instructions and comments to pass them to the generative AI model.

[1979] Step 5:

[1980] A generative AI model analyzes the data and generates visualizations (photos, 3D images, videos).

[1981] Input: Integrated data and comments

[1982] Output: Visualization data (highlighted photos, 3D images, videos)

[1983] What it does: The generative AI model analyzes the data and generates visualizations based on the prompt.

[1984] Step 6:

[1985] The server receives the generated data and stores it in a database.

[1986] Input: Generated visualization data

[1987] Output: Visualization data stored in a database

[1988] Specific operation: The server stores the generated data in a database and provides a history management function.

[1989] Step 7:

[1990] The server transmits the generated visualization data to the terminal.

[1991] Input: Visualization data stored in a database

[1992] Output: Visualization data sent to the terminal

[1993] Specific operation: The server retrieves visualization data from the database and sends it to the terminal.

[1994] Step 8:

[1995] The terminal provides an interactive display, and the user (patient or person being authenticated) can review the content.

[1996] Input: Visualization data sent to the device

[1997] Output: Visualization data reviewed by the user

[1998] Specific operation: The terminal displays the visualized data interactively, and the user can perform operations to check the data.

[1999] Through each of these steps, the system of the present invention can effectively process medical data and facial recognition data and present it to the user in an intuitive and easy-to-understand format.

[2000] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2001] A specific embodiment of the present invention is described below. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[2002] System Configuration

[2003] The system mainly includes the following elements:

[2004] 1. Data entry method

[2005] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[2006] 2. Pretreatment Methods

[2007] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[2008] 3. Comment and instruction input means

[2009] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[2010] 4. Generative AI Models

[2011] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[2012] 5. Display means

[2013] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[2014] 6. Database

[2015] The server has a database that centrally stores medical data, medical images, comments and instructions.

[2016] 7. History Management Methods

[2017] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[2018] 8. Emotion Engine

[2019] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[2020] Natural language description of the program

[2021] Data Entry and Preprocessing

[2022] 1. The user (doctor) selects the patient's medical data and medical images on the device and begins uploading.

[2023] 2. The device sends the selected file to the server.

[2024] 3. The server receives the data and images and performs preprocessing, including image noise reduction, resolution adjustment, and contrast enhancement. For medical data, processing such as unit unification and format conversion is also performed.

[2025] Entering comments and instructions

[2026] 1. The user (doctor) enters comments about the patient's condition and treatment instructions on the terminal.

[2027] 2. The device sends comments and instructions to the server.

[2028] 3. The server stores the comments and instructions in a database.

[2029] Data analysis and transformation

[2030] 1. The server integrates preprocessed medical data, medical images, and comments and invokes the generative AI model.

[2031] 2. The generative AI model analyzes the data and converts it into a format that is easy for patients to understand, such as photos highlighting the lesion, 3D images, or videos.

[2032] 3. The server receives the generated data and stores it in a database.

[2033] Emotion analysis and display adjustment

[2034] 1. The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time.

[2035] 2. The emotion engine recognizes the user's emotional state based on the analyzed data. For example, if the user is feeling anxious, it will display an option to immediately contact a call center staff member.

[2036] 3. The server dynamically adjusts the displayed data based on the results of the emotion engine, for example by reducing technical terms or emphasizing visuals to make it easier for users to understand.

[2037] Providing results

[2038] 1. The server sends the generated visualization data to the device.

[2039] 2. The terminal provides an interactive display where the user (patient) can view the converted data. The user can rotate the 3D model and zoom in and out to view the details.

[2040] 3. The user (patient) reviews the visualized data and understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface.

[2041] Specific examples

[2042] For a shoulder fracture diagnosis:

[2043] 1. The user (doctor) uploads the CT scan image of the patient's shoulder and the diagnosis results to the system from their terminal.

[2044] 2. The server performs preprocessing on the CT scan image by adjusting the resolution and removing noise.

[2045] 3. The user (doctor) enters a comment saying, "Please emphasize the fractured part of the shoulder."

[2046] 4. The server passes the data, images, and comments to a generative AI model, which generates a 3D model highlighting the fracture.

[2047] 5. The server sends the generated 3D model to the device.

[2048] 6. The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state.

[2049] 7. The server adapts the display to the patient's emotional state and provides additional information and explanations to reduce anxiety.

[2050] 8. The user (patient) checks the 3D model on the device and visually understands the condition of their shoulder fracture.

[2051] In this way, this system converts patients' medical data and medical images into easy-to-understand formats, and also takes into consideration the user's emotional state, thereby supporting communication between doctors and patients and promoting patient health management.

[2052] The processing flow will be explained below.

[2053] Step 1:

[2054] The user (doctor) selects the patient's medical data and medical images on the terminal and starts uploading. The doctor uses a dedicated file selection UI to upload CT scan images and diagnosis result files to the system.

[2055] Step 2:

[2056] The device will send the selected medical data and medical images to the server. The files will be uploaded to the server in batches, and the upload progress will be displayed in real time.

[2057] Step 3:

[2058] The server receives the medical data and medical images, checks the format and integrity of the received data, and generates an error message if there is an inconsistency.

[2059] Step 4:

[2060] The server performs preprocessing of the medical images received. Preprocessing includes image noise removal, resolution adjustment, contrast enhancement, etc. For medical data, processing such as unit unification and format conversion is performed.

[2061] Step 5:

[2062] The user (doctor) inputs comments and instructions on the terminal. The comment input interface has fields for describing the patient's condition and specific treatment instructions in detail.

[2063] Step 6:

[2064] The device sends the entered comments and instructions to the server, where the comments and instructions are sent together with the medical data and images.

[2065] Step 7:

[2066] The server stores the comments and instructions entered by the doctor in a database and, if necessary, formats the contents of the comments and instructions into a format that is easy to analyze.

[2067] Step 8:

[2068] The server combines the pre-processed medical data, medical images, and comments, and the combined data is passed to a generative AI model.

[2069] Step 9:

[2070] A generative AI model analyzes the combined data, identifying specific lesions in medical images and converting them into a patient-friendly format (e.g., a highlighted 3D image).

[2071] Step 10:

[2072] The generative AI model generates visualized data (e.g., 3D models or videos) based on the analysis results, and the generated data is sent back to the server.

[2073] Step 11:

[2074] The server stores the generated visualization data in a database, where the update history is also managed.

[2075] Step 12:

[2076] The server sends the generated visualization data to the user's (patient's) device, where the data is formatted for interactive viewing.

[2077] Step 13:

[2078] The terminal displays the visualized data in an interactive interface, allowing the user (patient) to freely rotate the 3D model and zoom in and out to view the details.

[2079] Step 14:

[2080] The device analyzes the user's (patient's) facial expressions and voice, and the emotion engine recognizes the patient's emotional state. For example, data is collected in real time using a camera or microphone, and emotion analysis is performed.

[2081] Step 15:

[2082] The server receives the analysis results of the emotion engine and dynamically adjusts the display content according to the patient's emotional state. For example, if the patient is highly anxious, it adds detailed explanations or displays options for asking questions to the doctor.

[2083] Step 16:

[2084] The user (patient) can view the tailored visualization data to understand the details of their health condition and treatment. Through the interface, the patient can ask questions to the doctor or request additional explanations.

[2085] In this way, the present invention facilitates communication between doctors and patients and promotes patient health management by visualizing medical data and images in an easy-to-understand manner and dynamically adjusting them according to the patient's emotional state.

[2086] Example 2

[2087] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2088] It is extremely difficult for patients to understand their own medical data and medical images. Poor communication between doctors and patients can lead to inadequate understanding of treatment and diagnosis. It is also important to appropriately adjust the content of explanations based on the patient's emotional state, but this has been difficult with conventional systems.

[2089] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2090] In this invention, the server includes means for integrating preprocessed data, images, and comments, calling a generative artificial intelligence model, and performing analysis; means for storing the generated data in a database; and means for displaying the generated data and receiving patient responses and questions on a terminal with an interactive display. This allows patients to intuitively understand their own medical data and medical images, enabling effective communication between doctors and patients. Furthermore, the display content can be dynamically adjusted according to the patient's emotional state, deepening the patient's understanding and reducing anxiety.

[2091] "Patient medical data" refers to information such as the patient's health condition, medical history, diagnosis, and treatment progress.

[2092] "Medical images" are image data, such as CT scans, MRIs, and X-rays, that visually show the internal structure and condition of a patient's body.

[2093] "Input means" refers to an interface or device that allows a user to provide medical data or medical images to the system.

[2094] A "server" is a central computer system that receives and processes input data.

[2095] "Preprocessing" refers to processes such as noise removal, resolution adjustment, and format conversion to improve the quality of input data.

[2096] "Doctor's comments and instructions" are written input of the doctor's professional opinions and instructions regarding the patient's condition and treatment plan.

[2097] A "database" is a data storage system for centrally managing and storing information about the system, such as medical data, medical images, comments, and instructions.

[2098] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[2099] An "interactive display" is an interface that allows users to manipulate and view the generated data.

[2100] The "means for accepting patient responses and questions" is an interface that allows patients to ask questions or give feedback on the displayed data.

[2101] A specific embodiment of the present invention will now be described. This system converts a patient's medical data and medical images into a format that is easy for the patient to understand, and furthermore, dynamically adjusts the displayed data by recognizing the user's emotions, thereby facilitating communication between doctors and patients.

[2102] System Configuration

[2103] The system mainly includes the following elements:

[2104] 1. Data entry method

[2105] The device provides an interface for inputting patient medical data (e.g., diagnostic results and measurements) and medical images (e.g., CT scans and MRI images).

[2106] 2. Pretreatment Methods

[2107] The server receives the input medical data and images and performs pre-processing such as format matching, noise removal, resolution adjustment, and contrast enhancement.

[2108] 3. Comment and instruction input means

[2109] The terminal provides an interface that accepts comments and instructions entered by a doctor regarding the patient's condition and treatment.

[2110] 4. Generative AI Models

[2111] The server combines the pre-processed data, medical images, and doctor comments and instructions, and uses a generative AI model to convert them into a format that is easy for the patient to understand (photos, 3D images, videos, etc.).

[2112] 5. Display means

[2113] The terminal provides the generated visualization data to the patient and provides an interface that allows for interactive display.

[2114] 6. Database

[2115] The server has a database that centrally stores medical data, medical images, comments and instructions.

[2116] 7. History Management Methods

[2117] It provides a history management function that allows the server to reanalyze and convert data based on the stored data.

[2118] 8. Emotion Engine

[2119] The server or terminal is equipped with an emotion engine that analyzes the user's facial expressions, voice, operation patterns, etc. and recognizes emotions.

[2120] Data Processing

[2121] 1. Data Entry

[2122] The user (doctor) selects the patient's medical data and medical images on the device and starts uploading them. The device then sends this data to the server.

[2123] 2. Pretreatment

[2124] The server preprocesses the medical data and images it receives. For image data, it performs noise removal, resolution adjustment, and contrast enhancement, and for medical data, it standardizes units and converts formats.

[2125] 3. Enter comments and instructions

[2126] The user (doctor) enters comments and instructions about the patient's condition and treatment into the terminal, which then sends this information to the server, which then stores this information in a database.

[2127] 4. Data Analysis and Transformation

[2128] The server combines the preprocessed medical data, medical images, and comments and invokes the generative AI model, which analyzes the data and converts it into a format that is easy for patients to understand (e.g., a 3D image highlighting the lesion).

[2129] 5. Emotion analysis and display adjustment

[2130] The device analyzes the user's (patient's) facial expressions and voice in real time, and the emotion engine uses this information to recognize the user's emotional state. The server dynamically adjusts the data displayed based on the emotion engine's results. For example, if the patient is feeling anxious, the server displays simplified explanations using technical terms.

[2131] 6. Interactive Display

[2132] The server sends the generated visualization data to the terminal, which then provides it to the patient through an interactive interface, allowing the patient to rotate, zoom in and out to view the data.

[2133] Specific examples

[2134] For a shoulder fracture diagnosis:

[2135] Example prompt sentence:

[2136] "Please analyze the CT scan of the patient's shoulder and generate a 3D model that highlights the fracture. The generated model should be intuitive for the patient."

[2137] By inputting this prompt into the generative AI model, a 3D image that is easy for patients to understand is generated, highlighting the fractured shoulder area, allowing patients to intuitively understand their condition and facilitating communication with their doctors.

[2138] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2139] Step 1:

[2140] A user (doctor) uses a terminal to select a patient's medical data (e.g., diagnosis results, measurements) and medical images (e.g., CT scans, MRI images) and initiates uploading to the system. The input at this time is the medical data and medical images selected by the user, and the output is the state in which these data have been uploaded to the terminal.

[2141] Step 2:

[2142] The terminal transmits the medical data and medical images selected by the user to the server. The input is the medical data and medical images on the terminal, and the output is the data transmitted to the server. HTTPS is used as the transmission protocol.

[2143] Step 3:

[2144] The server receives medical data and medical images sent from the terminal. The input is the data and images sent to the server, and the output is the data that has been received inside the server. The server reads this data for preprocessing.

[2145] Step 4:

[2146] The server preprocesses the medical data and medical images it receives. First, it performs noise removal, resolution adjustment, and contrast enhancement on the image data. Next, it standardizes the units and converts the format of the medical data. The input at this time is the raw data received by the server, and the output is the preprocessed medical data and medical images.

[2147] Step 5:

[2148] The user (doctor) inputs comments and instructions about the patient's condition and treatment into the terminal and sends the data to the system. The input is the comments and instructions entered by the doctor, and the output is the comments and instructions sent from the terminal to the server.

[2149] Step 6:

[2150] The terminal sends the entered comments and instructions to the server. The input here is the comments and instructions entered on the terminal, and the output is the data sent to the server.

[2151] Step 7:

[2152] The server integrates the preprocessed medical data, medical images, and comments and stores them in a database, where the inputs are the preprocessed data, images, and doctor's comments and instructions, and the output is the integrated data stored in the database.

[2153] Step 8:

[2154] The server generates a prompt to invoke the generative AI model. For example, it generates a prompt such as, "Analyze the CT scan image of the patient's shoulder and generate a 3D model that highlights the fractured area." The input is the integrated data and comments, and the output is the generated prompt.

[2155] Step 9:

[2156] The server sends data and prompts to the generative AI model for analysis and conversion. The input is the data integrated with the generated prompt, and the output is the data and prompt sent to the generative AI model.

[2157] Step 10:

[2158] The generative AI model analyzes the data and converts it into a format that is easy for the patient to understand (e.g., a photograph highlighting the lesion or a 3D image). The input to the generative AI model is the data and prompts it receives, and the output is the converted visualization data.

[2159] Step 11:

[2160] The server receives the analysis results from the generative AI model and stores them in a database. The input at this time is the analysis results from the generative AI model, and the output is the visualization data stored in the database.

[2161] Step 12:

[2162] The device analyzes the user's (patient's) facial expressions, voice, and operation patterns in real time. The input is the patient's facial expressions, voice, and operation patterns, and the output is analyzed emotional data.

[2163] Step 13:

[2164] The server dynamically adjusts the displayed data based on the results of the emotion engine. For example, if a patient is feeling anxious, it displays a simplified explanation using technical terms. The input is the emotion engine's results, and the output is the adjusted display data.

[2165] Step 14:

[2166] The server sends the generated visualization data to the terminal, where the input is the visualization data in the server and the output is the visualization data sent to the terminal.

[2167] Step 15:

[2168] The device provides the generated visualization data to the patient through an interactive interface, allowing the patient to rotate the data and zoom in and out to view the details. The input is the visualization data displayed on the device, and the output is the interface operated by the patient.

[2169] Step 16:

[2170] The user (patient) understands the details of their health condition and treatment. The patient can ask questions or seek additional explanations from the doctor through the interface. The input is the patient's feedback obtained through the interface, and the output is the doctor's additional explanation or response.

[2171] (Application example 2)

[2172] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2173] Autonomous vehicles require systems that can recognize the driver's emotions and state and provide appropriate responses based on that information. However, existing in-vehicle systems have not been able to analyze and recognize the driver's emotions in real time, and dynamically adjust the display to provide appropriate information accordingly. In particular, there is a need for systems that can provide information on relaxation and rest when the driver is feeling stressed or fatigued.

[2174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2175] In this invention, the server includes a data input means, a preprocessing means, and an analysis and display adjustment means, which allows the server to analyze the driver's emotions in real time and dynamically adjust the display and provide appropriate information accordingly.

[2176] "Data input means" means a device or method for inputting patient medical data and medical images into the system.

[2177] "Preprocessing means" refers to a device or method that processes input medical data or medical images, such as noise removal, resolution adjustment, and contrast enhancement, to convert them into a format that is easy to analyze.

[2178] "Means for inputting doctor's comments and instructions" refers to a device or method that allows a doctor to input comments about a patient's condition and treatment instructions into the system.

[2179] A "generative artificial intelligence model" is a machine learning model that analyzes input data and converts it into a format that is easy for patients to understand.

[2180] The "display means" is a display or interface for visually showing the data converted into the generated format to the user.

[2181] "Emotion recognition means" is a technology that analyzes a user's facial expressions, voice, operation patterns, etc. to identify the user's emotional state.

[2182] A "display adjustment means" is a device or method that dynamically changes the content or format of displayed data based on recognized emotions.

[2183] A "database" is an information system for centrally storing patient medical data, medical images, comments, instructions, etc.

[2184] MODE FOR CARRYING OUT THE INVENTION

[2185] This invention relates to a system for autonomous vehicles that recognizes the driver's emotions in real time and provides appropriate information. Specifically, it analyzes the driver's emotions using a camera and microphone and dynamically adjusts the display content to improve the driver's comfort and safety.

[2186] System Configuration

[2187] The system mainly includes the following elements:

[2188] 1. Data entry method:

[2189] The camera and microphone provide an interface for inputting the driver's facial expressions and voice.

[2190] 2. Pretreatment methods:

[2191] The server receives the input data and performs preprocessing such as image noise removal, resolution adjustment, and audio filtering.

[2192] 3. Emotion recognition means:

[2193] Emotion recognition software analyzes the pre-processed data and recognizes the driver's emotions.

[2194] 4. Generative AI Models:

[2195] The server uses the recognized emotion data to use a generative AI model to provide appropriate information to the driver.

[2196] 5. Display means:

[2197] An in-car display provides the generated data to the driver.

[2198] 6. Database:

[2199] The server has a database that stores emotion data and display content.

[2200] 7. Display adjustment means:

[2201] It includes a software module that dynamically changes the content and format of the data displayed based on the recognized emotion.

[2202] Hardware and Software

[2203] Hardware:

[2204] Camera: Using "Logitech C920" as an example

[2205] Microphone: Using "Blue Yeti" as an example

[2206] In-vehicle display: Using the "Tesla Model 3 dashboard display" as an example

[2207] Server: A server for processing and storing data.

[2208] software:

[2209] OpenCV: A library for processing video data from cameras

[2210] EmotionRecognizer: A library for facial expression recognition

[2211] DriverAssistantDisplay: A module for managing the display content

[2212] Data processing and calculation

[2213] The server preprocesses the data acquired from the camera and microphone and analyzes the driver's emotions using facial and voice recognition technology. The analyzed emotional data is then input into a generative AI model to generate appropriate information content. The generated information is then displayed on the in-vehicle display. The displayed content changes dynamically depending on the recognized emotion. For example, if the driver is feeling stressed, the system can provide information about the nearest rest area or relaxing music.

[2214] Specific examples

[2215] Example: A driver gets into a car and the camera starts capturing their facial expressions. For example, if the driver's facial expression is recognized as "stressed," the dashboard display will offer options such as "Find the nearest rest stop here" or "Play relaxing music." As the driver selects a relaxation measure, the display will dynamically update.

[2216] Example prompt sentence:

[2217] If the driver is stressed, create a list of what information to exclude and what to provide instead, and generate an example script that dynamically adjusts the display accordingly.

[2218] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2219] Step 1:

[2220] When a user gets into a car, the device activates the camera and microphone. The camera captures the driver's face, and the microphone collects audio. The input is the camera video and audio data, which are sent to the server. The server receives these data for preprocessing.

[2221] Step 2:

[2222] The server performs noise reduction and resolution adjustment on the received video data, and filtering on the audio data. It uses OpenCV for noise reduction and resolution adjustment, and an audio processing library for audio filtering. The preprocessed data is output.

[2223] Step 3:

[2224] The server passes the preprocessed video data to EmotionRecognizer, which analyzes the driver's facial expressions. The analysis results output the driver's emotions (e.g., relaxed, stressed, urgent).

[2225] Step 4:

[2226] The server analyzes the pre-processed audio data and performs additional emotion analysis based on volume and tone. The results of the speech recognition and facial expression recognition are integrated to provide a highly accurate emotion assessment. Analyzed emotion data is generated.

[2227] Step 5:

[2228] The server inputs the emotional data into the generative AI model and generates appropriate information content. The data generated using the prompt sentence is output from the generative AI model. Specific information is generated, such as "display route information if the user is relaxed" or "display rest stop information if the user is stressed."

[2229] Step 6:

[2230] The server sends the generated information to the in-vehicle display, where it is displayed by the terminal. The display content is dynamically adjusted in real time according to the recognized emotion. For example, if the driver is judged to be in a "stressed state" in step 5, information such as "The nearest rest stop is here" will be displayed.

[2231] Step 7:

[2232] The device monitors user actions, allowing the driver to make selections and operations based on the displayed information. The selections are then sent back to the server, and in some cases additional information is generated using a generative AI model, and the displayed content is updated.

[2233] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2234] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2235] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2236] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2237] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2238] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2239] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2240] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2241] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2242] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2243] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2244] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed ...

Claims

1. a means for inputting patient medical data; a means for inputting a medical image of a patient; means for preprocessing input medical data and medical images; a means for inputting physician comments and instructions based on the preprocessed data and images; using a generative artificial intelligence model to analyze input comments and instructions, medical data and medical images and translate them into a format that is easy for patients to understand; means for displaying the data converted into the generated format; A system including:

2. 10. The system of claim 1, wherein the generative artificial intelligence model includes means for analyzing medical images and converting them into a format that highlights specific lesions.

3. 10. The system of claim 1, further comprising a database for centrally storing patient medical data and medical images, as well as doctor comments and instructions.

4. 2. The system according to claim 1, further comprising a history management means for re-analyzing or converting the data based on the stored data.

5. 2. The system according to claim 1, further comprising means for transmitting the data converted into the generated format to a terminal of the patient for display.

Images