System

Abstract

A system is provided.SOLUTION: A system that provides a means for receiving clinical diagnostic data of a patient, a means for storing the received clinical diagnostic data in a database, a means for analyzing the stored clinical diagnostic data, a means for generating an AI diagnosis result based on the analysis result, a means for integrating the AI diagnosis result and a doctor's diagnosis result and displaying the result on a user interface, and a means for making a final diagnosis based on the integrated diagnosis result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When doctors make diagnoses, there are often issues with the accuracy of their diagnoses, especially in areas or situations where there is a shortage of specialists. It is difficult for doctors to efficiently and accurately analyze multiple clinical data sets and make appropriate diagnoses. In addition, second opinions are necessary to prevent misdiagnoses and oversights, but achieving them takes time and money. Therefore, there is a need for a system that improves diagnostic accuracy and reduces the burden on doctors. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a means for receiving a patient's clinical diagnostic data and storing it in a database. The stored data is analyzed, and an AI generates a diagnostic result based on the analysis results. Furthermore, by providing a means for integrating the AI's diagnostic results with those of a doctor and displaying them on a user interface, the doctor can make a final diagnosis while referring to the AI's second opinion. This improves diagnostic accuracy and reduces the risk of misdiagnosis or oversight. In addition, by storing the final diagnostic results and feedback in a database, a means is provided for using them as learning data for the AI, thereby continuously improving diagnostic accuracy.

[0006] "Patient clinical diagnostic data" refers to medical information collected from a patient, such as blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data.

[0007] "Database" refers to an information management system that organizes stored data and allows it to be quickly accessed, searched, and analyzed as needed.

[0008] "Means for analyzing" refers to a system that executes a program or algorithm to analyze stored clinical diagnostic data and detect specific diseases or abnormalities.

[0009] "AI" refers to programs and algorithms that generate diagnostic results based on artificial intelligence technology.

[0010] "Means for generating a diagnostic result" refers to the process by which the AI ​​outputs a diagnostic result for a particular disease or condition based on the analyzed data.

[0011] "Doctor's diagnosis result" refers to the diagnosis result reached by a doctor based on his or her own expertise and judgment, based on the patient's clinical diagnostic data.

[0012] "Means of integrating and displaying on the user interface" refers to a mechanism that combines the AI's diagnostic results and the doctor's diagnostic results into one and displays them on the screen in a format that the doctor can check.

[0013] "Means for making a final diagnosis" refers to the process by which a doctor determines a final diagnosis based on the integrated diagnostic results.

[0014] "Means for storing feedback in a database" refers to a mechanism for collecting opinions and information, such as the final diagnosis made by the doctor and how the AI ​​diagnosis was helpful in the diagnostic process, and storing this information in a database.

[0015] "User interface" refers to the design of easy-to-use screens and interactions that allow doctors to input data, check diagnostic results, and provide feedback. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention improves diagnostic accuracy by having a doctor and an AI simultaneously perform diagnoses based on the patient's clinical diagnostic data and then displaying the integrated results. A specific example of this system is described below.

[0038] Data collection and input

[0039] The user (doctor) inputs the patient's clinical diagnostic data. This data includes blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data. For example, blood test results include blood glucose and cholesterol levels, and in addition, chest X-ray images and heart sound data are also input.

[0040] Data transmission and storage

[0041] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored as numerical data, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[0042] Data analysis and diagnostic preparation

[0043] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the possibility of a diagnosis based on this analysis data and generates a list of diagnoses and their rationale.

[0044] Comparison of diagnoses by doctors and AI

[0045] The server compares the diagnostic results generated by the AI ​​with the diagnostic data entered by the user (doctor). Here, the AI ​​diagnostic results, the doctor's diagnostic results, and supplementary information are integrated to generate data to be displayed on the user interface. This allows the user (doctor) to make a final diagnosis while referring to the AI ​​diagnostic results.

[0046] Recording and feedback

[0047] The server stores the final diagnosis made by the user (doctor) in a database. It also receives feedback on how helpful the AI ​​diagnosis was during the diagnostic process. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[0048] User Interface

[0049] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[0050] Implementing specific examples

[0051] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. This is then used to make the final diagnosis and provide feedback.

[0052] The present invention enables doctors to make diagnoses more accurately and quickly, and can significantly improve the accuracy and efficiency of diagnosis.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0056] Step 2:

[0057] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[0058] Step 3:

[0059] The server stores the received clinical diagnostic data in a database: blood test results are stored in a numerical database, imaging diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored in an appropriate format.

[0060] Step 4:

[0061] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[0062] Step 5:

[0063] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, the module evaluates the possibility of diabetes, and if a chest X-ray image is used, the module evaluates the possibility of pneumonia.

[0064] Step 6:

[0065] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0066] Step 7:

[0067] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[0068] Step 8:

[0069] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[0070] Step 9:

[0071] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[0072] Step 10:

[0073] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] In modern medical settings, we often rely solely on doctors' diagnoses, which poses challenges in diagnostic accuracy and efficiency. In particular, integrating multiple diagnostic data to make a comprehensive diagnosis is time-consuming and laborious. Furthermore, feedback during the diagnostic process is not adequately collected and utilized, creating a need for the development of a diagnostic support system using artificial intelligence.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes means for receiving patient's biological information, means for storing the received biological information in a recording device, means for analyzing the stored biological information, means for an AI module to generate a diagnosis result based on the analysis result, means for integrating the AI ​​module's diagnosis result with the doctor's diagnosis result and displaying it on a display device, means for making a final diagnosis based on the integrated diagnosis result, and means for collecting feedback during the diagnostic process, thereby enabling the accuracy and efficiency of diagnosis to be improved.

[0079] "Patient's biological information" refers to data indicating the patient's health condition, and includes blood test results, diagnostic imaging data, auscultation data, visual examination data, palpation data, and the like.

[0080] The "receiving means" refers to a means for inputting the patient's vital signs into the system, such as a data input interface or a data transfer protocol.

[0081] "Recording device" refers to a device for storing received patient biometric information in digital form, such as a database or storage system.

[0082] The "analyzing means" refers to a means for carrying out data analysis necessary for making a medical diagnosis based on the stored biological information, and includes a numerical analysis module, an image analysis module, and the like.

[0083] An "artificial intelligence module" is a device or software that generates diagnostic results based on analyzed biometric information, and includes machine learning algorithms and AI models.

[0084] "Means for integrating and displaying" refers to a means for comparing and integrating the diagnostic results generated by the AI ​​module with those of the doctor and visually presenting them to the user, and refers to a display device or graphical user interface (GUI).

[0085] The "means for making a final diagnosis" refers to the means by which a doctor determines a final diagnosis based on the integrated diagnostic results, and includes a diagnostic support system and an interface that supports the doctor's judgment.

[0086] "Means for collecting feedback" refers to means for receiving, analyzing, and storing feedback provided during the diagnostic process, and refers to a feedback input form and a data analysis module.

[0087] The system of this invention collects patient biometric information, and doctors and artificial intelligence jointly perform diagnoses, and the results are integrated and displayed to improve the accuracy of diagnoses. Specific embodiments of the present invention are described below.

[0088] Data collection and input

[0089] The user (doctor) enters the patient's biometric information into the system, which includes the following data:

[0090] Blood test results (e.g., blood sugar 180 mg / dL, cholesterol 250 mg / dL)

[0091] Imaging data (e.g., chest x-rays)

[0092] Auscultation data (e.g., heart sound data)

[0093] Visual inspection data

[0094] Palpation data

[0095] The user inputs data into a form or template on the system's user interface and uploads image and audio data as files.

[0096] Data transmission and storage

[0097] The device sends the input biometric information to a server using a secure communication protocol such as HTTPS.

[0098] The server stores the received biometric information in a recording device as follows:

[0099] Storing blood test results in a numerical database

[0100] Storing diagnostic imaging data in an image database

[0101] Save auscultation, visual inspection, and palpation data as text or audio data

[0102] Data analysis and diagnostic preparation

[0103] The server passes the data stored in the recording device to the analysis module, which includes the following specific processes:

[0104] Blood test results are analyzed by a numerical analysis module and compared with defined normal ranges to identify abnormal values.

[0105] The diagnostic imaging data is analyzed by the image analysis module to identify abnormalities.

[0106] The results of these analyses are passed to an artificial intelligence module, which evaluates the possibility of diagnosis based on the analyzed data and generates a list of diagnoses and their rationale.

[0107] Comparison of diagnoses by doctors and AI

[0108] The server compares the diagnosis results generated by the AI ​​module with the diagnosis data entered by the user (doctor). The specific process includes:

[0109] Compare AI diagnosis results with doctor diagnosis results

[0110] Identify commonalities and differences to generate integrated diagnostic results

[0111] As supplementary information, the basis for the AI ​​diagnosis and the doctor's reasons for the diagnosis will also be integrated.

[0112] The integrated diagnostic results are generated as data to be displayed on a user interface.

[0113] Recording and feedback

[0114] The server stores the final diagnosis made by the user (doctor) in the recording device. Specific operations include:

[0115] Record the final diagnostic results in the appropriate database table

[0116] Collect feedback on the effectiveness of the AI ​​in the diagnostic process and store it as training data for the AI ​​module.

[0117] User Interface

[0118] The terminal provides a user interface with the following features:

[0119] Data entry screens: Forms and templates that allow doctors to easily enter data

[0120] Diagnostic result display screen: A graphical user interface (GUI) that makes it easy to compare AI diagnostic results with those of a doctor.

[0121] Feedback input screen: A form where doctors can enter brief feedback

[0122] Implementing specific examples

[0123] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which then stores it in a recording device. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made.

[0124] Prompt Sentence Examples

[0125] Here is an example of a prompt you can enter into the system:

[0126] "AI diagnosis will be performed based on the patient's clinical data. Please enter the following data.

[0127] Blood test results:

[0128] Blood glucose level: 180 mg / dL

[0129] Cholesterol level: 250 mg / dL

[0130] Imaging data: Chest X-ray image (file upload)

[0131] Generate your results and compare them with your doctor's diagnosis. View the results and collect feedback."

[0132] Using this prompt sentence, the system can make an accurate and efficient diagnosis, which helps support doctors in making diagnoses.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1: Data collection and input

[0135] The user (doctor) inputs the patient's biometric information into the system. This biometric information includes blood test results (e.g., blood glucose level 180 mg / dL, cholesterol level 250 mg / dL), diagnostic imaging data (e.g., chest X-ray image), auscultation data (e.g., heart sound data), visual examination data, and palpation data. The user (doctor) inputs this data using a form or template provided on the system's user interface and uploads the image and audio data as files. The input data is used as is in the next step.

[0136] Step 2: Send and store data

[0137] The device sends the entered biometric information to the server. This data transmission is done using a secure communication protocol such as HTTPS. The transmitted data is then stored in the appropriate database on the server. Specifically, blood test results are stored in a numerical database, diagnostic imaging data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text or audio data. This makes the biometric information available for analysis.

[0138] Step 3: Prepare for data analysis and diagnosis

[0139] The server passes the data stored in the recording device to the analysis module. Specifically, blood test results are first analyzed by a numerical analysis module. Abnormal values ​​are identified by comparison with normal ranges, and this information is passed to the next step. Image diagnostic data is analyzed by an image analysis module, and abnormal areas are identified. These analysis results are passed to an artificial intelligence module, which generates a diagnosis list and diagnostic rationale.

[0140] Step 4: Comparing the doctor's and AI's diagnoses

[0141] The server compares the diagnosis results generated by the AI ​​module with the diagnostic data entered by the user (doctor). Specifically, the AI ​​diagnosis results are compared with the doctor's diagnosis results, and similarities and differences are identified. The integrated diagnosis results, along with supplementary information (e.g., the basis of the AI ​​diagnosis and the doctor's reasons for the diagnosis), are generated as data to be displayed on the user interface. They are presented in an intuitive format so that the user can refer to the final diagnosis.

[0142] Step 5: Recording and feedback

[0143] The server stores the final diagnosis made by the user (doctor) in a recording device. Specifically, it records the final diagnosis in an appropriate database table and collects feedback on the effectiveness of the AI ​​in the diagnostic process. This feedback is stored as training data for the AI ​​module and used to improve diagnostic accuracy in the future.

[0144] Step 6: Navigate the User Interface

[0145] The terminal provides an intuitive user interface. Specifically, the data entry screen is equipped with forms and templates that doctors can use to easily enter data. The diagnosis result display screen provides a graphical user interface (GUI) designed to make it easy to compare the AI's diagnosis results with those of doctors. The feedback input screen provides a form that doctors can use to easily enter feedback, allowing for the collection of information to improve the quality of diagnoses.

[0146] These steps enable the system to achieve accurate and efficient diagnosis through collaboration between doctors and AI, improving the accuracy of the final diagnosis.

[0147] (Application example 1)

[0148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] Conventional medical diagnostic systems and factory quality inspection systems have limitations in accuracy and efficiency when integrating multiple diagnostic results. It has also been difficult to effectively integrate the results of AI and human diagnoses and reflect them in the final diagnosis. With the demand for improved diagnostic accuracy and efficiency in medical and factory settings, it is necessary to solve these issues.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0151] In this invention, the server

[0152] means for receiving clinical diagnostic data of a patient;

[0153] means for storing the received clinical diagnostic data in a database;

[0154] means for analyzing the stored clinical diagnostic data;

[0155] A means for AI to generate diagnostic results based on the analysis results, and

[0156] A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface;

[0157] A means for making a final diagnosis based on the integrated diagnostic results;

[0158] means for receiving quality inspection data;

[0159] means for analyzing the received quality inspection data;

[0160] A method for AI and robots to generate quality diagnosis results based on the analysis results, and

[0161] A means to integrate the diagnostic results of AI and robots and display them on a user interface,

[0162] Includes means for providing audio feedback.

[0163] This will enable improvements in diagnostic accuracy in the medical field and quality inspection accuracy in factories.

[0164] "Patient" refers to a person who is the subject of diagnosis or treatment at a medical institution.

[0165] "Clinical diagnostic data" refers to a variety of biological, imaging, audiological, and mechanical data collected to assess a patient's health status.

[0166] A "database" refers to a system for efficiently storing, retrieving, and managing data.

[0167] "Analysis" refers to performing computations to detect patterns or anomalies in the received data.

[0168] "AI" refers to artificial intelligence that can automatically perform specific tasks with high accuracy.

[0169] "Diagnosis results" refer to the evaluation results of health status and quality obtained based on data analysis.

[0170] "Physician" refers to a professional with higher education who diagnoses and treats patients.

[0171] "User interface" refers to the interface through which a user interacts with a system.

[0172] "Final diagnosis" refers to the final assessment or conclusion obtained by integrating the diagnostic results of the AI ​​and the doctor (or robot).

[0173] "Quality Inspection Data" means data collected to evaluate the quality of a product.

[0174] "Robot" refers to a mechanical device for performing tasks automatically or semi-automatically.

[0175] "Voice feedback" refers to the function of providing analysis and diagnostic results to the user via voice.

[0176] This invention is a system that uses AI and robots to perform highly accurate diagnoses based on patient clinical diagnostic data and factory quality inspection data, and then integrates and displays the results. This system can improve the accuracy and efficiency of diagnoses in both the medical and manufacturing fields. Specific examples are described below.

[0177] System overview

[0178] The system is composed of a server, a terminal, and a user (a doctor or a diagnostician). The system includes the following means:

[0179] 1. Means for receiving patient clinical diagnosis data: A means for users to input patient blood test results, image diagnosis data, auscultation data, visual examination data, palpation data, etc. through a terminal.

[0180] 2. Database storage means: A means for efficiently storing received data in a database. Blood test results are stored as numerical data, and diagnostic imaging data is stored as image files, audio data, etc.

[0181] 3. Data analysis method: This is a method for analyzing the stored data and detecting outliers, etc. Software such as Python, Keras, OpenCV, and Pyttsx3 is used for this.

[0182] 4. AI diagnostic result generation means: This is the means by which AI generates diagnostic results based on the analysis results. The AI ​​model uses a pre-trained model.

[0183] 5. Diagnostic result integration means: A means for integrating the AI ​​diagnostic results with the doctor's diagnostic results and displaying them on the user interface.

[0184] 6. Final diagnostic tool: A tool for making a final diagnosis based on the integrated diagnostic results.

[0185] 7. Quality inspection data receiving means: This is the means by which the factory robot receives the quality inspection data of the product.

[0186] 8. Quality inspection data analysis means: A means for analyzing received quality inspection data.

[0187] 9. Means for generating diagnostic results using AI and robots: A means for AI and robots to generate diagnostic results based on analyzed quality inspection data.

[0188] 10. Audio feedback means: A means for providing the diagnostic results to the user by voice.

[0189] Specific examples

[0190] Specific examples in the medical field

[0191] Clinical data for a hypothetical patient (a 45-year-old male) is entered, including blood glucose levels (180 mg / dL), cholesterol levels (250 mg / dL), and chest X-ray images. This data is sent to a server and stored in a database. Data analysis is then performed, and the AI ​​detects abnormal values ​​and generates diagnoses of diabetes and pneumonia. These results are displayed to the doctor through a user interface, who then makes a final diagnosis based on the integrated information.

[0192] Specific example of factory quality inspection

[0193] Factory robots receive image data as product quality inspection data, which is then analyzed by AI. For example, an image analysis algorithm is applied to determine whether there are scratches on the surface of the product. The diagnostic results of the AI ​​and robot are integrated and displayed on the user interface. As a result, quality control personnel can make the final quality judgment.

[0194] Prompt Sentence Examples

[0195] Analyze the product image below and output the diagnostic results regarding the product quality.

[0196] Example: main("product_image.jpg", "quality_model.h5", "robot_measured_data")

[0197] The configuration and implementation of this system can significantly improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[0198] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0199] Step 1:

[0200] The user inputs the patient's clinical diagnostic data into the terminal, including blood test results, imaging diagnostic data, auscultation data, visual examination data, palpation data, etc. This data is collected through an input form or file upload.

[0201] Step 2:

[0202] The terminal transmits the entered clinical diagnostic data to a server over the Internet, and the data is transmitted in multiple formats (text, numbers, images, and audio).

[0203] Step 3:

[0204] The server stores the received clinical diagnostic data in a database: blood test results in a numerical database, diagnostic imaging data in an image database, and voice data in a voice database.

[0205] Step 4:

[0206] The server analyzes the stored data: numerical data is analyzed using outlier detection algorithms, image data is analyzed using image analysis models to detect lesions, and audio data is analyzed using audio analysis algorithms.

[0207] Step 5:

[0208] The server uses the analysis results to generate a diagnosis using AI, which uses a pre-trained generative AI model to evaluate the disease name and likelihood of the condition.

[0209] Step 6:

[0210] The server integrates the AI's diagnosis results with those of the doctor, comparing the diagnosis results entered by the doctor with those generated by the AI ​​and integrating them based on certain rules.

[0211] Step 7:

[0212] The server displays the integrated diagnostic results on a user interface, through which the user (doctor) can confirm the diagnostic results and make a final diagnosis.

[0213] Step 8:

[0214] The user enters the final diagnosis and the server stores the results in a database, including any feedback or additional comments.

[0215] Step 9:

[0216] The user inputs factory quality inspection data into the terminal, including image files and sensor data.

[0217] Step 10:

[0218] The terminal transmits the input quality inspection data to the server.

[0219] Step 11:

[0220] The server analyzes the received quality inspection data. Image data is analyzed using image analysis models to detect product defects, and robot measurement data is analyzed using anomaly detection algorithms.

[0221] Step 12:

[0222] The server then uses the AI ​​and robots to generate quality diagnosis results based on the analysis results. The AI ​​model evaluates the product's quality, and the robot measures the product's physical characteristics.

[0223] Step 13:

[0224] The server integrates the diagnostic results of the AI ​​and robot and displays them on a user interface. The user (quality control officer) can then check the quality diagnostic results through this interface and make a final decision.

[0225] Step 14:

[0226] The server provides the diagnosis results to the user through voice feedback, using a speech synthesis library such as Pyttsx3.

[0227] Taking such steps can improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[0228] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0229] The system according to the present invention receives and analyzes the clinical diagnostic data of patients, and based on the results, AI generates diagnostic results, which are then integrated with the doctor's diagnostic results and displayed. In addition, it incorporates an emotion engine that recognizes the user's emotions, and utilizes the doctor's emotional state in diagnostic support. Specific embodiments of this system are described below.

[0230] Data collection and input

[0231] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0232] Data transmission and storage

[0233] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored in a numerical database, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[0234] Data analysis and diagnostic preparation

[0235] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale.

[0236] Comparison of diagnoses by doctors and AI

[0237] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0238] Emotion Engine Monitoring

[0239] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[0240] Emotion-based feedback and warnings

[0241] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the doctor's emotional state. For example, if the doctor is feeling stressed, it displays a warning message urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, it provides additional diagnostic support information.

[0242] Recording and feedback

[0243] The server stores the final diagnosis made by the user (doctor) in a database. It also receives and stores feedback on the user's emotional state during the diagnostic process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[0244] User Interface

[0245] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[0246] Implementing specific examples

[0247] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show a blood glucose level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes the data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[0248] The present invention allows doctors to make diagnoses more accurately and quickly, significantly improving the accuracy and efficiency of diagnoses. Furthermore, by utilizing the doctor's emotional state as diagnostic support, the quality of diagnoses can be further improved.

[0249] The processing flow will be explained below.

[0250] Step 1:

[0251] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0252] Step 2:

[0253] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[0254] Step 3:

[0255] The server stores the received clinical diagnostic data in a database. Blood test results are stored in a numerical database, image diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text and audio data.

[0256] Step 4:

[0257] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[0258] Step 5:

[0259] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, it evaluates the possibility of diabetes, and if a chest X-ray image is taken, it evaluates the possibility of pneumonia.

[0260] Step 6:

[0261] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0262] Step 7:

[0263] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[0264] Step 8:

[0265] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[0266] Step 9:

[0267] The server receives the emotion data sent from the emotion engine and provides diagnostic assistance based on the doctor's emotional state. For example, if the doctor is feeling stressed, a warning message is displayed urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, additional diagnostic assistance information is provided.

[0268] Step 10:

[0269] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[0270] Step 11:

[0271] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[0272] Step 12:

[0273] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[0274] Example 2

[0275] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0276] Conventional clinical diagnostic systems lack the integration of doctor and AI diagnoses, and lack diagnostic support that takes into account the doctor's emotional state. As a result, diagnostic accuracy and doctor work efficiency may decrease, and an appropriate diagnosis may not be made. A system that solves these problems and improves diagnostic accuracy and efficiency is needed.

[0277] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0278] In this invention, the server provides means for receiving clinical diagnostic data of patients, means for storing the received clinical diagnostic data in a database, means for analyzing the stored clinical diagnostic data, means for an AI to generate a diagnostic result based on the analysis result, means for integrating the AI's diagnostic result and the doctor's diagnostic result and displaying them on a user interface, means for recognizing the doctor's emotional state, means for providing feedback and warnings based on the emotional state, and means for making a final diagnosis based on the integrated diagnostic result, thereby improving the accuracy and efficiency of diagnosis and enabling appropriate diagnostic support that takes the doctor's emotional state into consideration.

[0279] "Patient clinical diagnostic data" includes medical information about the patient, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[0280] "Means for receiving" refers to a combination of hardware and software for electronically receiving patient clinical diagnostic data.

[0281] "Means for storing in a database" refers to a storage device and associated management software for structuring and archiving received clinical diagnostic data.

[0282] "Means for analysis" refers to algorithms and software for computer-based analysis of stored clinical diagnostic data to detect outliers and specific patterns.

[0283] "Means for artificial intelligence to generate diagnostic results" refers to software that uses machine learning models to evaluate diagnostic possibilities based on the analysis results and generates diagnostic results based on that.

[0284] "Means for integrating and displaying on a user interface" refers to software and hardware that compares and integrates the AI ​​diagnostic results with the doctor's diagnostic results and provides a user interface for visually displaying that information.

[0285] "Means for recognizing the doctor's emotional state" refers to sensors and analysis software for analyzing the doctor's voice and facial expressions to determine his or her emotional state.

[0286] "Means for providing feedback and warnings" refers to software and devices for generating and presenting appropriate advice and warnings in response to a perceived emotional state.

[0287] "Means for making a final diagnosis" refers to the process and tools that allow a physician to determine a final diagnosis using the integrated diagnostic results and feedback information.

[0288] The system according to the present invention receives and analyzes a patient's clinical diagnostic data, and then uses artificial intelligence to generate a diagnostic result based on the analysis results, which is then integrated with the doctor's diagnostic result and displayed. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system is characterized by utilizing the doctor's emotional state in diagnostic support. A specific embodiment of this system is described below.

[0289] Data collection and input

[0290] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. The terminal is provided with a form for data input, and for example, the user can enter a blood glucose level of 180 mg / dL, a cholesterol level of 250 mg / dL, and upload a chest X-ray image.

[0291] Data transmission and storage

[0292] The device structures the input clinical diagnostic data and sends it to the server using an HTTP request. The server receives this data in the appropriate format and stores it in a relational database, such as a MySQL® database. Numeric data is stored in the numeric database, image data in the image database, and auscultation, visual inspection, and palpation data in the text database and audio database.

[0293] Data analysis and diagnostic preparation

[0294] The server analyzes the stored data. For example, blood test results are analyzed using Python's numerical calculation libraries (NumPy and Pandas) to detect abnormalities. Chest X-ray images are analyzed to detect abnormalities using deep learning frameworks such as TENSORFLOW (registered trademark) and PyTorch. The analyzed data is passed to an AI module, which uses a generative AI model to evaluate diagnostic possibilities and generate a list of diagnoses and their rationale.

[0295] Comparison of diagnoses by doctors and AI

[0296] The server compares the AI-generated diagnosis with the diagnosis entered by the user (doctor), uses a comparison algorithm to identify matches and differences, adds complementary information (e.g., relevant research data and statistical data), and generates a unified diagnosis, which is ultimately presented to the doctor.

[0297] Emotion Engine Monitoring

[0298] While the user (doctor) is checking the diagnosis results, the device sends data to the emotion engine using a camera and microphone. The emotion engine analyzes facial expressions using libraries such as OpenCV and Dlib, and voice tone and pitch using a voice analysis library, to recognize the doctor's emotional state (stress level or lack of attention) in real time.

[0299] Emotion-based feedback and warnings

[0300] The server receives the emotional data sent from the emotion engine and generates appropriate feedback based on the doctor's emotional state. For example, if the doctor is feeling stressed, it sends a warning message to the device in real time via WebSocket, prompting the doctor to reconfirm the diagnosis. Furthermore, if the doctor's emotional state is determined to have a significant impact on the diagnosis, it provides AI-generated diagnostic support information.

[0301] Recording and feedback

[0302] The server stores the doctor's final diagnosis in a database. It also receives and analyzes feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support. This feedback is used for future AI training to improve diagnostic accuracy.

[0303] User Interface

[0304] The device provides a user interface that allows intuitive data entry, confirmation of diagnosis results, and feedback. The data entry screen displays input forms and templates, allowing doctors to quickly enter data. The diagnosis result display screen also provides a GUI that allows easy visual comparison of the AI's diagnosis results with those of the doctor. The feedback entry screen displays a simple input form, allowing doctors to enter feedback in a short amount of time.

[0305] Hypothetical example

[0306] Let's take the hypothetical case of a 45-year-old male patient. His blood test results show a blood sugar level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is uploaded. The device sends this data to the server, which analyzes it. The AI ​​module uses this data to evaluate the possibility of diabetes and pneumonia and generate a diagnosis. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[0307] Prompt Sentence Examples

[0308] The following prompt sentences can be input to a generative AI model to explain the specific functions of the system:

[0309] A 45-year-old male patient's blood test results are entered: blood sugar 180 mg / dL (abnormal) and cholesterol 250 mg / dL (high), and a chest x-ray is uploaded. Explain how the system will operate based on this data.

[0310] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0311] Program processing flow

[0312] Step 1: Data collection and entry

[0313] A user (doctor) needs to input a patient's clinical diagnostic data into a terminal. Input items include blood test results, image diagnostic data, auscultation data, visual examination data, and palpation data. For example, a blood glucose level of 180 mg / dL and a cholesterol level of 250 mg / dL are input as numerical values, and a chest X-ray image is uploaded as a file. This data is structured using an input form on the terminal.

[0314] Input: Patient clinical diagnostic data (numerical data, image data, etc.)

[0315] Output: Structured data format

[0316] Step 2: Send and store data

[0317] The device sends structured clinical diagnostic data to the server using HTTP requests. The server receives this data and stores it in a relational database, such as a MySQL database. Numeric data is stored in a numeric database, image data in an image database, and auscultation, visual inspection, and palpation data in a text database or audio database.

[0318] Input: Structured data format

[0319] Output: Data stored in the database

[0320] Step 3: Data analysis and diagnostic preparation

[0321] The server analyzes the stored data. Blood test results are analyzed using Python's numerical calculation libraries (NumPy, Pandas) to detect abnormal values. Chest X-ray images are analyzed using TensorFlow and PyTorch to detect abnormal areas. The analysis results are passed to an AI module, which uses a generative AI model to evaluate diagnostic feasibility and generate a diagnosis list and rationale.

[0322] Input: Data stored in a database

[0323] Output: Analysis results, diagnostic list

[0324] Step 4: Comparing diagnoses between doctors and AI

[0325] The server compares the diagnosis results generated by the generative AI model with those entered by the user (doctor). It uses a comparison algorithm to identify similarities and differences. It adds relevant supplementary information (related research data, statistical data, etc.) to generate a final integrated diagnosis.

[0326] Input: Analysis results, diagnosis list, doctor's diagnosis

[0327] Output: Integrated diagnostic results

[0328] Step 5: Emotion Engine Monitoring

[0329] While the user (doctor) is checking the diagnosis results, the device uses a camera and microphone to send emotional data to the emotion engine, which then analyzes facial expressions using OpenCV and Dlib, and voice tone and pitch using a voice analysis library to recognize the doctor's emotional state.

[0330] Input: Audio and video data of the doctor confirming the diagnosis

[0331] Output: Doctor's emotional state data (stress level, attention deficit)

[0332] Step 6: Emotion-based feedback and warnings

[0333] The server receives emotional state data sent from the emotion engine and generates feedback and warning messages based on that data. For example, if a doctor is feeling stressed, a warning message urging reconsideration is sent to the device via WebSocket and displayed.

[0334] Input: Emotional state data

[0335] Output: Feedback and warning messages

[0336] Step 7: Recording and feedback

[0337] The server stores the doctor's final diagnosis results in a database. It also receives feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support, and stores this information in the database. This feedback information is used for further learning by the AI, contributing to improving diagnostic accuracy.

[0338] Input: Final diagnosis results, feedback information

[0339] Output: Feedback information stored in a database

[0340] Step 8: User Interface

[0341] The device provides an intuitive user interface through which users (doctors) can input data, check diagnosis results, and provide feedback. The data input screen displays specific forms and templates. The diagnosis result display screen displays the AI's diagnosis results and the doctor's diagnosis results in an easy-to-read format. The feedback input screen provides a simple input form.

[0342] Input: Data entry, diagnostic result confirmation, feedback entry

[0343] Output: User feedback, diagnostic results

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] Conventional maintenance and fault diagnosis systems for factory robots only use analytical results based on the robot's sensor data, and do not reflect the emotional state of the maintenance staff. As a result, early detection of robot abnormalities or breakdowns is difficult due to mistakes and delayed judgment caused by stress or fatigue.

[0347] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving sensor data from the robot, means for analyzing the saved sensor data, means for an AI to evaluate the possibility of a failure based on the analysis results and generate a diagnosis result, means for receiving emotion data using an emotion engine that recognizes the emotional state of the maintenance staff, and means for providing diagnosis support based on the emotional state. This enables failure diagnosis support that takes into account the emotional state of the staff as well as the sensor data.

[0348] "Patient clinical diagnostic data" refers to medical data collected for diagnosis, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[0349] A "numerical database" refers to a database that stores numerical data such as blood test results and sensor data.

[0350] "Image database" refers to a database that stores image data such as X-ray images and CT scans.

[0351] "Analysis" refers to the data processing and evaluation process that detects abnormal values ​​and abnormal areas based on stored clinical diagnostic data and sensor data.

[0352] "AI generates diagnostic results" refers to the process of creating a diagnostic list and its rationale using a machine learning algorithm based on analyzed data.

[0353] "Doctor's diagnosis result" refers to the diagnostic judgment and conclusion made by the doctor regarding the condition of the patient or robot.

[0354] "User Interface" refers to the system's interactive screen through which a user (physician or maintenance technician) can input data, review diagnostic results, and provide feedback.

[0355] "Emotion engine" refers to technology that uses voice analysis and facial expression recognition technology to identify a user's emotional state and assess their stress and fatigue levels.

[0356] "Emotion Data" refers to data regarding emotional states (e.g., stress, joy, sadness, etc.) recognized by the emotion engine.

[0357] "Diagnostic support" refers to providing additional information and warning messages during the diagnostic process to help doctors and maintenance personnel make better decisions.

[0358] "Feedback" refers to information entered as evaluations and impressions of diagnostic results and work processes, which are used as learning data for the system.

[0359] In the present invention, a plurality of hardware and software components are integrated to provide a maintenance and fault diagnosis support system for a factory robot. Specific embodiments will be described below.

[0360] Data collection and input

[0361] The user (maintenance staff) inputs the robot's sensor data into a dedicated terminal. This data includes various sensor data such as temperature, vibration, and current. The terminal collects this data in real time and sends it to a server. A general-purpose PC or smart device is suitable as the terminal to be used.

[0362] Data transmission and storage

[0363] The device sends the input sensor data to the server, which stores the received data in a numerical database and an image database. For example, temperature and vibration data are stored in the numerical database and analyzed as needed.

[0364] Data analysis and fault diagnosis

[0365] The server analyzes the stored sensor data using a data processing module written in Python. The analysis results are passed to an AI module (built using TensorFlow / Keras), which evaluates the possibility of a fault and generates a diagnosis.

[0366] Display and compare diagnostic results

[0367] The server displays the diagnostic results generated by the AI ​​on a user interface, which provides an intuitive graphical user interface (GUI) that allows users (maintenance personnel) to check the diagnostic results.

[0368] Emotion Engine Monitoring

[0369] The device uses an emotion engine to recognize the emotional state of the person checking the diagnosis results. The emotion engine uses facial recognition technology (OpenCV) and voice analysis technology to detect stress and fatigue of the person.

[0370] Emotion-based feedback and warnings

[0371] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the person in charge is feeling stressed, the server displays a warning message urging the person to reconfirm the diagnosis result. In addition, if the person in charge's emotional state affects the diagnosis, the server provides additional diagnostic support information.

[0372] Recording and feedback

[0373] The server stores the user's final diagnosis results in a numerical database. It also receives and stores feedback on the user's emotional state during the diagnosis process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI, helping to improve its diagnostic accuracy in the future.

[0374] Implementing specific examples

[0375] In a factory, a maintenance worker performs a routine inspection of a factory robot. The worker operates a smart device to collect sensor data (such as temperature and vibration) in real time. This data is then sent to a server, where it is analyzed and displayed along with the results of a diagnosis made by an AI module. Meanwhile, an emotion engine recognizes the worker's stress level and provides feedback to encourage relaxation as needed.

[0376] (Example of prompts for a generative AI model)

[0377] "Detect stress levels in factory maintenance personnel and generate prompts for an assistant system that provides feedback to encourage breaks."

[0378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0379] Step 1:

[0380] The user (maintenance technician) begins maintenance work on a factory robot. First, various sensor data such as temperature, vibration, and current are collected from sensors connected to a dedicated terminal and input into the terminal. At this time, the data from each sensor is displayed in real time on the terminal's UI.

[0381] Step 2:

[0382] The device sends the collected sensor data to the server. The sensor data includes temperature values ​​measured every 0.1 seconds, vibration frequency spectrum, and current time series data. The device then converts the data into a specific format (e.g., JSON format) and sends it to the server via the network.

[0383] Step 3:

[0384] The server sequentially stores the received sensor data in a numeric database and an image database. When storing the data in the database, the data is separated by sensor type and stored in the appropriate field. For example, temperature data is stored in the "temperature" field, and vibration data is stored in the "vibration" field.

[0385] Step 4:

[0386] The server analyzes the stored sensor data. Specifically, it detects abnormal values ​​in temperature data, analyzes the frequency of vibration data, and recognizes patterns in current data. From this analysis, an anomaly detection algorithm is applied to extract abnormal values ​​and patterns. The analysis results include peak temperature values, the ratio of high-frequency components in vibration, and abnormal current peaks.

[0387] Step 5:

[0388] The server passes the analysis results to an AI module (using TensorFlow / Keras), which then generates a diagnosis. The AI ​​module evaluates current sensor data based on past learning data and predicts the likelihood of robot failure. The diagnosis results include the possibility of wear on specific parts, failure due to overheating, and damage due to abnormal vibration.

[0389] Step 6:

[0390] The server displays the diagnostic results generated by the AI ​​on a user interface. The user interface uses a GUI to visually display the diagnostic results, allowing the user to intuitively understand them. Specifically, it displays an overview of the diagnostic results, highlights the areas where abnormalities have occurred, and recommends actions (e.g., part replacement, adjustment).

[0391] Step 7:

[0392] The device uses an emotion engine to recognize the user's emotional state while checking the diagnosis results. The emotion engine captures the user's face with the device's built-in camera and analyzes their emotions using a facial expression recognition algorithm (OpenCV). This analysis determines whether the user is feeling stressed or relaxed.

[0393] Step 8:

[0394] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the emotion analysis detects that the user is feeling stressed, the server displays a warning message on the user interface. This message may include a message to reconfirm the diagnosis result or to take a break.

[0395] Step 9:

[0396] The server stores the final diagnosis results and emotion data provided by the user in a numerical database. The stored data will be used as training data to improve the accuracy of the diagnosis. In addition, the user can input feedback on the final diagnosis results via the user interface, and this feedback will also be stored in the database.

[0397] Step 10:

[0398] The server analyzes the stored final diagnosis results and emotion data and uses them to retrain the AI ​​model, a process that improves future diagnosis accuracy and overall system performance.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0400] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0402] [Second embodiment]

[0403] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0404] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0405] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0407] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0409] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0410] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0411] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0414] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0415] The system of the present invention improves diagnostic accuracy by having a doctor and an AI simultaneously perform diagnoses based on the patient's clinical diagnostic data and then displaying the integrated results. A specific example of this system is described below.

[0416] Data collection and input

[0417] The user (doctor) inputs the patient's clinical diagnostic data. This data includes blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data. For example, blood test results include blood glucose and cholesterol levels, and in addition, chest X-ray images and heart sound data are also input.

[0418] Data transmission and storage

[0419] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored as numerical data, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[0420] Data analysis and diagnostic preparation

[0421] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the possibility of a diagnosis based on this analysis data and generates a list of diagnoses and their rationale.

[0422] Comparison of diagnoses by doctors and AI

[0423] The server compares the diagnostic results generated by the AI ​​with the diagnostic data entered by the user (doctor). Here, the AI ​​diagnostic results, the doctor's diagnostic results, and supplementary information are integrated to generate data to be displayed on the user interface. This allows the user (doctor) to make a final diagnosis while referring to the AI ​​diagnostic results.

[0424] Recording and feedback

[0425] The server stores the final diagnosis made by the user (doctor) in a database. It also receives feedback on how helpful the AI ​​diagnosis was during the diagnostic process. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[0426] User Interface

[0427] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[0428] Implementing specific examples

[0429] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. This is then used to make the final diagnosis and provide feedback.

[0430] The present invention enables doctors to make diagnoses more accurately and quickly, and can significantly improve the accuracy and efficiency of diagnosis.

[0431] The processing flow will be explained below.

[0432] Step 1:

[0433] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0434] Step 2:

[0435] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[0436] Step 3:

[0437] The server stores the received clinical diagnostic data in a database: blood test results are stored in a numerical database, imaging diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored in an appropriate format.

[0438] Step 4:

[0439] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[0440] Step 5:

[0441] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, the module evaluates the possibility of diabetes, and if a chest X-ray image is used, the module evaluates the possibility of pneumonia.

[0442] Step 6:

[0443] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0444] Step 7:

[0445] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[0446] Step 8:

[0447] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[0448] Step 9:

[0449] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[0450] Step 10:

[0451] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[0452] Example 1

[0453] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0454] In modern medical settings, we often rely solely on doctors' diagnoses, which poses challenges in diagnostic accuracy and efficiency. In particular, integrating multiple diagnostic data to make a comprehensive diagnosis is time-consuming and laborious. Furthermore, feedback during the diagnostic process is not adequately collected and utilized, creating a need for the development of a diagnostic support system using artificial intelligence.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0456] In this invention, the server includes means for receiving patient's biological information, means for storing the received biological information in a recording device, means for analyzing the stored biological information, means for an AI module to generate a diagnosis result based on the analysis result, means for integrating the AI ​​module's diagnosis result with the doctor's diagnosis result and displaying it on a display device, means for making a final diagnosis based on the integrated diagnosis result, and means for collecting feedback during the diagnostic process, thereby enabling the accuracy and efficiency of diagnosis to be improved.

[0457] "Patient's biological information" refers to data indicating the patient's health condition, and includes blood test results, diagnostic imaging data, auscultation data, visual examination data, palpation data, and the like.

[0458] The "receiving means" refers to a means for inputting the patient's vital signs into the system, such as a data input interface or a data transfer protocol.

[0459] "Recording device" refers to a device for storing received patient biometric information in digital form, such as a database or storage system.

[0460] The "analyzing means" refers to a means for carrying out data analysis necessary for making a medical diagnosis based on the stored biological information, and includes a numerical analysis module, an image analysis module, and the like.

[0461] An "artificial intelligence module" is a device or software that generates diagnostic results based on analyzed biometric information, and includes machine learning algorithms and AI models.

[0462] "Means for integrating and displaying" refers to a means for comparing and integrating the diagnostic results generated by the AI ​​module with those of the doctor and visually presenting them to the user, and refers to a display device or graphical user interface (GUI).

[0463] The "means for making a final diagnosis" refers to the means by which a doctor determines a final diagnosis based on the integrated diagnostic results, and includes a diagnostic support system and an interface that supports the doctor's judgment.

[0464] "Means for collecting feedback" refers to means for receiving, analyzing, and storing feedback provided during the diagnostic process, and refers to a feedback input form and a data analysis module.

[0465] The system of this invention collects patient biometric information, and doctors and artificial intelligence jointly perform diagnoses, and the results are integrated and displayed to improve the accuracy of diagnoses. Specific embodiments of the present invention are described below.

[0466] Data collection and input

[0467] The user (doctor) enters the patient's biometric information into the system, which includes the following data:

[0468] Blood test results (e.g., blood sugar 180 mg / dL, cholesterol 250 mg / dL)

[0469] Imaging data (e.g., chest x-rays)

[0470] Auscultation data (e.g., heart sound data)

[0471] Visual inspection data

[0472] Palpation data

[0473] The user inputs data into a form or template on the system's user interface and uploads image and audio data as files.

[0474] Data transmission and storage

[0475] The device sends the input biometric information to a server using a secure communication protocol such as HTTPS.

[0476] The server stores the received biometric information in a recording device as follows:

[0477] Storing blood test results in a numerical database

[0478] Storing diagnostic imaging data in an image database

[0479] Save auscultation, visual inspection, and palpation data as text or audio data

[0480] Data analysis and diagnostic preparation

[0481] The server passes the data stored in the recording device to the analysis module, which includes the following specific processes:

[0482] Blood test results are analyzed by a numerical analysis module and compared with defined normal ranges to identify abnormal values.

[0483] The diagnostic imaging data is analyzed by the image analysis module to identify abnormalities.

[0484] The results of these analyses are passed to an artificial intelligence module, which evaluates the possibility of diagnosis based on the analyzed data and generates a list of diagnoses and their rationale.

[0485] Comparison of diagnoses by doctors and AI

[0486] The server compares the diagnosis results generated by the AI ​​module with the diagnosis data entered by the user (doctor). The specific process includes:

[0487] Compare AI diagnosis results with doctor diagnosis results

[0488] Identify commonalities and differences to generate integrated diagnostic results

[0489] As supplementary information, the basis for the AI ​​diagnosis and the doctor's reasons for the diagnosis will also be integrated.

[0490] The integrated diagnostic results are generated as data to be displayed on a user interface.

[0491] Recording and feedback

[0492] The server stores the final diagnosis made by the user (doctor) in the recording device. Specific operations include:

[0493] Record the final diagnostic results in the appropriate database table

[0494] Collect feedback on the effectiveness of the AI ​​in the diagnostic process and store it as training data for the AI ​​module.

[0495] User Interface

[0496] The terminal provides a user interface with the following features:

[0497] Data entry screens: Forms and templates that allow doctors to easily enter data

[0498] Diagnostic result display screen: A graphical user interface (GUI) that makes it easy to compare AI diagnostic results with those of a doctor.

[0499] Feedback input screen: A form where doctors can enter brief feedback

[0500] Implementing specific examples

[0501] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which then stores it in a recording device. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made.

[0502] Prompt Sentence Examples

[0503] Here is an example of a prompt you can enter into the system:

[0504] "AI diagnosis will be performed based on the patient's clinical data. Please enter the following data.

[0505] Blood test results:

[0506] Blood glucose level: 180 mg / dL

[0507] Cholesterol level: 250 mg / dL

[0508] Imaging data: Chest X-ray image (file upload)

[0509] Generate your results and compare them with your doctor's diagnosis. View the results and collect feedback."

[0510] Using this prompt sentence, the system can make an accurate and efficient diagnosis, which helps support doctors in making diagnoses.

[0511] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0512] Step 1: Data collection and input

[0513] The user (doctor) inputs the patient's biometric information into the system. This biometric information includes blood test results (e.g., blood glucose level 180 mg / dL, cholesterol level 250 mg / dL), diagnostic imaging data (e.g., chest X-ray image), auscultation data (e.g., heart sound data), visual examination data, and palpation data. The user (doctor) inputs this data using a form or template provided on the system's user interface and uploads the image and audio data as files. The input data is used as is in the next step.

[0514] Step 2: Send and store data

[0515] The device sends the entered biometric information to the server. This data transmission is done using a secure communication protocol such as HTTPS. The transmitted data is then stored in the appropriate database on the server. Specifically, blood test results are stored in a numerical database, diagnostic imaging data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text or audio data. This makes the biometric information available for analysis.

[0516] Step 3: Prepare for data analysis and diagnosis

[0517] The server passes the data stored in the recording device to the analysis module. Specifically, blood test results are first analyzed by a numerical analysis module. Abnormal values ​​are identified by comparison with normal ranges, and this information is passed to the next step. Image diagnostic data is analyzed by an image analysis module, and abnormal areas are identified. These analysis results are passed to an artificial intelligence module, which generates a diagnosis list and diagnostic rationale.

[0518] Step 4: Comparing the doctor's and AI's diagnoses

[0519] The server compares the diagnosis results generated by the AI ​​module with the diagnostic data entered by the user (doctor). Specifically, the AI ​​diagnosis results are compared with the doctor's diagnosis results, and similarities and differences are identified. The integrated diagnosis results, along with supplementary information (e.g., the basis of the AI ​​diagnosis and the doctor's reasons for the diagnosis), are generated as data to be displayed on the user interface. They are presented in an intuitive format so that the user can refer to the final diagnosis.

[0520] Step 5: Recording and feedback

[0521] The server stores the final diagnosis made by the user (doctor) in a recording device. Specifically, it records the final diagnosis in an appropriate database table and collects feedback on the effectiveness of the AI ​​in the diagnostic process. This feedback is stored as training data for the AI ​​module and used to improve diagnostic accuracy in the future.

[0522] Step 6: Navigate the User Interface

[0523] The terminal provides an intuitive user interface. Specifically, the data entry screen is equipped with forms and templates that doctors can use to easily enter data. The diagnosis result display screen provides a graphical user interface (GUI) designed to make it easy to compare the AI's diagnosis results with those of doctors. The feedback input screen provides a form that doctors can use to easily enter feedback, allowing for the collection of information to improve the quality of diagnoses.

[0524] These steps enable the system to achieve accurate and efficient diagnosis through collaboration between doctors and AI, improving the accuracy of the final diagnosis.

[0525] (Application example 1)

[0526] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0527] Conventional medical diagnostic systems and factory quality inspection systems have limitations in accuracy and efficiency when integrating multiple diagnostic results. It has also been difficult to effectively integrate the results of AI and human diagnoses and reflect them in the final diagnosis. With the demand for improved diagnostic accuracy and efficiency in medical and factory settings, it is necessary to solve these issues.

[0528] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0529] In this invention, the server

[0530] means for receiving clinical diagnostic data of a patient;

[0531] means for storing the received clinical diagnostic data in a database;

[0532] means for analyzing the stored clinical diagnostic data;

[0533] A means for AI to generate diagnostic results based on the analysis results, and

[0534] A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface;

[0535] A means for making a final diagnosis based on the integrated diagnostic results;

[0536] means for receiving quality inspection data;

[0537] means for analyzing the received quality inspection data;

[0538] A method for AI and robots to generate quality diagnosis results based on the analysis results, and

[0539] A means to integrate the diagnostic results of AI and robots and display them on a user interface,

[0540] Includes means for providing audio feedback.

[0541] This will enable improvements in diagnostic accuracy in the medical field and quality inspection accuracy in factories.

[0542] "Patient" refers to a person who is the subject of diagnosis or treatment at a medical institution.

[0543] "Clinical diagnostic data" refers to a variety of biological, imaging, audiological, and mechanical data collected to assess a patient's health status.

[0544] A "database" refers to a system for efficiently storing, retrieving, and managing data.

[0545] "Analysis" refers to performing computations to detect patterns or anomalies in the received data.

[0546] "AI" refers to artificial intelligence that can automatically perform specific tasks with high accuracy.

[0547] "Diagnosis results" refer to the evaluation results of health status and quality obtained based on data analysis.

[0548] "Physician" refers to a professional with higher education who diagnoses and treats patients.

[0549] "User interface" refers to the interface through which a user interacts with a system.

[0550] "Final diagnosis" refers to the final assessment or conclusion obtained by integrating the diagnostic results of the AI ​​and the doctor (or robot).

[0551] "Quality Inspection Data" means data collected to evaluate the quality of a product.

[0552] "Robot" refers to a mechanical device for performing tasks automatically or semi-automatically.

[0553] "Voice feedback" refers to the function of providing analysis and diagnostic results to the user via voice.

[0554] This invention is a system that uses AI and robots to perform highly accurate diagnoses based on patient clinical diagnostic data and factory quality inspection data, and then integrates and displays the results. This system can improve the accuracy and efficiency of diagnoses in both the medical and manufacturing fields. Specific examples are described below.

[0555] System overview

[0556] The system is composed of a server, a terminal, and a user (a doctor or a diagnostician). The system includes the following means:

[0557] 1. Means for receiving patient clinical diagnosis data: A means for users to input patient blood test results, image diagnosis data, auscultation data, visual examination data, palpation data, etc. through a terminal.

[0558] 2. Database storage means: A means for efficiently storing received data in a database. Blood test results are stored as numerical data, and diagnostic imaging data is stored as image files, audio data, etc.

[0559] 3. Data analysis method: This is a method for analyzing the stored data and detecting outliers, etc. Software such as Python, Keras, OpenCV, and Pyttsx3 is used for this.

[0560] 4. AI diagnostic result generation means: This is the means by which AI generates diagnostic results based on the analysis results. The AI ​​model uses a pre-trained model.

[0561] 5. Diagnostic result integration means: A means for integrating the AI ​​diagnostic results with the doctor's diagnostic results and displaying them on the user interface.

[0562] 6. Final diagnostic tool: A tool for making a final diagnosis based on the integrated diagnostic results.

[0563] 7. Quality inspection data receiving means: This is the means by which the factory robot receives the quality inspection data of the product.

[0564] 8. Quality inspection data analysis means: A means for analyzing received quality inspection data.

[0565] 9. Means for generating diagnostic results using AI and robots: A means for AI and robots to generate diagnostic results based on analyzed quality inspection data.

[0566] 10. Audio feedback means: A means for providing the diagnostic results to the user by voice.

[0567] Specific examples

[0568] Specific examples in the medical field

[0569] Clinical data for a hypothetical patient (a 45-year-old male) is entered, including blood glucose levels (180 mg / dL), cholesterol levels (250 mg / dL), and chest X-ray images. This data is sent to a server and stored in a database. Data analysis is then performed, and the AI ​​detects abnormal values ​​and generates diagnoses of diabetes and pneumonia. These results are displayed to the doctor through a user interface, who then makes a final diagnosis based on the integrated information.

[0570] Specific example of factory quality inspection

[0571] Factory robots receive image data as product quality inspection data, which is then analyzed by AI. For example, an image analysis algorithm is applied to determine whether there are scratches on the surface of the product. The diagnostic results of the AI ​​and robot are integrated and displayed on the user interface. As a result, quality control personnel can make the final quality judgment.

[0572] Prompt Sentence Examples

[0573] Analyze the product image below and output the diagnostic results regarding the product quality.

[0574] Example: main("product_image.jpg", "quality_model.h5", "robot_measured_data")

[0575] The configuration and implementation of this system can significantly improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[0576] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0577] Step 1:

[0578] The user inputs the patient's clinical diagnostic data into the terminal, including blood test results, imaging diagnostic data, auscultation data, visual examination data, palpation data, etc. This data is collected through an input form or file upload.

[0579] Step 2:

[0580] The terminal transmits the entered clinical diagnostic data to a server over the Internet, and the data is transmitted in multiple formats (text, numbers, images, and audio).

[0581] Step 3:

[0582] The server stores the received clinical diagnostic data in a database: blood test results in a numerical database, diagnostic imaging data in an image database, and voice data in a voice database.

[0583] Step 4:

[0584] The server analyzes the stored data: numerical data is analyzed using outlier detection algorithms, image data is analyzed using image analysis models to detect lesions, and audio data is analyzed using audio analysis algorithms.

[0585] Step 5:

[0586] The server uses the analysis results to generate a diagnosis using AI, which uses a pre-trained generative AI model to evaluate the disease name and likelihood of the condition.

[0587] Step 6:

[0588] The server integrates the AI's diagnosis results with those of the doctor, comparing the diagnosis results entered by the doctor with those generated by the AI ​​and integrating them based on certain rules.

[0589] Step 7:

[0590] The server displays the integrated diagnostic results on a user interface, through which the user (doctor) can confirm the diagnostic results and make a final diagnosis.

[0591] Step 8:

[0592] The user enters the final diagnosis and the server stores the results in a database, including any feedback or additional comments.

[0593] Step 9:

[0594] The user inputs factory quality inspection data into the terminal, including image files and sensor data.

[0595] Step 10:

[0596] The terminal transmits the input quality inspection data to the server.

[0597] Step 11:

[0598] The server analyzes the received quality inspection data. Image data is analyzed using image analysis models to detect product defects, and robot measurement data is analyzed using anomaly detection algorithms.

[0599] Step 12:

[0600] The server then uses the AI ​​and robots to generate quality diagnosis results based on the analysis results. The AI ​​model evaluates the product's quality, and the robot measures the product's physical characteristics.

[0601] Step 13:

[0602] The server integrates the diagnostic results of the AI ​​and robot and displays them on a user interface. The user (quality control officer) can then check the quality diagnostic results through this interface and make a final decision.

[0603] Step 14:

[0604] The server provides the diagnosis results to the user through voice feedback, using a speech synthesis library such as Pyttsx3.

[0605] Taking such steps can improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[0606] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0607] The system according to the present invention receives and analyzes the clinical diagnostic data of patients, and based on the results, AI generates diagnostic results, which are then integrated with the doctor's diagnostic results and displayed. In addition, it incorporates an emotion engine that recognizes the user's emotions, and utilizes the doctor's emotional state in diagnostic support. Specific embodiments of this system are described below.

[0608] Data collection and input

[0609] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0610] Data transmission and storage

[0611] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored in a numerical database, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[0612] Data analysis and diagnostic preparation

[0613] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale.

[0614] Comparison of diagnoses by doctors and AI

[0615] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0616] Emotion Engine Monitoring

[0617] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[0618] Emotion-based feedback and warnings

[0619] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the doctor's emotional state. For example, if the doctor is feeling stressed, it displays a warning message urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, it provides additional diagnostic support information.

[0620] Recording and feedback

[0621] The server stores the final diagnosis made by the user (doctor) in a database. It also receives and stores feedback on the user's emotional state during the diagnostic process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[0622] User Interface

[0623] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[0624] Implementing specific examples

[0625] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show a blood glucose level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes the data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[0626] The present invention allows doctors to make diagnoses more accurately and quickly, significantly improving the accuracy and efficiency of diagnoses. Furthermore, by utilizing the doctor's emotional state as diagnostic support, the quality of diagnoses can be further improved.

[0627] The processing flow will be explained below.

[0628] Step 1:

[0629] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0630] Step 2:

[0631] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[0632] Step 3:

[0633] The server stores the received clinical diagnostic data in a database. Blood test results are stored in a numerical database, image diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text and audio data.

[0634] Step 4:

[0635] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[0636] Step 5:

[0637] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, it evaluates the possibility of diabetes, and if a chest X-ray image is taken, it evaluates the possibility of pneumonia.

[0638] Step 6:

[0639] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0640] Step 7:

[0641] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[0642] Step 8:

[0643] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[0644] Step 9:

[0645] The server receives the emotion data sent from the emotion engine and provides diagnostic assistance based on the doctor's emotional state. For example, if the doctor is feeling stressed, a warning message is displayed urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, additional diagnostic assistance information is provided.

[0646] Step 10:

[0647] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[0648] Step 11:

[0649] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[0650] Step 12:

[0651] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[0652] Example 2

[0653] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0654] Conventional clinical diagnostic systems lack the integration of doctor and AI diagnoses, and lack diagnostic support that takes into account the doctor's emotional state. As a result, diagnostic accuracy and doctor work efficiency may decrease, and an appropriate diagnosis may not be made. A system that solves these problems and improves diagnostic accuracy and efficiency is needed.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0656] In this invention, the server provides means for receiving clinical diagnostic data of patients, means for storing the received clinical diagnostic data in a database, means for analyzing the stored clinical diagnostic data, means for an AI to generate a diagnostic result based on the analysis result, means for integrating the AI's diagnostic result and the doctor's diagnostic result and displaying them on a user interface, means for recognizing the doctor's emotional state, means for providing feedback and warnings based on the emotional state, and means for making a final diagnosis based on the integrated diagnostic result, thereby improving the accuracy and efficiency of diagnosis and enabling appropriate diagnostic support that takes the doctor's emotional state into consideration.

[0657] "Patient clinical diagnostic data" includes medical information about the patient, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[0658] "Means for receiving" refers to a combination of hardware and software for electronically receiving patient clinical diagnostic data.

[0659] "Means for storing in a database" refers to a storage device and associated management software for structuring and archiving received clinical diagnostic data.

[0660] "Means for analysis" refers to algorithms and software for computer-based analysis of stored clinical diagnostic data to detect outliers and specific patterns.

[0661] "Means for artificial intelligence to generate diagnostic results" refers to software that uses machine learning models to evaluate diagnostic possibilities based on the analysis results and generates diagnostic results based on that.

[0662] "Means for integrating and displaying on a user interface" refers to software and hardware that compares and integrates the AI ​​diagnostic results with the doctor's diagnostic results and provides a user interface for visually displaying that information.

[0663] "Means for recognizing the doctor's emotional state" refers to sensors and analysis software for analyzing the doctor's voice and facial expressions to determine his or her emotional state.

[0664] "Means for providing feedback and warnings" refers to software and devices for generating and presenting appropriate advice and warnings in response to a perceived emotional state.

[0665] "Means for making a final diagnosis" refers to the process and tools that allow a physician to determine a final diagnosis using the integrated diagnostic results and feedback information.

[0666] The system according to the present invention receives and analyzes a patient's clinical diagnostic data, and then uses artificial intelligence to generate a diagnostic result based on the analysis results, which is then integrated with the doctor's diagnostic result and displayed. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system is characterized by utilizing the doctor's emotional state in diagnostic support. A specific embodiment of this system is described below.

[0667] Data collection and input

[0668] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. The terminal is provided with a form for data input, and for example, the user can enter a blood glucose level of 180 mg / dL, a cholesterol level of 250 mg / dL, and upload a chest X-ray image.

[0669] Data transmission and storage

[0670] The device structures the input clinical diagnostic data and sends it to the server using an HTTP request. The server receives this data in the appropriate format and stores it in a relational database, such as a MySQL database. Numeric data is stored in the numeric database, image data in the image database, and auscultation, visual inspection, and palpation data in the text database and audio database.

[0671] Data analysis and diagnostic preparation

[0672] The server analyzes the stored data. For example, blood test results are analyzed using Python's numerical computing libraries (NumPy and Pandas) to detect abnormalities. Chest X-ray images are analyzed to detect abnormalities using deep learning frameworks such as TensorFlow and PyTorch. The analyzed data is passed to an AI module, which uses a generative AI model to evaluate diagnostic possibilities and generate a list of diagnoses and their rationales.

[0673] Comparison of diagnoses by doctors and AI

[0674] The server compares the AI-generated diagnosis with the diagnosis entered by the user (doctor), uses a comparison algorithm to identify matches and differences, adds complementary information (e.g., relevant research data and statistical data), and generates a unified diagnosis, which is ultimately presented to the doctor.

[0675] Emotion Engine Monitoring

[0676] While the user (doctor) is checking the diagnosis results, the device sends data to the emotion engine using a camera and microphone. The emotion engine analyzes facial expressions using libraries such as OpenCV and Dlib, and voice tone and pitch using a voice analysis library, to recognize the doctor's emotional state (stress level or lack of attention) in real time.

[0677] Emotion-based feedback and warnings

[0678] The server receives the emotional data sent from the emotion engine and generates appropriate feedback based on the doctor's emotional state. For example, if the doctor is feeling stressed, it sends a warning message to the device in real time via WebSocket, prompting the doctor to reconfirm the diagnosis. Furthermore, if the doctor's emotional state is determined to have a significant impact on the diagnosis, it provides AI-generated diagnostic support information.

[0679] Recording and feedback

[0680] The server stores the doctor's final diagnosis in a database. It also receives and analyzes feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support. This feedback is used for future AI training to improve diagnostic accuracy.

[0681] User Interface

[0682] The device provides a user interface that allows intuitive data entry, confirmation of diagnosis results, and feedback. The data entry screen displays input forms and templates, allowing doctors to quickly enter data. The diagnosis result display screen also provides a GUI that allows easy visual comparison of the AI's diagnosis results with those of the doctor. The feedback entry screen displays a simple input form, allowing doctors to enter feedback in a short amount of time.

[0683] Hypothetical example

[0684] Let's take the hypothetical case of a 45-year-old male patient. His blood test results show a blood sugar level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is uploaded. The device sends this data to the server, which analyzes it. The AI ​​module uses this data to evaluate the possibility of diabetes and pneumonia and generate a diagnosis. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[0685] Prompt Sentence Examples

[0686] The following prompt sentences can be input to a generative AI model to explain the specific functions of the system:

[0687] A 45-year-old male patient's blood test results are entered: blood sugar 180 mg / dL (abnormal) and cholesterol 250 mg / dL (high), and a chest x-ray is uploaded. Explain how the system will operate based on this data.

[0688] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0689] Program processing flow

[0690] Step 1: Data collection and entry

[0691] A user (doctor) needs to input a patient's clinical diagnostic data into a terminal. Input items include blood test results, image diagnostic data, auscultation data, visual examination data, and palpation data. For example, a blood glucose level of 180 mg / dL and a cholesterol level of 250 mg / dL are input as numerical values, and a chest X-ray image is uploaded as a file. This data is structured using an input form on the terminal.

[0692] Input: Patient clinical diagnostic data (numerical data, image data, etc.)

[0693] Output: Structured data format

[0694] Step 2: Send and store data

[0695] The device sends structured clinical diagnostic data to the server using HTTP requests. The server receives this data and stores it in a relational database, such as a MySQL database. Numeric data is stored in a numeric database, image data in an image database, and auscultation, visual inspection, and palpation data in a text database or audio database.

[0696] Input: Structured data format

[0697] Output: Data stored in the database

[0698] Step 3: Data analysis and diagnostic preparation

[0699] The server analyzes the stored data. Blood test results are analyzed using Python's numerical calculation libraries (NumPy, Pandas) to detect abnormal values. Chest X-ray images are analyzed using TensorFlow and PyTorch to detect abnormal areas. The analysis results are passed to an AI module, which uses a generative AI model to evaluate diagnostic feasibility and generate a diagnosis list and rationale.

[0700] Input: Data stored in a database

[0701] Output: Analysis results, diagnostic list

[0702] Step 4: Comparing diagnoses between doctors and AI

[0703] The server compares the diagnosis results generated by the generative AI model with those entered by the user (doctor). It uses a comparison algorithm to identify similarities and differences. It adds relevant supplementary information (related research data, statistical data, etc.) to generate a final integrated diagnosis.

[0704] Input: Analysis results, diagnosis list, doctor's diagnosis

[0705] Output: Integrated diagnostic results

[0706] Step 5: Emotion Engine Monitoring

[0707] While the user (doctor) is checking the diagnosis results, the device uses a camera and microphone to send emotional data to the emotion engine, which then analyzes facial expressions using OpenCV and Dlib, and voice tone and pitch using a voice analysis library to recognize the doctor's emotional state.

[0708] Input: Audio and video data of the doctor confirming the diagnosis

[0709] Output: Doctor's emotional state data (stress level, attention deficit)

[0710] Step 6: Emotion-based feedback and warnings

[0711] The server receives emotional state data sent from the emotion engine and generates feedback and warning messages based on that data. For example, if a doctor is feeling stressed, a warning message urging reconsideration is sent to the device via WebSocket and displayed.

[0712] Input: Emotional state data

[0713] Output: Feedback and warning messages

[0714] Step 7: Recording and feedback

[0715] The server stores the doctor's final diagnosis results in a database. It also receives feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support, and stores this information in the database. This feedback information is used for further learning by the AI, contributing to improving diagnostic accuracy.

[0716] Input: Final diagnosis results, feedback information

[0717] Output: Feedback information stored in a database

[0718] Step 8: User Interface

[0719] The device provides an intuitive user interface through which users (doctors) can input data, check diagnosis results, and provide feedback. The data input screen displays specific forms and templates. The diagnosis result display screen displays the AI's diagnosis results and the doctor's diagnosis results in an easy-to-read format. The feedback input screen provides a simple input form.

[0720] Input: Data entry, diagnostic result confirmation, feedback entry

[0721] Output: User feedback, diagnostic results

[0722] (Application example 2)

[0723] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Conventional maintenance and fault diagnosis systems for factory robots only use analytical results based on the robot's sensor data, and do not reflect the emotional state of the maintenance staff. As a result, early detection of robot abnormalities or breakdowns is difficult due to mistakes and delayed judgment caused by stress or fatigue.

[0725] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving sensor data from the robot, means for analyzing the saved sensor data, means for an AI to evaluate the possibility of a failure based on the analysis results and generate a diagnosis result, means for receiving emotion data using an emotion engine that recognizes the emotional state of the maintenance staff, and means for providing diagnosis support based on the emotional state. This enables failure diagnosis support that takes into account the emotional state of the staff as well as the sensor data.

[0726] "Patient clinical diagnostic data" refers to medical data collected for diagnosis, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[0727] A "numerical database" refers to a database that stores numerical data such as blood test results and sensor data.

[0728] "Image database" refers to a database that stores image data such as X-ray images and CT scans.

[0729] "Analysis" refers to the data processing and evaluation process that detects abnormal values ​​and abnormal areas based on stored clinical diagnostic data and sensor data.

[0730] "AI generates diagnostic results" refers to the process of creating a diagnostic list and its rationale using a machine learning algorithm based on analyzed data.

[0731] "Doctor's diagnosis result" refers to the diagnostic judgment and conclusion made by the doctor regarding the condition of the patient or robot.

[0732] "User Interface" refers to the system's interactive screen through which a user (physician or maintenance technician) can input data, review diagnostic results, and provide feedback.

[0733] "Emotion engine" refers to technology that uses voice analysis and facial expression recognition technology to identify a user's emotional state and assess their stress and fatigue levels.

[0734] "Emotion Data" refers to data regarding emotional states (e.g., stress, joy, sadness, etc.) recognized by the emotion engine.

[0735] "Diagnostic support" refers to providing additional information and warning messages during the diagnostic process to help doctors and maintenance personnel make better decisions.

[0736] "Feedback" refers to information entered as evaluations and impressions of diagnostic results and work processes, which are used as learning data for the system.

[0737] In the present invention, a plurality of hardware and software components are integrated to provide a maintenance and fault diagnosis support system for a factory robot. Specific embodiments will be described below.

[0738] Data collection and input

[0739] The user (maintenance staff) inputs the robot's sensor data into a dedicated terminal. This data includes various sensor data such as temperature, vibration, and current. The terminal collects this data in real time and sends it to a server. A general-purpose PC or smart device is suitable as the terminal to be used.

[0740] Data transmission and storage

[0741] The device sends the input sensor data to the server, which stores the received data in a numerical database and an image database. For example, temperature and vibration data are stored in the numerical database and analyzed as needed.

[0742] Data analysis and fault diagnosis

[0743] The server analyzes the stored sensor data using a data processing module written in Python. The analysis results are passed to an AI module (built using TensorFlow / Keras), which evaluates the possibility of a fault and generates a diagnosis.

[0744] Display and compare diagnostic results

[0745] The server displays the diagnostic results generated by the AI ​​on a user interface, which provides an intuitive graphical user interface (GUI) that allows users (maintenance personnel) to check the diagnostic results.

[0746] Emotion Engine Monitoring

[0747] The device uses an emotion engine to recognize the emotional state of the person checking the diagnosis results. The emotion engine uses facial recognition technology (OpenCV) and voice analysis technology to detect stress and fatigue of the person.

[0748] Emotion-based feedback and warnings

[0749] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the person in charge is feeling stressed, the server displays a warning message urging the person to reconfirm the diagnosis result. In addition, if the person in charge's emotional state affects the diagnosis, the server provides additional diagnostic support information.

[0750] Recording and feedback

[0751] The server stores the user's final diagnosis results in a numerical database. It also receives and stores feedback on the user's emotional state during the diagnosis process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI, helping to improve its diagnostic accuracy in the future.

[0752] Implementing specific examples

[0753] In a factory, a maintenance worker performs a routine inspection of a factory robot. The worker operates a smart device to collect sensor data (such as temperature and vibration) in real time. This data is then sent to a server, where it is analyzed and displayed along with the results of a diagnosis made by an AI module. Meanwhile, an emotion engine recognizes the worker's stress level and provides feedback to encourage relaxation as needed.

[0754] (Example of prompts for a generative AI model)

[0755] "Detect stress levels in factory maintenance personnel and generate prompts for an assistant system that provides feedback to encourage breaks."

[0756] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0757] Step 1:

[0758] The user (maintenance technician) begins maintenance work on a factory robot. First, various sensor data such as temperature, vibration, and current are collected from sensors connected to a dedicated terminal and input into the terminal. At this time, the data from each sensor is displayed in real time on the terminal's UI.

[0759] Step 2:

[0760] The device sends the collected sensor data to the server. The sensor data includes temperature values ​​measured every 0.1 seconds, vibration frequency spectrum, and current time series data. The device then converts the data into a specific format (e.g., JSON format) and sends it to the server via the network.

[0761] Step 3:

[0762] The server sequentially stores the received sensor data in a numeric database and an image database. When storing the data in the database, the data is separated by sensor type and stored in the appropriate field. For example, temperature data is stored in the "temperature" field, and vibration data is stored in the "vibration" field.

[0763] Step 4:

[0764] The server analyzes the stored sensor data. Specifically, it detects abnormal values ​​in temperature data, analyzes the frequency of vibration data, and recognizes patterns in current data. From this analysis, an anomaly detection algorithm is applied to extract abnormal values ​​and patterns. The analysis results include peak temperature values, the ratio of high-frequency components in vibration, and abnormal current peaks.

[0765] Step 5:

[0766] The server passes the analysis results to an AI module (using TensorFlow / Keras), which then generates a diagnosis. The AI ​​module evaluates current sensor data based on past learning data and predicts the likelihood of robot failure. The diagnosis results include the possibility of wear on specific parts, failure due to overheating, and damage due to abnormal vibration.

[0767] Step 6:

[0768] The server displays the diagnostic results generated by the AI ​​on a user interface. The user interface uses a GUI to visually display the diagnostic results, allowing the user to intuitively understand them. Specifically, it displays an overview of the diagnostic results, highlights the areas where abnormalities have occurred, and recommends actions (e.g., part replacement, adjustment).

[0769] Step 7:

[0770] The device uses an emotion engine to recognize the user's emotional state while checking the diagnosis results. The emotion engine captures the user's face with the device's built-in camera and analyzes their emotions using a facial expression recognition algorithm (OpenCV). This analysis determines whether the user is feeling stressed or relaxed.

[0771] Step 8:

[0772] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the emotion analysis detects that the user is feeling stressed, the server displays a warning message on the user interface. This message may include a message to reconfirm the diagnosis result or to take a break.

[0773] Step 9:

[0774] The server stores the final diagnosis results and emotion data provided by the user in a numerical database. The stored data will be used as training data to improve the accuracy of the diagnosis. In addition, the user can input feedback on the final diagnosis results via the user interface, and this feedback will also be stored in the database.

[0775] Step 10:

[0776] The server analyzes the stored final diagnosis results and emotion data and uses them to retrain the AI ​​model, a process that improves future diagnosis accuracy and overall system performance.

[0777] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0778] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0779] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0780] [Third embodiment]

[0781] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0782] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0783] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0784] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0785] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0786] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0787] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0788] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0789] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0790] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0791] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0792] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0793] The system of the present invention improves diagnostic accuracy by having a doctor and an AI simultaneously perform diagnoses based on the patient's clinical diagnostic data and then displaying the integrated results. A specific example of this system is described below.

[0794] Data collection and input

[0795] The user (doctor) inputs the patient's clinical diagnostic data. This data includes blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data. For example, blood test results include blood glucose and cholesterol levels, and in addition, chest X-ray images and heart sound data are also input.

[0796] Data transmission and storage

[0797] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored as numerical data, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[0798] Data analysis and diagnostic preparation

[0799] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the possibility of a diagnosis based on this analysis data and generates a list of diagnoses and their rationale.

[0800] Comparison of diagnoses by doctors and AI

[0801] The server compares the diagnostic results generated by the AI ​​with the diagnostic data entered by the user (doctor). Here, the AI ​​diagnostic results, the doctor's diagnostic results, and supplementary information are integrated to generate data to be displayed on the user interface. This allows the user (doctor) to make a final diagnosis while referring to the AI ​​diagnostic results.

[0802] Recording and feedback

[0803] The server stores the final diagnosis made by the user (doctor) in a database. It also receives feedback on how helpful the AI ​​diagnosis was during the diagnostic process. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[0804] User Interface

[0805] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[0806] Implementing specific examples

[0807] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. This is then used to make the final diagnosis and provide feedback.

[0808] The present invention enables doctors to make diagnoses more accurately and quickly, and can significantly improve the accuracy and efficiency of diagnosis.

[0809] The processing flow will be explained below.

[0810] Step 1:

[0811] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0812] Step 2:

[0813] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[0814] Step 3:

[0815] The server stores the received clinical diagnostic data in a database: blood test results are stored in a numerical database, imaging diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored in an appropriate format.

[0816] Step 4:

[0817] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[0818] Step 5:

[0819] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, the module evaluates the possibility of diabetes, and if a chest X-ray image is used, the module evaluates the possibility of pneumonia.

[0820] Step 6:

[0821] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0822] Step 7:

[0823] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[0824] Step 8:

[0825] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[0826] Step 9:

[0827] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[0828] Step 10:

[0829] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[0830] Example 1

[0831] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0832] In modern medical settings, we often rely solely on doctors' diagnoses, which poses challenges in diagnostic accuracy and efficiency. In particular, integrating multiple diagnostic data to make a comprehensive diagnosis is time-consuming and laborious. Furthermore, feedback during the diagnostic process is not adequately collected and utilized, creating a need for the development of a diagnostic support system using artificial intelligence.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0834] In this invention, the server includes means for receiving patient's biological information, means for storing the received biological information in a recording device, means for analyzing the stored biological information, means for an AI module to generate a diagnosis result based on the analysis result, means for integrating the AI ​​module's diagnosis result with the doctor's diagnosis result and displaying it on a display device, means for making a final diagnosis based on the integrated diagnosis result, and means for collecting feedback during the diagnostic process, thereby enabling the accuracy and efficiency of diagnosis to be improved.

[0835] "Patient's biological information" refers to data indicating the patient's health condition, and includes blood test results, diagnostic imaging data, auscultation data, visual examination data, palpation data, and the like.

[0836] The "receiving means" refers to a means for inputting the patient's vital signs into the system, such as a data input interface or a data transfer protocol.

[0837] "Recording device" refers to a device for storing received patient biometric information in digital form, such as a database or storage system.

[0838] The "analyzing means" refers to a means for carrying out data analysis necessary for making a medical diagnosis based on the stored biological information, and includes a numerical analysis module, an image analysis module, and the like.

[0839] An "artificial intelligence module" is a device or software that generates diagnostic results based on analyzed biometric information, and includes machine learning algorithms and AI models.

[0840] "Means for integrating and displaying" refers to a means for comparing and integrating the diagnostic results generated by the AI ​​module with those of the doctor and visually presenting them to the user, and refers to a display device or graphical user interface (GUI).

[0841] The "means for making a final diagnosis" refers to the means by which a doctor determines a final diagnosis based on the integrated diagnostic results, and includes a diagnostic support system and an interface that supports the doctor's judgment.

[0842] "Means for collecting feedback" refers to means for receiving, analyzing, and storing feedback provided during the diagnostic process, and refers to a feedback input form and a data analysis module.

[0843] The system of this invention collects patient biometric information, and doctors and artificial intelligence jointly perform diagnoses, and the results are integrated and displayed to improve the accuracy of diagnoses. Specific embodiments of the present invention are described below.

[0844] Data collection and input

[0845] The user (doctor) enters the patient's biometric information into the system, which includes the following data:

[0846] Blood test results (e.g., blood sugar 180 mg / dL, cholesterol 250 mg / dL)

[0847] Imaging data (e.g., chest x-rays)

[0848] Auscultation data (e.g., heart sound data)

[0849] Visual inspection data

[0850] Palpation data

[0851] The user inputs data into a form or template on the system's user interface and uploads image and audio data as files.

[0852] Data transmission and storage

[0853] The device sends the input biometric information to a server using a secure communication protocol such as HTTPS.

[0854] The server stores the received biometric information in a recording device as follows:

[0855] Storing blood test results in a numerical database

[0856] Storing diagnostic imaging data in an image database

[0857] Save auscultation, visual inspection, and palpation data as text or audio data

[0858] Data analysis and diagnostic preparation

[0859] The server passes the data stored in the recording device to the analysis module, which includes the following specific processes:

[0860] Blood test results are analyzed by a numerical analysis module and compared with defined normal ranges to identify abnormal values.

[0861] The diagnostic imaging data is analyzed by the image analysis module to identify abnormalities.

[0862] The results of these analyses are passed to an artificial intelligence module, which evaluates the possibility of diagnosis based on the analyzed data and generates a list of diagnoses and their rationale.

[0863] Comparison of diagnoses by doctors and AI

[0864] The server compares the diagnosis results generated by the AI ​​module with the diagnosis data entered by the user (doctor). The specific process includes:

[0865] Compare AI diagnosis results with doctor diagnosis results

[0866] Identify commonalities and differences to generate integrated diagnostic results

[0867] As supplementary information, the basis for the AI ​​diagnosis and the doctor's reasons for the diagnosis will also be integrated.

[0868] The integrated diagnostic results are generated as data to be displayed on a user interface.

[0869] Recording and feedback

[0870] The server stores the final diagnosis made by the user (doctor) in the recording device. Specific operations include:

[0871] Record the final diagnostic results in the appropriate database table

[0872] Collect feedback on the effectiveness of the AI ​​in the diagnostic process and store it as training data for the AI ​​module.

[0873] User Interface

[0874] The terminal provides a user interface with the following features:

[0875] Data entry screens: Forms and templates that allow doctors to easily enter data

[0876] Diagnostic result display screen: A graphical user interface (GUI) that makes it easy to compare AI diagnostic results with those of a doctor.

[0877] Feedback input screen: A form where doctors can enter brief feedback

[0878] Implementing specific examples

[0879] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which then stores it in a recording device. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made.

[0880] Prompt Sentence Examples

[0881] Here is an example of a prompt you can enter into the system:

[0882] "AI diagnosis will be performed based on the patient's clinical data. Please enter the following data.

[0883] Blood test results:

[0884] Blood glucose level: 180 mg / dL

[0885] Cholesterol level: 250 mg / dL

[0886] Imaging data: Chest X-ray image (file upload)

[0887] Generate your results and compare them with your doctor's diagnosis. View the results and collect feedback."

[0888] Using this prompt sentence, the system can make an accurate and efficient diagnosis, which helps support doctors in making diagnoses.

[0889] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0890] Step 1: Data collection and input

[0891] The user (doctor) inputs the patient's biometric information into the system. This biometric information includes blood test results (e.g., blood glucose level 180 mg / dL, cholesterol level 250 mg / dL), diagnostic imaging data (e.g., chest X-ray image), auscultation data (e.g., heart sound data), visual examination data, and palpation data. The user (doctor) inputs this data using a form or template provided on the system's user interface and uploads the image and audio data as files. The input data is used as is in the next step.

[0892] Step 2: Send and store data

[0893] The device sends the entered biometric information to the server. This data transmission is done using a secure communication protocol such as HTTPS. The transmitted data is then stored in the appropriate database on the server. Specifically, blood test results are stored in a numerical database, diagnostic imaging data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text or audio data. This makes the biometric information available for analysis.

[0894] Step 3: Prepare for data analysis and diagnosis

[0895] The server passes the data stored in the recording device to the analysis module. Specifically, blood test results are first analyzed by a numerical analysis module. Abnormal values ​​are identified by comparison with normal ranges, and this information is passed to the next step. Image diagnostic data is analyzed by an image analysis module, and abnormal areas are identified. These analysis results are passed to an artificial intelligence module, which generates a diagnosis list and diagnostic rationale.

[0896] Step 4: Comparing the doctor's and AI's diagnoses

[0897] The server compares the diagnosis results generated by the AI ​​module with the diagnostic data entered by the user (doctor). Specifically, the AI ​​diagnosis results are compared with the doctor's diagnosis results, and similarities and differences are identified. The integrated diagnosis results, along with supplementary information (e.g., the basis of the AI ​​diagnosis and the doctor's reasons for the diagnosis), are generated as data to be displayed on the user interface. They are presented in an intuitive format so that the user can refer to the final diagnosis.

[0898] Step 5: Recording and feedback

[0899] The server stores the final diagnosis made by the user (doctor) in a recording device. Specifically, it records the final diagnosis in an appropriate database table and collects feedback on the effectiveness of the AI ​​in the diagnostic process. This feedback is stored as training data for the AI ​​module and used to improve diagnostic accuracy in the future.

[0900] Step 6: Navigate the User Interface

[0901] The terminal provides an intuitive user interface. Specifically, the data entry screen is equipped with forms and templates that doctors can use to easily enter data. The diagnosis result display screen provides a graphical user interface (GUI) designed to make it easy to compare the AI's diagnosis results with those of doctors. The feedback input screen provides a form that doctors can use to easily enter feedback, allowing for the collection of information to improve the quality of diagnoses.

[0902] These steps enable the system to achieve accurate and efficient diagnosis through collaboration between doctors and AI, improving the accuracy of the final diagnosis.

[0903] (Application example 1)

[0904] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0905] Conventional medical diagnostic systems and factory quality inspection systems have limitations in accuracy and efficiency when integrating multiple diagnostic results. It has also been difficult to effectively integrate the results of AI and human diagnoses and reflect them in the final diagnosis. With the demand for improved diagnostic accuracy and efficiency in medical and factory settings, it is necessary to solve these issues.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0907] In this invention, the server

[0908] means for receiving clinical diagnostic data of a patient;

[0909] means for storing the received clinical diagnostic data in a database;

[0910] means for analyzing the stored clinical diagnostic data;

[0911] A means for AI to generate diagnostic results based on the analysis results, and

[0912] A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface;

[0913] A means for making a final diagnosis based on the integrated diagnostic results;

[0914] means for receiving quality inspection data;

[0915] means for analyzing the received quality inspection data;

[0916] A method for AI and robots to generate quality diagnosis results based on the analysis results, and

[0917] A means to integrate the diagnostic results of AI and robots and display them on a user interface,

[0918] Includes means for providing audio feedback.

[0919] This will enable improvements in diagnostic accuracy in the medical field and quality inspection accuracy in factories.

[0920] "Patient" refers to a person who is the subject of diagnosis or treatment at a medical institution.

[0921] "Clinical diagnostic data" refers to a variety of biological, imaging, audiological, and mechanical data collected to assess a patient's health status.

[0922] A "database" refers to a system for efficiently storing, retrieving, and managing data.

[0923] "Analysis" refers to performing computations to detect patterns or anomalies in the received data.

[0924] "AI" refers to artificial intelligence that can automatically perform specific tasks with high accuracy.

[0925] "Diagnosis results" refer to the evaluation results of health status and quality obtained based on data analysis.

[0926] "Physician" refers to a professional with higher education who diagnoses and treats patients.

[0927] "User interface" refers to the interface through which a user interacts with a system.

[0928] "Final diagnosis" refers to the final assessment or conclusion obtained by integrating the diagnostic results of the AI ​​and the doctor (or robot).

[0929] "Quality Inspection Data" means data collected to evaluate the quality of a product.

[0930] "Robot" refers to a mechanical device for performing tasks automatically or semi-automatically.

[0931] "Voice feedback" refers to the function of providing analysis and diagnostic results to the user via voice.

[0932] This invention is a system that uses AI and robots to perform highly accurate diagnoses based on patient clinical diagnostic data and factory quality inspection data, and then integrates and displays the results. This system can improve the accuracy and efficiency of diagnoses in both the medical and manufacturing fields. Specific examples are described below.

[0933] System overview

[0934] The system is composed of a server, a terminal, and a user (a doctor or a diagnostician). The system includes the following means:

[0935] 1. Means for receiving patient clinical diagnosis data: A means for users to input patient blood test results, image diagnosis data, auscultation data, visual examination data, palpation data, etc. through a terminal.

[0936] 2. Database storage means: A means for efficiently storing received data in a database. Blood test results are stored as numerical data, and diagnostic imaging data is stored as image files, audio data, etc.

[0937] 3. Data analysis method: This is a method for analyzing the stored data and detecting outliers, etc. Software such as Python, Keras, OpenCV, and Pyttsx3 is used for this.

[0938] 4. AI diagnostic result generation means: This is the means by which AI generates diagnostic results based on the analysis results. The AI ​​model uses a pre-trained model.

[0939] 5. Diagnostic result integration means: A means for integrating the AI ​​diagnostic results with the doctor's diagnostic results and displaying them on the user interface.

[0940] 6. Final diagnostic tool: A tool for making a final diagnosis based on the integrated diagnostic results.

[0941] 7. Quality inspection data receiving means: This is the means by which the factory robot receives the quality inspection data of the product.

[0942] 8. Quality inspection data analysis means: A means for analyzing received quality inspection data.

[0943] 9. Means for generating diagnostic results using AI and robots: A means for AI and robots to generate diagnostic results based on analyzed quality inspection data.

[0944] 10. Audio feedback means: A means for providing the diagnostic results to the user by voice.

[0945] Specific examples

[0946] Specific examples in the medical field

[0947] Clinical data for a hypothetical patient (a 45-year-old male) is entered, including blood glucose levels (180 mg / dL), cholesterol levels (250 mg / dL), and chest X-ray images. This data is sent to a server and stored in a database. Data analysis is then performed, and the AI ​​detects abnormal values ​​and generates diagnoses of diabetes and pneumonia. These results are displayed to the doctor through a user interface, who then makes a final diagnosis based on the integrated information.

[0948] Specific example of factory quality inspection

[0949] Factory robots receive image data as product quality inspection data, which is then analyzed by AI. For example, an image analysis algorithm is applied to determine whether there are scratches on the surface of the product. The diagnostic results of the AI ​​and robot are integrated and displayed on the user interface. As a result, quality control personnel can make the final quality judgment.

[0950] Prompt Sentence Examples

[0951] Analyze the product image below and output the diagnostic results regarding the product quality.

[0952] Example: main("product_image.jpg", "quality_model.h5", "robot_measured_data")

[0953] The configuration and implementation of this system can significantly improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[0954] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0955] Step 1:

[0956] The user inputs the patient's clinical diagnostic data into the terminal, including blood test results, imaging diagnostic data, auscultation data, visual examination data, palpation data, etc. This data is collected through an input form or file upload.

[0957] Step 2:

[0958] The terminal transmits the entered clinical diagnostic data to a server over the Internet, and the data is transmitted in multiple formats (text, numbers, images, and audio).

[0959] Step 3:

[0960] The server stores the received clinical diagnostic data in a database: blood test results in a numerical database, diagnostic imaging data in an image database, and voice data in a voice database.

[0961] Step 4:

[0962] The server analyzes the stored data: numerical data is analyzed using outlier detection algorithms, image data is analyzed using image analysis models to detect lesions, and audio data is analyzed using audio analysis algorithms.

[0963] Step 5:

[0964] The server uses the analysis results to generate a diagnosis using AI, which uses a pre-trained generative AI model to evaluate the disease name and likelihood of the condition.

[0965] Step 6:

[0966] The server integrates the AI's diagnosis results with those of the doctor, comparing the diagnosis results entered by the doctor with those generated by the AI ​​and integrating them based on certain rules.

[0967] Step 7:

[0968] The server displays the integrated diagnostic results on a user interface, through which the user (doctor) can confirm the diagnostic results and make a final diagnosis.

[0969] Step 8:

[0970] The user enters the final diagnosis and the server stores the results in a database, including any feedback or additional comments.

[0971] Step 9:

[0972] The user inputs factory quality inspection data into the terminal, including image files and sensor data.

[0973] Step 10:

[0974] The terminal transmits the input quality inspection data to the server.

[0975] Step 11:

[0976] The server analyzes the received quality inspection data. Image data is analyzed using image analysis models to detect product defects, and robot measurement data is analyzed using anomaly detection algorithms.

[0977] Step 12:

[0978] The server then uses the AI ​​and robots to generate quality diagnosis results based on the analysis results. The AI ​​model evaluates the product's quality, and the robot measures the product's physical characteristics.

[0979] Step 13:

[0980] The server integrates the diagnostic results of the AI ​​and robot and displays them on a user interface. The user (quality control officer) can then check the quality diagnostic results through this interface and make a final decision.

[0981] Step 14:

[0982] The server provides the diagnosis results to the user through voice feedback, using a speech synthesis library such as Pyttsx3.

[0983] Taking such steps can improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[0984] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0985] The system according to the present invention receives and analyzes the clinical diagnostic data of patients, and based on the results, AI generates diagnostic results, which are then integrated with the doctor's diagnostic results and displayed. In addition, it incorporates an emotion engine that recognizes the user's emotions, and utilizes the doctor's emotional state in diagnostic support. Specific embodiments of this system are described below.

[0986] Data collection and input

[0987] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[0988] Data transmission and storage

[0989] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored in a numerical database, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[0990] Data analysis and diagnostic preparation

[0991] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale.

[0992] Comparison of diagnoses by doctors and AI

[0993] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[0994] Emotion Engine Monitoring

[0995] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[0996] Emotion-based feedback and warnings

[0997] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the doctor's emotional state. For example, if the doctor is feeling stressed, it displays a warning message urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, it provides additional diagnostic support information.

[0998] Recording and feedback

[0999] The server stores the final diagnosis made by the user (doctor) in a database. It also receives and stores feedback on the user's emotional state during the diagnostic process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[1000] User Interface

[1001] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[1002] Implementing specific examples

[1003] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show a blood glucose level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes the data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[1004] The present invention allows doctors to make diagnoses more accurately and quickly, significantly improving the accuracy and efficiency of diagnoses. Furthermore, by utilizing the doctor's emotional state as diagnostic support, the quality of diagnoses can be further improved.

[1005] The processing flow will be explained below.

[1006] Step 1:

[1007] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[1008] Step 2:

[1009] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[1010] Step 3:

[1011] The server stores the received clinical diagnostic data in a database. Blood test results are stored in a numerical database, image diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text and audio data.

[1012] Step 4:

[1013] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[1014] Step 5:

[1015] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, it evaluates the possibility of diabetes, and if a chest X-ray image is taken, it evaluates the possibility of pneumonia.

[1016] Step 6:

[1017] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[1018] Step 7:

[1019] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[1020] Step 8:

[1021] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[1022] Step 9:

[1023] The server receives the emotion data sent from the emotion engine and provides diagnostic assistance based on the doctor's emotional state. For example, if the doctor is feeling stressed, a warning message is displayed urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, additional diagnostic assistance information is provided.

[1024] Step 10:

[1025] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[1026] Step 11:

[1027] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[1028] Step 12:

[1029] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[1030] Example 2

[1031] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1032] Conventional clinical diagnostic systems lack the integration of doctor and AI diagnoses, and lack diagnostic support that takes into account the doctor's emotional state. As a result, diagnostic accuracy and doctor work efficiency may decrease, and an appropriate diagnosis may not be made. A system that solves these problems and improves diagnostic accuracy and efficiency is needed.

[1033] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1034] In this invention, the server provides means for receiving clinical diagnostic data of patients, means for storing the received clinical diagnostic data in a database, means for analyzing the stored clinical diagnostic data, means for an AI to generate a diagnostic result based on the analysis result, means for integrating the AI's diagnostic result and the doctor's diagnostic result and displaying them on a user interface, means for recognizing the doctor's emotional state, means for providing feedback and warnings based on the emotional state, and means for making a final diagnosis based on the integrated diagnostic result, thereby improving the accuracy and efficiency of diagnosis and enabling appropriate diagnostic support that takes the doctor's emotional state into consideration.

[1035] "Patient clinical diagnostic data" includes medical information about the patient, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[1036] "Means for receiving" refers to a combination of hardware and software for electronically receiving patient clinical diagnostic data.

[1037] "Means for storing in a database" refers to a storage device and associated management software for structuring and archiving received clinical diagnostic data.

[1038] "Means for analysis" refers to algorithms and software for computer-based analysis of stored clinical diagnostic data to detect outliers and specific patterns.

[1039] "Means for artificial intelligence to generate diagnostic results" refers to software that uses machine learning models to evaluate diagnostic possibilities based on the analysis results and generates diagnostic results based on that.

[1040] "Means for integrating and displaying on a user interface" refers to software and hardware that compares and integrates the AI ​​diagnostic results with the doctor's diagnostic results and provides a user interface for visually displaying that information.

[1041] "Means for recognizing the doctor's emotional state" refers to sensors and analysis software for analyzing the doctor's voice and facial expressions to determine his or her emotional state.

[1042] "Means for providing feedback and warnings" refers to software and devices for generating and presenting appropriate advice and warnings in response to a perceived emotional state.

[1043] "Means for making a final diagnosis" refers to the process and tools that allow a physician to determine a final diagnosis using the integrated diagnostic results and feedback information.

[1044] The system according to the present invention receives and analyzes a patient's clinical diagnostic data, and then uses artificial intelligence to generate a diagnostic result based on the analysis results, which is then integrated with the doctor's diagnostic result and displayed. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system is characterized by utilizing the doctor's emotional state in diagnostic support. A specific embodiment of this system is described below.

[1045] Data collection and input

[1046] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. The terminal is provided with a form for data input, and for example, the user can enter a blood glucose level of 180 mg / dL, a cholesterol level of 250 mg / dL, and upload a chest X-ray image.

[1047] Data transmission and storage

[1048] The device structures the input clinical diagnostic data and sends it to the server using an HTTP request. The server receives this data in the appropriate format and stores it in a relational database, such as a MySQL database. Numeric data is stored in the numeric database, image data in the image database, and auscultation, visual inspection, and palpation data in the text database and audio database.

[1049] Data analysis and diagnostic preparation

[1050] The server analyzes the stored data. For example, blood test results are analyzed using Python's numerical computing libraries (NumPy and Pandas) to detect abnormalities. Chest X-ray images are analyzed to detect abnormalities using deep learning frameworks such as TensorFlow and PyTorch. The analyzed data is passed to an AI module, which uses a generative AI model to evaluate diagnostic possibilities and generate a list of diagnoses and their rationales.

[1051] Comparison of diagnoses by doctors and AI

[1052] The server compares the AI-generated diagnosis with the diagnosis entered by the user (doctor), uses a comparison algorithm to identify matches and differences, adds complementary information (e.g., relevant research data and statistical data), and generates a unified diagnosis, which is ultimately presented to the doctor.

[1053] Emotion Engine Monitoring

[1054] While the user (doctor) is checking the diagnosis results, the device sends data to the emotion engine using a camera and microphone. The emotion engine analyzes facial expressions using libraries such as OpenCV and Dlib, and voice tone and pitch using a voice analysis library, to recognize the doctor's emotional state (stress level or lack of attention) in real time.

[1055] Emotion-based feedback and warnings

[1056] The server receives the emotional data sent from the emotion engine and generates appropriate feedback based on the doctor's emotional state. For example, if the doctor is feeling stressed, it sends a warning message to the device in real time via WebSocket, prompting the doctor to reconfirm the diagnosis. Furthermore, if the doctor's emotional state is determined to have a significant impact on the diagnosis, it provides AI-generated diagnostic support information.

[1057] Recording and feedback

[1058] The server stores the doctor's final diagnosis in a database. It also receives and analyzes feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support. This feedback is used for future AI training to improve diagnostic accuracy.

[1059] User Interface

[1060] The device provides a user interface that allows intuitive data entry, confirmation of diagnosis results, and feedback. The data entry screen displays input forms and templates, allowing doctors to quickly enter data. The diagnosis result display screen also provides a GUI that allows easy visual comparison of the AI's diagnosis results with those of the doctor. The feedback entry screen displays a simple input form, allowing doctors to enter feedback in a short amount of time.

[1061] Hypothetical example

[1062] Let's take the hypothetical case of a 45-year-old male patient. His blood test results show a blood sugar level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is uploaded. The device sends this data to the server, which analyzes it. The AI ​​module uses this data to evaluate the possibility of diabetes and pneumonia and generate a diagnosis. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[1063] Prompt Sentence Examples

[1064] The following prompt sentences can be input to a generative AI model to explain the specific functions of the system:

[1065] A 45-year-old male patient's blood test results are entered: blood sugar 180 mg / dL (abnormal) and cholesterol 250 mg / dL (high), and a chest x-ray is uploaded. Explain how the system will operate based on this data.

[1066] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1067] Program processing flow

[1068] Step 1: Data collection and entry

[1069] A user (doctor) needs to input a patient's clinical diagnostic data into a terminal. Input items include blood test results, image diagnostic data, auscultation data, visual examination data, and palpation data. For example, a blood glucose level of 180 mg / dL and a cholesterol level of 250 mg / dL are input as numerical values, and a chest X-ray image is uploaded as a file. This data is structured using an input form on the terminal.

[1070] Input: Patient clinical diagnostic data (numerical data, image data, etc.)

[1071] Output: Structured data format

[1072] Step 2: Send and store data

[1073] The device sends structured clinical diagnostic data to the server using HTTP requests. The server receives this data and stores it in a relational database, such as a MySQL database. Numeric data is stored in a numeric database, image data in an image database, and auscultation, visual inspection, and palpation data in a text database or audio database.

[1074] Input: Structured data format

[1075] Output: Data stored in the database

[1076] Step 3: Data analysis and diagnostic preparation

[1077] The server analyzes the stored data. Blood test results are analyzed using Python's numerical calculation libraries (NumPy, Pandas) to detect abnormal values. Chest X-ray images are analyzed using TensorFlow and PyTorch to detect abnormal areas. The analysis results are passed to an AI module, which uses a generative AI model to evaluate diagnostic feasibility and generate a diagnosis list and rationale.

[1078] Input: Data stored in a database

[1079] Output: Analysis results, diagnostic list

[1080] Step 4: Comparing diagnoses between doctors and AI

[1081] The server compares the diagnosis results generated by the generative AI model with those entered by the user (doctor). It uses a comparison algorithm to identify similarities and differences. It adds relevant supplementary information (related research data, statistical data, etc.) to generate a final integrated diagnosis.

[1082] Input: Analysis results, diagnosis list, doctor's diagnosis

[1083] Output: Integrated diagnostic results

[1084] Step 5: Emotion Engine Monitoring

[1085] While the user (doctor) is checking the diagnosis results, the device uses a camera and microphone to send emotional data to the emotion engine, which then analyzes facial expressions using OpenCV and Dlib, and voice tone and pitch using a voice analysis library to recognize the doctor's emotional state.

[1086] Input: Audio and video data of the doctor confirming the diagnosis

[1087] Output: Doctor's emotional state data (stress level, attention deficit)

[1088] Step 6: Emotion-based feedback and warnings

[1089] The server receives emotional state data sent from the emotion engine and generates feedback and warning messages based on that data. For example, if a doctor is feeling stressed, a warning message urging reconsideration is sent to the device via WebSocket and displayed.

[1090] Input: Emotional state data

[1091] Output: Feedback and warning messages

[1092] Step 7: Recording and feedback

[1093] The server stores the doctor's final diagnosis results in a database. It also receives feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support, and stores this information in the database. This feedback information is used for further learning by the AI, contributing to improving diagnostic accuracy.

[1094] Input: Final diagnosis results, feedback information

[1095] Output: Feedback information stored in a database

[1096] Step 8: User Interface

[1097] The device provides an intuitive user interface through which users (doctors) can input data, check diagnosis results, and provide feedback. The data input screen displays specific forms and templates. The diagnosis result display screen displays the AI's diagnosis results and the doctor's diagnosis results in an easy-to-read format. The feedback input screen provides a simple input form.

[1098] Input: Data entry, diagnostic result confirmation, feedback entry

[1099] Output: User feedback, diagnostic results

[1100] (Application example 2)

[1101] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1102] Conventional maintenance and fault diagnosis systems for factory robots only use analytical results based on the robot's sensor data, and do not reflect the emotional state of the maintenance staff. As a result, early detection of robot abnormalities or breakdowns is difficult due to mistakes and delayed judgment caused by stress or fatigue.

[1103] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving sensor data from the robot, means for analyzing the saved sensor data, means for an AI to evaluate the possibility of a failure based on the analysis results and generate a diagnosis result, means for receiving emotion data using an emotion engine that recognizes the emotional state of the maintenance staff, and means for providing diagnosis support based on the emotional state. This enables failure diagnosis support that takes into account the emotional state of the staff as well as the sensor data.

[1104] "Patient clinical diagnostic data" refers to medical data collected for diagnosis, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[1105] A "numerical database" refers to a database that stores numerical data such as blood test results and sensor data.

[1106] "Image database" refers to a database that stores image data such as X-ray images and CT scans.

[1107] "Analysis" refers to the data processing and evaluation process that detects abnormal values ​​and abnormal areas based on stored clinical diagnostic data and sensor data.

[1108] "AI generates diagnostic results" refers to the process of creating a diagnostic list and its rationale using a machine learning algorithm based on analyzed data.

[1109] "Doctor's diagnosis result" refers to the diagnostic judgment and conclusion made by the doctor regarding the condition of the patient or robot.

[1110] "User Interface" refers to the system's interactive screen through which a user (physician or maintenance technician) can input data, review diagnostic results, and provide feedback.

[1111] "Emotion engine" refers to technology that uses voice analysis and facial expression recognition technology to identify a user's emotional state and assess their stress and fatigue levels.

[1112] "Emotion Data" refers to data regarding emotional states (e.g., stress, joy, sadness, etc.) recognized by the emotion engine.

[1113] "Diagnostic support" refers to providing additional information and warning messages during the diagnostic process to help doctors and maintenance personnel make better decisions.

[1114] "Feedback" refers to information entered as evaluations and impressions of diagnostic results and work processes, which are used as learning data for the system.

[1115] In the present invention, a plurality of hardware and software components are integrated to provide a maintenance and fault diagnosis support system for a factory robot. Specific embodiments will be described below.

[1116] Data collection and input

[1117] The user (maintenance staff) inputs the robot's sensor data into a dedicated terminal. This data includes various sensor data such as temperature, vibration, and current. The terminal collects this data in real time and sends it to a server. A general-purpose PC or smart device is suitable as the terminal to be used.

[1118] Data transmission and storage

[1119] The device sends the input sensor data to the server, which stores the received data in a numerical database and an image database. For example, temperature and vibration data are stored in the numerical database and analyzed as needed.

[1120] Data analysis and fault diagnosis

[1121] The server analyzes the stored sensor data using a data processing module written in Python. The analysis results are passed to an AI module (built using TensorFlow / Keras), which evaluates the possibility of a fault and generates a diagnosis.

[1122] Display and compare diagnostic results

[1123] The server displays the diagnostic results generated by the AI ​​on a user interface, which provides an intuitive graphical user interface (GUI) that allows users (maintenance personnel) to check the diagnostic results.

[1124] Emotion Engine Monitoring

[1125] The device uses an emotion engine to recognize the emotional state of the person checking the diagnosis results. The emotion engine uses facial recognition technology (OpenCV) and voice analysis technology to detect stress and fatigue of the person.

[1126] Emotion-based feedback and warnings

[1127] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the person in charge is feeling stressed, the server displays a warning message urging the person to reconfirm the diagnosis result. In addition, if the person in charge's emotional state affects the diagnosis, the server provides additional diagnostic support information.

[1128] Recording and feedback

[1129] The server stores the user's final diagnosis results in a numerical database. It also receives and stores feedback on the user's emotional state during the diagnosis process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI, helping to improve its diagnostic accuracy in the future.

[1130] Implementing specific examples

[1131] In a factory, a maintenance worker performs a routine inspection of a factory robot. The worker operates a smart device to collect sensor data (such as temperature and vibration) in real time. This data is then sent to a server, where it is analyzed and displayed along with the results of a diagnosis made by an AI module. Meanwhile, an emotion engine recognizes the worker's stress level and provides feedback to encourage relaxation as needed.

[1132] (Example of prompts for a generative AI model)

[1133] "Detect stress levels in factory maintenance personnel and generate prompts for an assistant system that provides feedback to encourage breaks."

[1134] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1135] Step 1:

[1136] The user (maintenance technician) begins maintenance work on a factory robot. First, various sensor data such as temperature, vibration, and current are collected from sensors connected to a dedicated terminal and input into the terminal. At this time, the data from each sensor is displayed in real time on the terminal's UI.

[1137] Step 2:

[1138] The device sends the collected sensor data to the server. The sensor data includes temperature values ​​measured every 0.1 seconds, vibration frequency spectrum, and current time series data. The device then converts the data into a specific format (e.g., JSON format) and sends it to the server via the network.

[1139] Step 3:

[1140] The server sequentially stores the received sensor data in a numeric database and an image database. When storing the data in the database, the data is separated by sensor type and stored in the appropriate field. For example, temperature data is stored in the "temperature" field, and vibration data is stored in the "vibration" field.

[1141] Step 4:

[1142] The server analyzes the stored sensor data. Specifically, it detects abnormal values ​​in temperature data, analyzes the frequency of vibration data, and recognizes patterns in current data. From this analysis, an anomaly detection algorithm is applied to extract abnormal values ​​and patterns. The analysis results include peak temperature values, the ratio of high-frequency components in vibration, and abnormal current peaks.

[1143] Step 5:

[1144] The server passes the analysis results to an AI module (using TensorFlow / Keras), which then generates a diagnosis. The AI ​​module evaluates current sensor data based on past learning data and predicts the likelihood of robot failure. The diagnosis results include the possibility of wear on specific parts, failure due to overheating, and damage due to abnormal vibration.

[1145] Step 6:

[1146] The server displays the diagnostic results generated by the AI ​​on a user interface. The user interface uses a GUI to visually display the diagnostic results, allowing the user to intuitively understand them. Specifically, it displays an overview of the diagnostic results, highlights the areas where abnormalities have occurred, and recommends actions (e.g., part replacement, adjustment).

[1147] Step 7:

[1148] The device uses an emotion engine to recognize the user's emotional state while checking the diagnosis results. The emotion engine captures the user's face with the device's built-in camera and analyzes their emotions using a facial expression recognition algorithm (OpenCV). This analysis determines whether the user is feeling stressed or relaxed.

[1149] Step 8:

[1150] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the emotion analysis detects that the user is feeling stressed, the server displays a warning message on the user interface. This message may include a message to reconfirm the diagnosis result or to take a break.

[1151] Step 9:

[1152] The server stores the final diagnosis results and emotion data provided by the user in a numerical database. The stored data will be used as training data to improve the accuracy of the diagnosis. In addition, the user can input feedback on the final diagnosis results via the user interface, and this feedback will also be stored in the database.

[1153] Step 10:

[1154] The server analyzes the stored final diagnosis results and emotion data and uses them to retrain the AI ​​model, a process that improves future diagnosis accuracy and overall system performance.

[1155] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1156] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1157] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1158] [Fourth embodiment]

[1159] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1160] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1161] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1162] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1163] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1165] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1166] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1167] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1168] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1169] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1170] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1171] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1172] The system of the present invention improves diagnostic accuracy by having a doctor and an AI simultaneously perform diagnoses based on the patient's clinical diagnostic data and then displaying the integrated results. A specific example of this system is described below.

[1173] Data collection and input

[1174] The user (doctor) inputs the patient's clinical diagnostic data. This data includes blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data. For example, blood test results include blood glucose and cholesterol levels, and in addition, chest X-ray images and heart sound data are also input.

[1175] Data transmission and storage

[1176] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored as numerical data, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[1177] Data analysis and diagnostic preparation

[1178] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the possibility of a diagnosis based on this analysis data and generates a list of diagnoses and their rationale.

[1179] Comparison of diagnoses by doctors and AI

[1180] The server compares the diagnostic results generated by the AI ​​with the diagnostic data entered by the user (doctor). Here, the AI ​​diagnostic results, the doctor's diagnostic results, and supplementary information are integrated to generate data to be displayed on the user interface. This allows the user (doctor) to make a final diagnosis while referring to the AI ​​diagnostic results.

[1181] Recording and feedback

[1182] The server stores the final diagnosis made by the user (doctor) in a database. It also receives feedback on how helpful the AI ​​diagnosis was during the diagnostic process. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[1183] User Interface

[1184] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[1185] Implementing specific examples

[1186] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. This is then used to make the final diagnosis and provide feedback.

[1187] The present invention enables doctors to make diagnoses more accurately and quickly, and can significantly improve the accuracy and efficiency of diagnosis.

[1188] The processing flow will be explained below.

[1189] Step 1:

[1190] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[1191] Step 2:

[1192] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[1193] Step 3:

[1194] The server stores the received clinical diagnostic data in a database: blood test results are stored in a numerical database, imaging diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored in an appropriate format.

[1195] Step 4:

[1196] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[1197] Step 5:

[1198] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, the module evaluates the possibility of diabetes, and if a chest X-ray image is used, the module evaluates the possibility of pneumonia.

[1199] Step 6:

[1200] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[1201] Step 7:

[1202] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[1203] Step 8:

[1204] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[1205] Step 9:

[1206] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[1207] Step 10:

[1208] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[1209] Example 1

[1210] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1211] In modern medical settings, we often rely solely on doctors' diagnoses, which poses challenges in diagnostic accuracy and efficiency. In particular, integrating multiple diagnostic data to make a comprehensive diagnosis is time-consuming and laborious. Furthermore, feedback during the diagnostic process is not adequately collected and utilized, creating a need for the development of a diagnostic support system using artificial intelligence.

[1212] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1213] In this invention, the server includes means for receiving patient's biological information, means for storing the received biological information in a recording device, means for analyzing the stored biological information, means for an AI module to generate a diagnosis result based on the analysis result, means for integrating the AI ​​module's diagnosis result with the doctor's diagnosis result and displaying it on a display device, means for making a final diagnosis based on the integrated diagnosis result, and means for collecting feedback during the diagnostic process, thereby enabling the accuracy and efficiency of diagnosis to be improved.

[1214] "Patient's biological information" refers to data indicating the patient's health condition, and includes blood test results, diagnostic imaging data, auscultation data, visual examination data, palpation data, and the like.

[1215] The "receiving means" refers to a means for inputting the patient's vital signs into the system, such as a data input interface or a data transfer protocol.

[1216] "Recording device" refers to a device for storing received patient biometric information in digital form, such as a database or storage system.

[1217] The "analyzing means" refers to a means for carrying out data analysis necessary for making a medical diagnosis based on the stored biological information, and includes a numerical analysis module, an image analysis module, and the like.

[1218] An "artificial intelligence module" is a device or software that generates diagnostic results based on analyzed biometric information, and includes machine learning algorithms and AI models.

[1219] "Means for integrating and displaying" refers to a means for comparing and integrating the diagnostic results generated by the AI ​​module with those of the doctor and visually presenting them to the user, and refers to a display device or graphical user interface (GUI).

[1220] The "means for making a final diagnosis" refers to the means by which a doctor determines a final diagnosis based on the integrated diagnostic results, and includes a diagnostic support system and an interface that supports the doctor's judgment.

[1221] "Means for collecting feedback" refers to means for receiving, analyzing, and storing feedback provided during the diagnostic process, and refers to a feedback input form and a data analysis module.

[1222] The system of this invention collects patient biometric information, and doctors and artificial intelligence jointly perform diagnoses, and the results are integrated and displayed to improve the accuracy of diagnoses. Specific embodiments of the present invention are described below.

[1223] Data collection and input

[1224] The user (doctor) enters the patient's biometric information into the system, which includes the following data:

[1225] Blood test results (e.g., blood sugar 180 mg / dL, cholesterol 250 mg / dL)

[1226] Imaging data (e.g., chest x-rays)

[1227] Auscultation data (e.g., heart sound data)

[1228] Visual inspection data

[1229] Palpation data

[1230] The user inputs data into a form or template on the system's user interface and uploads image and audio data as files.

[1231] Data transmission and storage

[1232] The device sends the input biometric information to a server using a secure communication protocol such as HTTPS.

[1233] The server stores the received biometric information in a recording device as follows:

[1234] Storing blood test results in a numerical database

[1235] Storing diagnostic imaging data in an image database

[1236] Save auscultation, visual inspection, and palpation data as text or audio data

[1237] Data analysis and diagnostic preparation

[1238] The server passes the data stored in the recording device to the analysis module, which includes the following specific processes:

[1239] Blood test results are analyzed by a numerical analysis module and compared with defined normal ranges to identify abnormal values.

[1240] The diagnostic imaging data is analyzed by the image analysis module to identify abnormalities.

[1241] The results of these analyses are passed to an artificial intelligence module, which evaluates the possibility of diagnosis based on the analyzed data and generates a list of diagnoses and their rationale.

[1242] Comparison of diagnoses by doctors and AI

[1243] The server compares the diagnosis results generated by the AI ​​module with the diagnosis data entered by the user (doctor). The specific process includes:

[1244] Compare AI diagnosis results with doctor diagnosis results

[1245] Identify commonalities and differences to generate integrated diagnostic results

[1246] As supplementary information, the basis for the AI ​​diagnosis and the doctor's reasons for the diagnosis will also be integrated.

[1247] The integrated diagnostic results are generated as data to be displayed on a user interface.

[1248] Recording and feedback

[1249] The server stores the final diagnosis made by the user (doctor) in the recording device. Specific operations include:

[1250] Record the final diagnostic results in the appropriate database table

[1251] Collect feedback on the effectiveness of the AI ​​in the diagnostic process and store it as training data for the AI ​​module.

[1252] User Interface

[1253] The terminal provides a user interface with the following features:

[1254] Data entry screens: Forms and templates that allow doctors to easily enter data

[1255] Diagnostic result display screen: A graphical user interface (GUI) that makes it easy to compare AI diagnostic results with those of a doctor.

[1256] Feedback input screen: A form where doctors can enter brief feedback

[1257] Implementing specific examples

[1258] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show that his blood sugar level is 180 mg / dL (abnormal) and his cholesterol level is 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which then stores it in a recording device. The server analyzes this data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made.

[1259] Prompt Sentence Examples

[1260] Here is an example of a prompt you can enter into the system:

[1261] "AI diagnosis will be performed based on the patient's clinical data. Please enter the following data.

[1262] Blood test results:

[1263] Blood glucose level: 180 mg / dL

[1264] Cholesterol level: 250 mg / dL

[1265] Imaging data: Chest X-ray image (file upload)

[1266] Generate your results and compare them with your doctor's diagnosis. View the results and collect feedback."

[1267] Using this prompt sentence, the system can make an accurate and efficient diagnosis, which helps support doctors in making diagnoses.

[1268] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1269] Step 1: Data collection and input

[1270] The user (doctor) inputs the patient's biometric information into the system. This biometric information includes blood test results (e.g., blood glucose level 180 mg / dL, cholesterol level 250 mg / dL), diagnostic imaging data (e.g., chest X-ray image), auscultation data (e.g., heart sound data), visual examination data, and palpation data. The user (doctor) inputs this data using a form or template provided on the system's user interface and uploads the image and audio data as files. The input data is used as is in the next step.

[1271] Step 2: Send and store data

[1272] The device sends the entered biometric information to the server. This data transmission is done using a secure communication protocol such as HTTPS. The transmitted data is then stored in the appropriate database on the server. Specifically, blood test results are stored in a numerical database, diagnostic imaging data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text or audio data. This makes the biometric information available for analysis.

[1273] Step 3: Prepare for data analysis and diagnosis

[1274] The server passes the data stored in the recording device to the analysis module. Specifically, blood test results are first analyzed by a numerical analysis module. Abnormal values ​​are identified by comparison with normal ranges, and this information is passed to the next step. Image diagnostic data is analyzed by an image analysis module, and abnormal areas are identified. These analysis results are passed to an artificial intelligence module, which generates a diagnosis list and diagnostic rationale.

[1275] Step 4: Comparing the doctor's and AI's diagnoses

[1276] The server compares the diagnosis results generated by the AI ​​module with the diagnostic data entered by the user (doctor). Specifically, the AI ​​diagnosis results are compared with the doctor's diagnosis results, and similarities and differences are identified. The integrated diagnosis results, along with supplementary information (e.g., the basis of the AI ​​diagnosis and the doctor's reasons for the diagnosis), are generated as data to be displayed on the user interface. They are presented in an intuitive format so that the user can refer to the final diagnosis.

[1277] Step 5: Recording and feedback

[1278] The server stores the final diagnosis made by the user (doctor) in a recording device. Specifically, it records the final diagnosis in an appropriate database table and collects feedback on the effectiveness of the AI ​​in the diagnostic process. This feedback is stored as training data for the AI ​​module and used to improve diagnostic accuracy in the future.

[1279] Step 6: Navigate the User Interface

[1280] The terminal provides an intuitive user interface. Specifically, the data entry screen is equipped with forms and templates that doctors can use to easily enter data. The diagnosis result display screen provides a graphical user interface (GUI) designed to make it easy to compare the AI's diagnosis results with those of doctors. The feedback input screen provides a form that doctors can use to easily enter feedback, allowing for the collection of information to improve the quality of diagnoses.

[1281] These steps enable the system to achieve accurate and efficient diagnosis through collaboration between doctors and AI, improving the accuracy of the final diagnosis.

[1282] (Application example 1)

[1283] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1284] Conventional medical diagnostic systems and factory quality inspection systems have limitations in accuracy and efficiency when integrating multiple diagnostic results. It has also been difficult to effectively integrate the results of AI and human diagnoses and reflect them in the final diagnosis. With the demand for improved diagnostic accuracy and efficiency in medical and factory settings, it is necessary to solve these issues.

[1285] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1286] In this invention, the server

[1287] means for receiving clinical diagnostic data of a patient;

[1288] means for storing the received clinical diagnostic data in a database;

[1289] means for analyzing the stored clinical diagnostic data;

[1290] A means for AI to generate diagnostic results based on the analysis results, and

[1291] A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface;

[1292] A means for making a final diagnosis based on the integrated diagnostic results;

[1293] means for receiving quality inspection data;

[1294] means for analyzing the received quality inspection data;

[1295] A method for AI and robots to generate quality diagnosis results based on the analysis results, and

[1296] A means to integrate the diagnostic results of AI and robots and display them on a user interface,

[1297] Includes means for providing audio feedback.

[1298] This will enable improvements in diagnostic accuracy in the medical field and quality inspection accuracy in factories.

[1299] "Patient" refers to a person who is the subject of diagnosis or treatment at a medical institution.

[1300] "Clinical diagnostic data" refers to a variety of biological, imaging, audiological, and mechanical data collected to assess a patient's health status.

[1301] A "database" refers to a system for efficiently storing, retrieving, and managing data.

[1302] "Analysis" refers to performing computations to detect patterns or anomalies in the received data.

[1303] "AI" refers to artificial intelligence that can automatically perform specific tasks with high accuracy.

[1304] "Diagnosis results" refer to the evaluation results of health status and quality obtained based on data analysis.

[1305] "Physician" refers to a professional with higher education who diagnoses and treats patients.

[1306] "User interface" refers to the interface through which a user interacts with a system.

[1307] "Final diagnosis" refers to the final assessment or conclusion obtained by integrating the diagnostic results of the AI ​​and the doctor (or robot).

[1308] "Quality Inspection Data" means data collected to evaluate the quality of a product.

[1309] "Robot" refers to a mechanical device for performing tasks automatically or semi-automatically.

[1310] "Voice feedback" refers to the function of providing analysis and diagnostic results to the user via voice.

[1311] This invention is a system that uses AI and robots to perform highly accurate diagnoses based on patient clinical diagnostic data and factory quality inspection data, and then integrates and displays the results. This system can improve the accuracy and efficiency of diagnoses in both the medical and manufacturing fields. Specific examples are described below.

[1312] System overview

[1313] The system is composed of a server, a terminal, and a user (a doctor or a diagnostician). The system includes the following means:

[1314] 1. Means for receiving patient clinical diagnosis data: A means for users to input patient blood test results, image diagnosis data, auscultation data, visual examination data, palpation data, etc. through a terminal.

[1315] 2. Database storage means: A means for efficiently storing received data in a database. Blood test results are stored as numerical data, and diagnostic imaging data is stored as image files, audio data, etc.

[1316] 3. Data analysis method: This is a method for analyzing the stored data and detecting outliers, etc. Software such as Python, Keras, OpenCV, and Pyttsx3 is used for this.

[1317] 4. AI diagnostic result generation means: This is the means by which AI generates diagnostic results based on the analysis results. The AI ​​model uses a pre-trained model.

[1318] 5. Diagnostic result integration means: A means for integrating the AI ​​diagnostic results with the doctor's diagnostic results and displaying them on the user interface.

[1319] 6. Final diagnostic tool: A tool for making a final diagnosis based on the integrated diagnostic results.

[1320] 7. Quality inspection data receiving means: This is the means by which the factory robot receives the quality inspection data of the product.

[1321] 8. Quality inspection data analysis means: A means for analyzing received quality inspection data.

[1322] 9. Means for generating diagnostic results using AI and robots: A means for AI and robots to generate diagnostic results based on analyzed quality inspection data.

[1323] 10. Audio feedback means: A means for providing the diagnostic results to the user by voice.

[1324] Specific examples

[1325] Specific examples in the medical field

[1326] Clinical data for a hypothetical patient (a 45-year-old male) is entered, including blood glucose levels (180 mg / dL), cholesterol levels (250 mg / dL), and chest X-ray images. This data is sent to a server and stored in a database. Data analysis is then performed, and the AI ​​detects abnormal values ​​and generates diagnoses of diabetes and pneumonia. These results are displayed to the doctor through a user interface, who then makes a final diagnosis based on the integrated information.

[1327] Specific example of factory quality inspection

[1328] Factory robots receive image data as product quality inspection data, which is then analyzed by AI. For example, an image analysis algorithm is applied to determine whether there are scratches on the surface of the product. The diagnostic results of the AI ​​and robot are integrated and displayed on the user interface. As a result, quality control personnel can make the final quality judgment.

[1329] Prompt Sentence Examples

[1330] Analyze the product image below and output the diagnostic results regarding the product quality.

[1331] Example: main("product_image.jpg", "quality_model.h5", "robot_measured_data")

[1332] The configuration and implementation of this system can significantly improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[1333] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1334] Step 1:

[1335] The user inputs the patient's clinical diagnostic data into the terminal, including blood test results, imaging diagnostic data, auscultation data, visual examination data, palpation data, etc. This data is collected through an input form or file upload.

[1336] Step 2:

[1337] The terminal transmits the entered clinical diagnostic data to a server over the Internet, and the data is transmitted in multiple formats (text, numbers, images, and audio).

[1338] Step 3:

[1339] The server stores the received clinical diagnostic data in a database: blood test results in a numerical database, diagnostic imaging data in an image database, and voice data in a voice database.

[1340] Step 4:

[1341] The server analyzes the stored data: numerical data is analyzed using outlier detection algorithms, image data is analyzed using image analysis models to detect lesions, and audio data is analyzed using audio analysis algorithms.

[1342] Step 5:

[1343] The server uses the analysis results to generate a diagnosis using AI, which uses a pre-trained generative AI model to evaluate the disease name and likelihood of the condition.

[1344] Step 6:

[1345] The server integrates the AI's diagnosis results with those of the doctor, comparing the diagnosis results entered by the doctor with those generated by the AI ​​and integrating them based on certain rules.

[1346] Step 7:

[1347] The server displays the integrated diagnostic results on a user interface, through which the user (doctor) can confirm the diagnostic results and make a final diagnosis.

[1348] Step 8:

[1349] The user enters the final diagnosis and the server stores the results in a database, including any feedback or additional comments.

[1350] Step 9:

[1351] The user inputs factory quality inspection data into the terminal, including image files and sensor data.

[1352] Step 10:

[1353] The terminal transmits the input quality inspection data to the server.

[1354] Step 11:

[1355] The server analyzes the received quality inspection data. Image data is analyzed using image analysis models to detect product defects, and robot measurement data is analyzed using anomaly detection algorithms.

[1356] Step 12:

[1357] The server then uses the AI ​​and robots to generate quality diagnosis results based on the analysis results. The AI ​​model evaluates the product's quality, and the robot measures the product's physical characteristics.

[1358] Step 13:

[1359] The server integrates the diagnostic results of the AI ​​and robot and displays them on a user interface. The user (quality control officer) can then check the quality diagnostic results through this interface and make a final decision.

[1360] Step 14:

[1361] The server provides the diagnosis results to the user through voice feedback, using a speech synthesis library such as Pyttsx3.

[1362] Taking such steps can improve the accuracy and efficiency of diagnostics in medical and industrial settings.

[1363] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1364] The system according to the present invention receives and analyzes the clinical diagnostic data of patients, and based on the results, AI generates diagnostic results, which are then integrated with the doctor's diagnostic results and displayed. In addition, it incorporates an emotion engine that recognizes the user's emotions, and utilizes the doctor's emotional state in diagnostic support. Specific embodiments of this system are described below.

[1365] Data collection and input

[1366] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[1367] Data transmission and storage

[1368] The terminal sends the entered clinical diagnostic data to the server. The server stores the received data in a database. Blood test results are stored in a numerical database, and image diagnostic data is stored in an image database. Auscultation, visual inspection, and palpation data are stored as text and audio data.

[1369] Data analysis and diagnostic preparation

[1370] The server analyzes the stored data. For example, blood test results are analyzed by a numerical module to detect abnormal values. Diagnostic imaging data is analyzed by an image analysis module to detect abnormalities. The analyzed data is passed to an AI module. The AI ​​module evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale.

[1371] Comparison of diagnoses by doctors and AI

[1372] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[1373] Emotion Engine Monitoring

[1374] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[1375] Emotion-based feedback and warnings

[1376] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the doctor's emotional state. For example, if the doctor is feeling stressed, it displays a warning message urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, it provides additional diagnostic support information.

[1377] Recording and feedback

[1378] The server stores the final diagnosis made by the user (doctor) in a database. It also receives and stores feedback on the user's emotional state during the diagnostic process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI ​​and will help improve diagnostic accuracy in the future.

[1379] User Interface

[1380] The terminal provides a user interface that allows users (doctors) to intuitively input data, check diagnosis results, and send feedback. The data input screen provides forms and templates that doctors can use to easily enter data, and the diagnosis result display screen provides an intuitive graphical user interface (GUI) that makes it easy to compare the AI's diagnosis results with those of the doctor. The feedback input screen also provides a simple input form for doctors to enter feedback.

[1381] Implementing specific examples

[1382] Let's take the example of a hypothetical patient, a 45-year-old male. His blood test results show a blood glucose level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is also uploaded. The device sends this data to the server, which stores it in a database. The server analyzes the data, and an AI module evaluates the likelihood of diabetes and pneumonia and generates a diagnosis. Based on this, a final diagnosis and feedback are made. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[1383] The present invention allows doctors to make diagnoses more accurately and quickly, significantly improving the accuracy and efficiency of diagnoses. Furthermore, by utilizing the doctor's emotional state as diagnostic support, the quality of diagnoses can be further improved.

[1384] The processing flow will be explained below.

[1385] Step 1:

[1386] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. For example, the user inputs blood glucose and cholesterol levels and uploads chest X-ray images.

[1387] Step 2:

[1388] The terminal transmits the entered clinical diagnostic data to a server, where the data is transmitted securely in electronic form.

[1389] Step 3:

[1390] The server stores the received clinical diagnostic data in a database. Blood test results are stored in a numerical database, image diagnostic data is stored in an image database, and auscultation, visual inspection, and palpation data are stored as text and audio data.

[1391] Step 4:

[1392] The server analyzes the stored clinical diagnostic data. Numerical data is analyzed by a numerical module to detect abnormal values. Image data is analyzed by an image analysis module to detect abnormalities. The analysis results are output in a standardized format.

[1393] Step 5:

[1394] The server passes the analyzed data to the AI ​​module, which evaluates the diagnostic possibilities based on the analyzed data and generates a list of diagnoses and their rationale. For example, if blood sugar levels are high, it evaluates the possibility of diabetes, and if a chest X-ray image is taken, it evaluates the possibility of pneumonia.

[1395] Step 6:

[1396] The server compares the AI-generated diagnosis results with those entered by the user (doctor), integrates the comparison results with relevant supplementary information (e.g., related research data), and generates data to be displayed.

[1397] Step 7:

[1398] The terminal displays the integrated diagnostic results on a user interface, allowing the user (doctor) to make a final diagnosis based on the AI's diagnostic results.

[1399] Step 8:

[1400] The device uses an emotion engine to recognize the user's (doctor's) emotional state while checking the diagnosis results. The emotion engine uses voice analysis and facial expression recognition technology to detect the doctor's stress level and lack of attention.

[1401] Step 9:

[1402] The server receives the emotion data sent from the emotion engine and provides diagnostic assistance based on the doctor's emotional state. For example, if the doctor is feeling stressed, a warning message is displayed urging the doctor to reconfirm the diagnosis. In addition, if the doctor's emotional state affects the diagnosis, additional diagnostic assistance information is provided.

[1403] Step 10:

[1404] The user (doctor) checks the final diagnosis result and inputs feedback, such as how useful the AI ​​diagnosis was and its accuracy.

[1405] Step 11:

[1406] The server stores the final diagnosis results and feedback in a database, which can then be used as learning data for the AI, helping to continuously improve diagnostic accuracy.

[1407] Step 12:

[1408] The terminal provides an intuitive user interface for users (doctors), allowing for smooth data entry, confirmation of diagnosis results, and feedback.

[1409] Example 2

[1410] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1411] Conventional clinical diagnostic systems lack the integration of doctor and AI diagnoses, and lack diagnostic support that takes into account the doctor's emotional state. As a result, diagnostic accuracy and doctor work efficiency may decrease, and an appropriate diagnosis may not be made. A system that solves these problems and improves diagnostic accuracy and efficiency is needed.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1413] In this invention, the server provides means for receiving clinical diagnostic data of patients, means for storing the received clinical diagnostic data in a database, means for analyzing the stored clinical diagnostic data, means for an AI to generate a diagnostic result based on the analysis result, means for integrating the AI's diagnostic result and the doctor's diagnostic result and displaying them on a user interface, means for recognizing the doctor's emotional state, means for providing feedback and warnings based on the emotional state, and means for making a final diagnosis based on the integrated diagnostic result, thereby improving the accuracy and efficiency of diagnosis and enabling appropriate diagnostic support that takes the doctor's emotional state into consideration.

[1414] "Patient clinical diagnostic data" includes medical information about the patient, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[1415] "Means for receiving" refers to a combination of hardware and software for electronically receiving patient clinical diagnostic data.

[1416] "Means for storing in a database" refers to a storage device and associated management software for structuring and archiving received clinical diagnostic data.

[1417] "Means for analysis" refers to algorithms and software for computer-based analysis of stored clinical diagnostic data to detect outliers and specific patterns.

[1418] "Means for artificial intelligence to generate diagnostic results" refers to software that uses machine learning models to evaluate diagnostic possibilities based on the analysis results and generates diagnostic results based on that.

[1419] "Means for integrating and displaying on a user interface" refers to software and hardware that compares and integrates the AI ​​diagnostic results with the doctor's diagnostic results and provides a user interface for visually displaying that information.

[1420] "Means for recognizing the doctor's emotional state" refers to sensors and analysis software for analyzing the doctor's voice and facial expressions to determine his or her emotional state.

[1421] "Means for providing feedback and warnings" refers to software and devices for generating and presenting appropriate advice and warnings in response to a perceived emotional state.

[1422] "Means for making a final diagnosis" refers to the process and tools that allow a physician to determine a final diagnosis using the integrated diagnostic results and feedback information.

[1423] The system according to the present invention receives and analyzes a patient's clinical diagnostic data, and then uses artificial intelligence to generate a diagnostic result based on the analysis results, which is then integrated with the doctor's diagnostic result and displayed. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system is characterized by utilizing the doctor's emotional state in diagnostic support. A specific embodiment of this system is described below.

[1424] Data collection and input

[1425] The user (doctor) inputs the patient's clinical diagnostic data into the terminal. This data includes blood test results, imaging diagnostic data, auscultation data, visual examination data, and palpation data. The terminal is provided with a form for data input, and for example, the user can enter a blood glucose level of 180 mg / dL, a cholesterol level of 250 mg / dL, and upload a chest X-ray image.

[1426] Data transmission and storage

[1427] The device structures the input clinical diagnostic data and sends it to the server using an HTTP request. The server receives this data in the appropriate format and stores it in a relational database, such as a MySQL database. Numeric data is stored in the numeric database, image data in the image database, and auscultation, visual inspection, and palpation data in the text database and audio database.

[1428] Data analysis and diagnostic preparation

[1429] The server analyzes the stored data. For example, blood test results are analyzed using Python's numerical computing libraries (NumPy and Pandas) to detect abnormalities. Chest X-ray images are analyzed to detect abnormalities using deep learning frameworks such as TensorFlow and PyTorch. The analyzed data is passed to an AI module, which uses a generative AI model to evaluate diagnostic possibilities and generate a list of diagnoses and their rationales.

[1430] Comparison of diagnoses by doctors and AI

[1431] The server compares the AI-generated diagnosis with the diagnosis entered by the user (doctor), uses a comparison algorithm to identify matches and differences, adds complementary information (e.g., relevant research data and statistical data), and generates a unified diagnosis, which is ultimately presented to the doctor.

[1432] Emotion Engine Monitoring

[1433] While the user (doctor) is checking the diagnosis results, the device sends data to the emotion engine using a camera and microphone. The emotion engine analyzes facial expressions using libraries such as OpenCV and Dlib, and voice tone and pitch using a voice analysis library, to recognize the doctor's emotional state (stress level or lack of attention) in real time.

[1434] Emotion-based feedback and warnings

[1435] The server receives the emotional data sent from the emotion engine and generates appropriate feedback based on the doctor's emotional state. For example, if the doctor is feeling stressed, it sends a warning message to the device in real time via WebSocket, prompting the doctor to reconfirm the diagnosis. Furthermore, if the doctor's emotional state is determined to have a significant impact on the diagnosis, it provides AI-generated diagnostic support information.

[1436] Recording and feedback

[1437] The server stores the doctor's final diagnosis in a database. It also receives and analyzes feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support. This feedback is used for future AI training to improve diagnostic accuracy.

[1438] User Interface

[1439] The device provides a user interface that allows intuitive data entry, confirmation of diagnosis results, and feedback. The data entry screen displays input forms and templates, allowing doctors to quickly enter data. The diagnosis result display screen also provides a GUI that allows easy visual comparison of the AI's diagnosis results with those of the doctor. The feedback entry screen displays a simple input form, allowing doctors to enter feedback in a short amount of time.

[1440] Hypothetical example

[1441] Let's take the hypothetical case of a 45-year-old male patient. His blood test results show a blood sugar level of 180 mg / dL (abnormal) and a cholesterol level of 250 mg / dL (high), and a chest X-ray image is uploaded. The device sends this data to the server, which analyzes it. The AI ​​module uses this data to evaluate the possibility of diabetes and pneumonia and generate a diagnosis. During this process, the emotion engine monitors the doctor's emotional state and provides diagnostic assistance as needed.

[1442] Prompt Sentence Examples

[1443] The following prompt sentences can be input to a generative AI model to explain the specific functions of the system:

[1444] A 45-year-old male patient's blood test results are entered: blood sugar 180 mg / dL (abnormal) and cholesterol 250 mg / dL (high), and a chest x-ray is uploaded. Explain how the system will operate based on this data.

[1445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1446] Program processing flow

[1447] Step 1: Data collection and entry

[1448] A user (doctor) needs to input a patient's clinical diagnostic data into a terminal. Input items include blood test results, image diagnostic data, auscultation data, visual examination data, and palpation data. For example, a blood glucose level of 180 mg / dL and a cholesterol level of 250 mg / dL are input as numerical values, and a chest X-ray image is uploaded as a file. This data is structured using an input form on the terminal.

[1449] Input: Patient clinical diagnostic data (numerical data, image data, etc.)

[1450] Output: Structured data format

[1451] Step 2: Send and store data

[1452] The device sends structured clinical diagnostic data to the server using HTTP requests. The server receives this data and stores it in a relational database, such as a MySQL database. Numeric data is stored in a numeric database, image data in an image database, and auscultation, visual inspection, and palpation data in a text database or audio database.

[1453] Input: Structured data format

[1454] Output: Data stored in the database

[1455] Step 3: Data analysis and diagnostic preparation

[1456] The server analyzes the stored data. Blood test results are analyzed using Python's numerical calculation libraries (NumPy, Pandas) to detect abnormal values. Chest X-ray images are analyzed using TensorFlow and PyTorch to detect abnormal areas. The analysis results are passed to an AI module, which uses a generative AI model to evaluate diagnostic feasibility and generate a diagnosis list and rationale.

[1457] Input: Data stored in a database

[1458] Output: Analysis results, diagnostic list

[1459] Step 4: Comparing diagnoses between doctors and AI

[1460] The server compares the diagnosis results generated by the generative AI model with those entered by the user (doctor). It uses a comparison algorithm to identify similarities and differences. It adds relevant supplementary information (related research data, statistical data, etc.) to generate a final integrated diagnosis.

[1461] Input: Analysis results, diagnosis list, doctor's diagnosis

[1462] Output: Integrated diagnostic results

[1463] Step 5: Emotion Engine Monitoring

[1464] While the user (doctor) is checking the diagnosis results, the device uses a camera and microphone to send emotional data to the emotion engine, which then analyzes facial expressions using OpenCV and Dlib, and voice tone and pitch using a voice analysis library to recognize the doctor's emotional state.

[1465] Input: Audio and video data of the doctor confirming the diagnosis

[1466] Output: Doctor's emotional state data (stress level, attention deficit)

[1467] Step 6: Emotion-based feedback and warnings

[1468] The server receives emotional state data sent from the emotion engine and generates feedback and warning messages based on that data. For example, if a doctor is feeling stressed, a warning message urging reconsideration is sent to the device via WebSocket and displayed.

[1469] Input: Emotional state data

[1470] Output: Feedback and warning messages

[1471] Step 7: Recording and feedback

[1472] The server stores the doctor's final diagnosis results in a database. It also receives feedback on the patient's emotional state during the diagnostic process and the usefulness of the AI's diagnostic support, and stores this information in the database. This feedback information is used for further learning by the AI, contributing to improving diagnostic accuracy.

[1473] Input: Final diagnosis results, feedback information

[1474] Output: Feedback information stored in a database

[1475] Step 8: User Interface

[1476] The device provides an intuitive user interface through which users (doctors) can input data, check diagnosis results, and provide feedback. The data input screen displays specific forms and templates. The diagnosis result display screen displays the AI's diagnosis results and the doctor's diagnosis results in an easy-to-read format. The feedback input screen provides a simple input form.

[1477] Input: Data entry, diagnostic result confirmation, feedback entry

[1478] Output: User feedback, diagnostic results

[1479] (Application example 2)

[1480] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1481] Conventional maintenance and fault diagnosis systems for factory robots only use analytical results based on the robot's sensor data, and do not reflect the emotional state of the maintenance staff. As a result, early detection of robot abnormalities or breakdowns is difficult due to mistakes and delayed judgment caused by stress or fatigue.

[1482] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving sensor data from the robot, means for analyzing the saved sensor data, means for an AI to evaluate the possibility of a failure based on the analysis results and generate a diagnosis result, means for receiving emotion data using an emotion engine that recognizes the emotional state of the maintenance staff, and means for providing diagnosis support based on the emotional state. This enables failure diagnosis support that takes into account the emotional state of the staff as well as the sensor data.

[1483] "Patient clinical diagnostic data" refers to medical data collected for diagnosis, such as blood test results, diagnostic imaging data, auscultation data, visual examination data, and palpation data.

[1484] A "numerical database" refers to a database that stores numerical data such as blood test results and sensor data.

[1485] "Image database" refers to a database that stores image data such as X-ray images and CT scans.

[1486] "Analysis" refers to the data processing and evaluation process that detects abnormal values ​​and abnormal areas based on stored clinical diagnostic data and sensor data.

[1487] "AI generates diagnostic results" refers to the process of creating a diagnostic list and its rationale using a machine learning algorithm based on analyzed data.

[1488] "Doctor's diagnosis result" refers to the diagnostic judgment and conclusion made by the doctor regarding the condition of the patient or robot.

[1489] "User Interface" refers to the system's interactive screen through which a user (physician or maintenance technician) can input data, review diagnostic results, and provide feedback.

[1490] "Emotion engine" refers to technology that uses voice analysis and facial expression recognition technology to identify a user's emotional state and assess their stress and fatigue levels.

[1491] "Emotion Data" refers to data regarding emotional states (e.g., stress, joy, sadness, etc.) recognized by the emotion engine.

[1492] "Diagnostic support" refers to providing additional information and warning messages during the diagnostic process to help doctors and maintenance personnel make better decisions.

[1493] "Feedback" refers to information entered as evaluations and impressions of diagnostic results and work processes, which are used as learning data for the system.

[1494] In the present invention, a plurality of hardware and software components are integrated to provide a maintenance and fault diagnosis support system for a factory robot. Specific embodiments will be described below.

[1495] Data collection and input

[1496] The user (maintenance staff) inputs the robot's sensor data into a dedicated terminal. This data includes various sensor data such as temperature, vibration, and current. The terminal collects this data in real time and sends it to a server. A general-purpose PC or smart device is suitable as the terminal to be used.

[1497] Data transmission and storage

[1498] The device sends the input sensor data to the server, which stores the received data in a numerical database and an image database. For example, temperature and vibration data are stored in the numerical database and analyzed as needed.

[1499] Data analysis and fault diagnosis

[1500] The server analyzes the stored sensor data using a data processing module written in Python. The analysis results are passed to an AI module (built using TensorFlow / Keras), which evaluates the possibility of a fault and generates a diagnosis.

[1501] Display and compare diagnostic results

[1502] The server displays the diagnostic results generated by the AI ​​on a user interface, which provides an intuitive graphical user interface (GUI) that allows users (maintenance personnel) to check the diagnostic results.

[1503] Emotion Engine Monitoring

[1504] The device uses an emotion engine to recognize the emotional state of the person checking the diagnosis results. The emotion engine uses facial recognition technology (OpenCV) and voice analysis technology to detect stress and fatigue of the person.

[1505] Emotion-based feedback and warnings

[1506] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the person in charge is feeling stressed, the server displays a warning message urging the person to reconfirm the diagnosis result. In addition, if the person in charge's emotional state affects the diagnosis, the server provides additional diagnostic support information.

[1507] Recording and feedback

[1508] The server stores the user's final diagnosis results in a numerical database. It also receives and stores feedback on the user's emotional state during the diagnosis process and how helpful the AI ​​diagnosis was. This feedback is used as learning data for the AI, helping to improve its diagnostic accuracy in the future.

[1509] Implementing specific examples

[1510] In a factory, a maintenance worker performs a routine inspection of a factory robot. The worker operates a smart device to collect sensor data (such as temperature and vibration) in real time. This data is then sent to a server, where it is analyzed and displayed along with the results of a diagnosis made by an AI module. Meanwhile, an emotion engine recognizes the worker's stress level and provides feedback to encourage relaxation as needed.

[1511] (Example of prompts for a generative AI model)

[1512] "Detect stress levels in factory maintenance personnel and generate prompts for an assistant system that provides feedback to encourage breaks."

[1513] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1514] Step 1:

[1515] The user (maintenance technician) begins maintenance work on a factory robot. First, various sensor data such as temperature, vibration, and current are collected from sensors connected to a dedicated terminal and input into the terminal. At this time, the data from each sensor is displayed in real time on the terminal's UI.

[1516] Step 2:

[1517] The device sends the collected sensor data to the server. The sensor data includes temperature values ​​measured every 0.1 seconds, vibration frequency spectrum, and current time series data. The device then converts the data into a specific format (e.g., JSON format) and sends it to the server via the network.

[1518] Step 3:

[1519] The server sequentially stores the received sensor data in a numeric database and an image database. When storing the data in the database, the data is separated by sensor type and stored in the appropriate field. For example, temperature data is stored in the "temperature" field, and vibration data is stored in the "vibration" field.

[1520] Step 4:

[1521] The server analyzes the stored sensor data. Specifically, it detects abnormal values ​​in temperature data, analyzes the frequency of vibration data, and recognizes patterns in current data. From this analysis, an anomaly detection algorithm is applied to extract abnormal values ​​and patterns. The analysis results include peak temperature values, the ratio of high-frequency components in vibration, and abnormal current peaks.

[1522] Step 5:

[1523] The server passes the analysis results to an AI module (using TensorFlow / Keras), which then generates a diagnosis. The AI ​​module evaluates current sensor data based on past learning data and predicts the likelihood of robot failure. The diagnosis results include the possibility of wear on specific parts, failure due to overheating, and damage due to abnormal vibration.

[1524] Step 6:

[1525] The server displays the diagnostic results generated by the AI ​​on a user interface. The user interface uses a GUI to visually display the diagnostic results, allowing the user to intuitively understand them. Specifically, it displays an overview of the diagnostic results, highlights the areas where abnormalities have occurred, and recommends actions (e.g., part replacement, adjustment).

[1526] Step 7:

[1527] The device uses an emotion engine to recognize the user's emotional state while checking the diagnosis results. The emotion engine captures the user's face with the device's built-in camera and analyzes their emotions using a facial expression recognition algorithm (OpenCV). This analysis determines whether the user is feeling stressed or relaxed.

[1528] Step 8:

[1529] The server receives the emotion data sent from the emotion engine and provides diagnostic support based on the emotional state of the person in charge. For example, if the emotion analysis detects that the user is feeling stressed, the server displays a warning message on the user interface. This message may include a message to reconfirm the diagnosis result or to take a break.

[1530] Step 9:

[1531] The server stores the final diagnosis results and emotion data provided by the user in a numerical database. The stored data will be used as training data to improve the accuracy of the diagnosis. In addition, the user can input feedback on the final diagnosis results via the user interface, and this feedback will also be stored in the database.

[1532] Step 10:

[1533] The server analyzes the stored final diagnosis results and emotion data and uses them to retrain the AI ​​model, a process that improves future diagnosis accuracy and overall system performance.

[1534] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1535] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1536] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1537] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1538] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1539] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1540] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1541] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1542] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1543] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1544] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1545] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1546] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1547] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1548] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1549] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1550] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1551] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1552] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1553] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1554] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1555] The following is further disclosed regarding the above embodiment.

[1556] (Claim 1)

[1557] means for receiving clinical diagnostic data of a patient;

[1558] means for storing the received clinical diagnostic data in a database;

[1559] means for analyzing the stored clinical diagnostic data;

[1560] A means for AI to generate diagnostic results based on the analysis results, and

[1561] A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface;

[1562] A system that provides a means to make a final diagnosis based on the integrated diagnostic results.

[1563] (Claim 2)

[1564] 10. The system of claim 1, further comprising means for storing the final diagnosis results and feedback in a database.

[1565] (Claim 3)

[1566] 10. The system of claim 1, further comprising means for allowing a physician to easily input data, review diagnostic results, and provide feedback via a user interface.

[1567] "Example 1"

[1568] (Claim 1)

[1569] means for receiving patient biometric information;

[1570] means for storing the received biometric information in a recording device;

[1571] means for analyzing the stored biometric information;

[1572] A means for the artificial intelligence module to generate a diagnosis result based on the analysis result;

[1573] a means for integrating the diagnostic results of the artificial intelligence module and the diagnostic results of the doctor and displaying them on a display device;

[1574] A means for making a final diagnosis based on the integrated diagnostic results;

[1575] A system including a means for collecting feedback during the diagnostic process.

[1576] (Claim 2)

[1577] 10. The system of claim 1, further comprising means for storing the final diagnostic results and feedback in a recording device.

[1578] (Claim 3)

[1579] 10. The system of claim 1, further comprising means for allowing a physician to easily input data, confirm diagnostic results, and input feedback via a display device.

[1580] "Application Example 1"

[1581] (Claim 1)

[1582] means for receiving clinical diagnostic data of a patient;

[1583] means for storing the received clinical diagnostic data in a database;

[1584] means for analyzing the stored clinical diagnostic data;

[1585] A means for AI to generate diagnostic results based on the analysis results, and

[1586] A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface;

[1587] A means for making a final diagnosis based on the integrated diagnostic results;

[1588] means for receiving quality inspection data;

[1589] means for analyzing the received quality inspection data;

[1590] A method for AI and robots to generate quality diagnosis results based on the analysis results, and

[1591] A means to integrate the diagnostic results of AI and robots and display them on a user interface,

[1592] A system including a means for providing audio feedback.

[1593] (Claim 2)

[1594] 10. The system of claim 1, further comprising means for storing the final diagnosis results and feedback in a database.

[1595] (Claim 3)

[1596] 10. The system of claim 1, further comprising means for enabling a diagnostician to easily input data, check diagnostic results, and input feedback via a user interface.

[1597] "Example 2: Combining Emotion Engines"

[1598] (Claim 1)

[1599] means for receiving clinical diagnostic data of a patient;

[1600] means for storing the received clinical diagnostic data in a database;

[1601] means for analyzing the stored clinical diagnostic data;

[1602] A means for the AI ​​to generate a diagnosis based on the analysis results;

[1603] A means for integrating the AI ​​diagnosis results and the doctor's diagnosis results and displaying them on a user interface;

[1604] a means for recognizing the emotional state of the physician;

[1605] a means for providing feedback or warnings based on emotional state;

[1606] A system that provides a means to make a final diagnosis based on the integrated diagnostic results.

[1607] (Claim 2)

[1608] 10. The system of claim 1, further comprising means for storing the final diagnosis results and feedback in a database.

[1609] (Claim 3)

[1610] 10. The system of claim 1, further comprising means for allowing a physician to easily input data, review diagnostic results, and provide feedback via a user interface.

[1611] "Application example 2 when combining emotion engines"

[1612] (Claim 1)

[1613] means for receiving clinical diagnostic data of a patient;

[1614] means for storing the received clinical diagnostic data in a numerical database and an image database;

[1615] means for analyzing the stored clinical diagnostic data;

[1616] A means for AI to evaluate the possibility of diagnosis based on the analysis results and generate a diagnosis result;

[1617] A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface;

[1618] A means for making a final diagnosis based on the integrated diagnostic results;

[1619] means for receiving emotion data using an emotion engine that recognizes the emotional state of a user (doctor);

[1620] A system including a means for providing diagnostic assistance based on the emotional state of a physician.

[1621] (Claim 2)

[1622] 10. The system of claim 1, further comprising means for storing the final diagnosis results and feedback and emotional state data in a numerical database.

[1623] (Claim 3)

[1624] 10. The system of claim 1, further comprising means for allowing a physician to easily enter data, review diagnostic results, enter feedback, and submit emotional data via a user interface. [Explanation of symbols]

[1625] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving clinical diagnostic data of a patient; means for storing the received clinical diagnostic data in a database; means for analyzing the stored clinical diagnostic data; A means for AI to generate diagnostic results based on the analysis results, and A means to integrate the AI ​​diagnosis results and the doctor's diagnosis results and display them on a user interface; A system that provides a means to make a final diagnosis based on the integrated diagnostic results.

2. 10. The system of claim 1, further comprising means for storing the final diagnostic results and feedback in a database.

3. 10. The system of claim 1, further comprising means for allowing a physician to easily input data, confirm diagnostic results, and provide feedback via a user interface.

Images